matrix_synapse_systemd_service_post_start_delay_seconds is assigned a string value, and setup fails while creating the service file. It is impossible to compare str and int.
Switch the systemd ExecStartPost health check from docker exec + curl
to polling docker inspect for container health status. This piggybacks
on the container image's built-in HEALTHCHECK instead of duplicating it.
Also add a configurable container health interval (5s for Traefik setups,
15s otherwise) to speed up startup readiness detection without affecting
non-Traefik deployments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, we had a 10-second magical delay.
Now we first do a healthcheck to figure out when it really is up.
Then, we do the same 10-second magical delay to account for the time it
may take for a reverse-proxy (like Traefik) to pick up Synapse's routes.
This is an attempt at optimizing service startup.
The effect is most pronounced when many services are restarted one by one.
The systemd service manager role sometimes does this - for example when `just install-service synapse` runs.
In such cases, a 5-second delay for each Synapse worker service
(or other bridge/bot service that waits on the homeserver) quickly adds up to a lot.
When services are all stopped fully and then started, the effect is not so pronounced, because
`matrix-synapse.service` starts first and pulls all worker services (defined as `Wants=` for it).
Later on, when the systemd service manager role "starts" these worker services, they're started already.
Even if they had a 5-second wait each, it would have happened in parallel.
Many of these do depend on the Synapse master process (`matrix-synapse.service`),
so it makes sense to do it.
Furthermore, we're doing it so that one can stop the `matrix-synapse.service`
service and have systemd cascade this into stopping all the workers as well.
This is useful for easily stopping all of Synapse, so that Postgres
upgrades (`--tags=upgrade-postgres`) can happen cleanly.
Postgres upgrades currently stop `devture_postgres_systemd_services_to_stop_for_maintenance_list` which
includes Synapse, but stopping just the master process and leaving workers running is not safe enough and sometimes leads to errors like:
> ERROR: insert or update on table "event_forward_extremities" violates foreign key constraint "event_forward_extremities_event_id"
With this dependency in place, stopping `matrix-synapse.service` will stop all Synapse processes.
This was mostly affecting the stream writer (events) worker, which was
being reported as unhealthy. It wasn't causing any issues, but it just
looked odd and was confusing people.
As an alternative to hitting the regular `/health` healthcheck route (on
the "client" API which this stream writer does not expose),
we may have went for hitting some "replication" API endpoint instead.
This is more complicated and likely unnecessary.
Amélie-Laura Lilith Krejčí
Slavi Pantaleev
Suguru Hirahara
Michael Hollister