The whoami-based approach is now the only implementation for sync worker routing.
It works with all token types (native Synapse, MAS, etc.) and is automatically
enabled when sync workers exist.
The old map-based approach only worked with native Synapse tokens (syt_<b64>_...)
and would give poor results with MAS or other auth systems.
This adds a new routing mechanism for sync workers that resolves access tokens
to usernames via Synapse's whoami endpoint, enabling true user-level sticky
routing regardless of which device or token is used.
Previously, sticky routing relied on parsing the username from native Synapse
tokens (`syt_<base64 username>_...`), which only works with native Synapse auth
and provides device-level stickiness at best. This new approach works with any
auth system (native Synapse, MAS, etc.) because Synapse handles token validation
internally.
Implementation uses nginx's auth_request module with an njs script because:
- The whoami lookup requires an async HTTP subrequest (ngx.fetch)
- js_set handlers must return synchronously and don't support async operations
- auth_request allows the async lookup to complete, then captures the result
via response headers into nginx variables
The njs script:
- Extracts access tokens from Authorization header or query parameter
- Calls Synapse's whoami endpoint to resolve token -> username
- Caches results in a shared memory zone to minimize latency
- Returns the username via a `X-User-Identifier` header
The username is then used by nginx's upstream hash directive for consistent
worker selection. This leverages nginx's built-in health checking and failover.
* Enable Internal Admin API Access separately from Public access.
* Add Config variable for Draupnir Hijack command
And also make the internal admin API be automatically activated when this capability is used.
* Apply suggestions from code review
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Further Refine Internal Admin API
* Add Non Worker Labels for Internal Admin API
* Variable Rename
* Add validation rules for Internal Synapse admin API
* Add Draupnir Admin API required config validation.
* Override `matrix_synapse_reverse_proxy_companion_container_labels_internal_client_synapse_admin_api_traefik_entrypoints` via group vars
* Wire `matrix_bot_draupnir_admin_api_enabled` to `matrix_bot_draupnir_config_admin_enableMakeRoomAdminCommand` in Draupnir's `defaults/main.yml`
* Remove unnecessary `matrix_bot_draupnir_admin_api_enabled` override from `group_vars/matrix_servers`
The same value is now (more appropriately) defined in Draupnir's `defaults/main.yml` file anyway.
* Add additional condition (`matrix_bot_draupnir_enabled`) for enabling `matrix_synapse_container_labels_internal_client_synapse_admin_api_enabled`
* Use a separate task for validating `matrix_bot_draupnir_admin_api_enabled` when `matrix_bot_draupnir_config_admin_enableMakeRoomAdminCommand`
The other task deals with checking for null and not-blank and can't handle booleans properly.
---------
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
Until now, most sections were specifying their own values for these.
For `client_max_body_size`, a value of 25MB was hardcoded in most places.
This was generally OK, but..
Some sections (those generated by the `render_locations_to_upstream` macro), were not specifying these options
and were ending up with a default value for configuration options for `client_max_body_size` (likely 1MB), etc.
From now on:
- we use individual variables for defining these for the Client-Server
and Federation API and apply these once at the `server` level
- we keep auto-determining the `client_max_body_size` for the
Client-Server API based on `matrix_synapse_max_upload_size_mb`
- we keep auto-calculating the `client_max_body_size` for the Federation
API based on the one for the Client API, but now also add a "minimum"
value (`matrix_synapse_reverse_proxy_companion_federation_api_client_max_body_size_mb_minimum: 100`)
to ensure we don't go too low
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/4100
This:
- brings consistency - no more mixing `_name_prefix` and `_registry_prefix`
- adds extensibility - a future patch will allow reconfiguring all registry prefixes for all roles in the playbook
We still have `_docker_` vs `_container_` inconsistencies.
These may be worked on later.
This is done for a few reasons:
- less globals and more indepdendence for each role is better. We rely
on various externally-hosted roles and they don't rely on this global
either.
- `matrix_container_global_registry_prefix` could make people think they
could just override this variable and have all their images pull from
elsewhere. This is rarely the case, unless you've taken special care
to mirror all the various components (from their respective
registries) to your own. In such a case, you probably know what you're
mirroring and can adjust individual variables.
- nowadays, various components live on different registries.
With Docker Inc tightening rate limits for Docker Hub, it's even more
likely that we'll see increased diversity in where images are hosted
Related to 0241c71a4c
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3270#issuecomment-2143782962
With this change, it should be possible for people to adjust the Docker
dependency from `docker.service` to something else (e.g. `pkg-ContainerManager-dockerd.service`),
or to completely eliminate it by setting `devture_systemd_docker_base_docker_service_name` to an empty string.
This makes it easier for people to use the playbook against a Synology DSM server.
We noticed that the reporting function in Element is broken, at least when using the 'specialized-workers' preset.
This changes the `main_override_locations_regex` of the reverse proxy companion to ensure that requests to `/_matrix/client/v3/rooms/<roomid>/report/<message>` always land on the main process.
After some checking, it seems like there's `/_synapse/client/oidc`,
but no such thing as `/_synapse/oidc`.
I'm not sure why we've been reverse-proxying these paths for so long
(even in as far back as the `matrix-nginx-proxy` days), but it's time we
put a stop to it.
The OIDC docs have been simplified. There's no need to ask people to
expose the useless `/_synapse/oidc` endpoint. OIDC requires
`/_synapse/client/oidc` and `/_synapse/client` is exposed by default
already.
Issues and Pull Requests were not migrated to the new
organization/repository, so `matrix-org/synapse/pull` and
`matrix-org/synapse/issues` references were kept as-is.
`matrix-org/synapse-s3-storage-provider` references were also kept,
as that module still continues living under the `matrix-org` organization.
This patch mainly aims to change documentation-related things, not actual
usage in full yet. For polish that, another more comprehensive patch is coming later.
This was meant to serve as an intermediary for services needing to reach
the homeserver. It was used like that for a while in this
`bye-bye-nginx-proxy` branch, but was never actually public.
It has recently been superseded by homeserver-like services injecting
themselves into a new internal Traefik entrypoint
(see `matrix_playbook_internal_matrix_client_api_traefik_entrypoint_*`),
so `matrix-homeserver-proxy` is no longer necessary.
---
This is probably a good moment to share some benchmarks and reasons
for going with the internal Traefik entrypoint as opposed to this nginx
service.
1. (1400 rps) Directly to Synapse (`ab -n 1000 -c 100 http://matrix-synapse:8008/_matrix/client/versions`
2. (~900 rps) Via `matrix-homeserver-proxy` (nginx) proxying to Synapse (`ab -n 1000 -c 100 http://matrix-homeserver-proxy:8008/_matrix/client/versions`)
3. (~1200 rps) Via the new internal entrypoint of Traefik (`matrix-internal-matrix-client-api`) proxying to Synapse (`ab -n 1000 -c 100 http://matrix-traefik:8008/_matrix/client/versions`)
Besides Traefik being quicker for some reason, there are also other
benefits to not having this `matrix-homeserver-proxy` component:
- we can reuse what we have in terms of labels. Services can register a few extra labels on the new Traefik entrypoint
- we don't need services (like `matrix-media-repo`) to inject custom nginx configs into `matrix-homeserver-proxy`. They just need to register labels, like they do already.
- Traefik seems faster than nginx on this benchmark for some reason, which is a nice bonus
- no need to run one extra container (`matrix-homeserver-proxy`) and execute one extra Ansible role
- no need to maintain a setup where some people run the `matrix-homeserver-proxy` component (because they have route-stealing services like `matrix-media-repo` enabled) and others run an optimized setup without this component and everything needs to be rewired to talk to the homeserver directly. Now, everyone can go through Traefik and we can all run an identical setup
Downsides of the new Traefik entrypoint setup are that:
- all addon services that need to talk to the homeserver now depend on Traefik
- people running their own Traefik setup will be inconvenienced - they
need to manage one additional entrypoint
We'd be adding integration with an internal Traefik entrypoint
(`matrix_playbook_internal_matrix_client_api_traefik_entrypoint`),
so renaming helps disambiguate things.
There's no need for deperecation tasks, because the old names
have only been part of this `bye-bye-nginx-proxy` branch and not used by
anyone publicly.
This also updates validation tasks and documentation, pointing to
variables in the matrix-synapse role which don't currently exist yet
(e.g. `matrix_synapse_container_labels_client_synapse_admin_api_enabled`).
These variables will be added soon, as Traefik labels are added to the
`matrix-synapse` role. At that point, the `matrix-synapse-reverse-proxy-companion` role
will be updated to also use them.
renovate[bot]
Slavi Pantaleev
Catalan Lover
Suguru Hirahara
Aine
David Mehren
Michael Hollister
Charles Wright
Samuel Meenzen