Related to c6d8a68e77

Related to https://github.com/element-hq/synapse/pull/17421

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/3427

Related to:
- e4f545c452
- 2481b7dfa4

We've prepared for this by adding the `main` process to the `instance_map` a long time ago,
in 49cb8b7b11.

Related to

- c15e9a0edb
- 01a0527892

This paves the way for installing other roles into `roles/galaxy` using `ansible-galaxy`,
similar to how it's done in:

- https://github.com/spantaleev/gitea-docker-ansible-deploy
- https://github.com/spantaleev/nextcloud-docker-ansible-deploy

In the near future, we'll be removing a lot of the shared role code from here
and using upstream roles for it. Some of the core `matrix-*` roles have
already been extracted out into other reusable roles:

- https://github.com/devture/com.devture.ansible.role.postgres
- https://github.com/devture/com.devture.ansible.role.systemd_docker_base
- https://github.com/devture/com.devture.ansible.role.timesync
- https://github.com/devture/com.devture.ansible.role.vars_preserver
- https://github.com/devture/com.devture.ansible.role.playbook_runtime_messages
- https://github.com/devture/com.devture.ansible.role.playbook_help

We just need to migrate to those.

As stream writer workers are also powered by the `generic_worker`
Synapse app, this necessitated that we provide means for distinguishing
between them and regular `generic_workers`.

I've also taken the time to optimize nginx configuration generation
(more Jinja2 macro usage, less duplication).

Worker names have also changed.
Workers are now named sequentially like this:
- `matrix-synapse-worker-0-generic`
- `matrix-synapse-worker-1-stream-writer-typing`
- `matrix-synapse-worker-2-pusher`

instead of `matrix-synapse-worker_generic_worker-18111` (indexed with a
port number).

People who modify `matrix_synapse_workers_enabled_list` directly will
need to adjust their configuration.

`frontend_proxy` workers have been superseded by `generic_worker` workers.
Related to https://github.com/matrix-org/synapse/pull/13645

Currently, Synapse workers ignore the X-Forwarded headers, which leads to internal Docker IP addresses randomly appearing in the users' device list.

This adds the `x_forwarded: true` option to the worker config, fixing the issue.

`::` leads to errors like:

> socket.gaierror: [Errno -9] Address family for hostname not supported

Not specifying bind addresses for the worker resulted in this warning:

> synapse.app - 47 - WARNING - None - Failed to listen on 0.0.0.0, continuing because listening on [::]

Additionally, metrics listening only on 127.0.0.1 seems like a no-op.
Only having it accessible from within the container is likely not what
we intend. Changed that to all interfaces as well.

Whether it actually gets exposed or not depends on the systemd service
and `matrix_synapse_workers_container_host_bind_address`.

> synapse.app.generic_worker - 606 - WARNING - None - Metrics listener configured, but enable_metrics is not True!

This switches the `docker exec` method of spawning
Synapse workers inside the `matrix-synapse` container with
dedicated containers for each worker.

We also have dedicated systemd services for each worker,
so this are now:
- more consistent with everything else (we don't use systemd
instantiated services anywhere)
- we don't need the "parse systemd instance name into worker name +
port" part
- we don't need to keep track of PIDs manually
- we don't need jq (less depenendencies)
- workers dying would be restarted by systemd correctly, like any other
service
- `docker ps` shows each worker separately and we can observe resource
usage

.. to mitigate log noise - WARNING:
Failed to listen on 0.0.0.0, continuing because listening on [::]

As seen on TV:
https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md#monitoring-workers

(yes, a bit awkward and brittle… xD)

also, worker.yaml.j2:
  - hone worker_name
  - remove worker_pid_file entry (would only be used if worker_daemonize
    set to true; also, synapse only knows about the container namespace
    and thus can not provide the required host-view PID)

& restrict federation listener; frontend_proxy / user_dir don't need it

· needs documentation; no checks yet for port clashes or typos in worker name
· according to https://github.com/matrix-org/synapse/wiki/Workers-setup-with-nginx#results
  about 90% of requests go to the synchrotron endpoint
· thus, the synchrotron worker is especially suited to be load-balanced
· most of the other workers are documented to support only a single instance
· https://github.com/matrix-org/synapse/blob/master/docs/workers.md