Related to:

- https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1287
- https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1288

.. but does it in a more complete way

Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1247

Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1225

Conversation: https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1083#discussion_r638671860

Signed-off-by: Ahmad Haghighi <haghighi@fedoraproject.org>

Signed-off-by: Ahmad Haghighi <haghighi@fedoraproject.org>

This reverts commit 1fb54a37cb.

Seems like it's been pulled or something. It used to exist, but not
anymore. Not sure what's going on.

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1017

Related to
https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1010

Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1010

This reverts commit f825c7c263.

Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1010

Fixes a regression introduced in ffe649a240

Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/958

Fixes a regression caused by a5ee39266c.

If the user id and group id were different than 991:991
(which used to be a hardcoded default for us long ago),
there was a mismatch between what Synapse was trying to use (991:991)
and what it was actually started with (in `--user=..`). It was then
trying to change ownership, which was failing.

This was mostly affecting newer installations which were not using the
991:991 defaults we had long ago (since a1c5a197a9).

This allows us to benefit from helpful things it does for us,
like enabling jemalloc: https://github.com/matrix-org/synapse/pull/8553

We weren't going through `start.py` before, because it was causing some
conflict with our `docker run --user=...` stuff, but it doesn't seem
to be a problem anymore.

Having done this, we won't need to do things like
https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/941
anymore.

Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456

`::` leads to errors like:

> socket.gaierror: [Errno -9] Address family for hostname not supported

This will make data migration easier.

These are just defensive cleanup tasks that we run.
In the good case, there's nothing to kill or remove, so they trigger an
error like this:

> Error response from daemon: Cannot kill container: something: No such container: something

and:

> Error: No such container: something

People often ask us if this is a problem, so instead of always having to
answer with "no, this is to be expected", we'd rather eliminate it now
and make logs cleaner.

In the event that:
- a container is really stuck and needs cleanup using kill/rm
- and cleanup fails, and we fail to report it because of error
suppression (`2>/dev/null`)

.. we'd still get an error when launching ("container name already in use .."),
so it shouldn't be too hard to investigate.

Not specifying bind addresses for the worker resulted in this warning:

> synapse.app - 47 - WARNING - None - Failed to listen on 0.0.0.0, continuing because listening on [::]

Additionally, metrics listening only on 127.0.0.1 seems like a no-op.
Only having it accessible from within the container is likely not what
we intend. Changed that to all interfaces as well.

Whether it actually gets exposed or not depends on the systemd service
and `matrix_synapse_workers_container_host_bind_address`.

> synapse.app.generic_worker - 606 - WARNING - None - Metrics listener configured, but enable_metrics is not True!

This switches the `docker exec` method of spawning
Synapse workers inside the `matrix-synapse` container with
dedicated containers for each worker.

We also have dedicated systemd services for each worker,
so this are now:
- more consistent with everything else (we don't use systemd
instantiated services anywhere)
- we don't need the "parse systemd instance name into worker name +
port" part
- we don't need to keep track of PIDs manually
- we don't need jq (less depenendencies)
- workers dying would be restarted by systemd correctly, like any other
service
- `docker ps` shows each worker separately and we can observe resource
usage

There was a `matrix_nginx_proxy_enabled|default(False)` check, but:
- it didn't seem to work reliably for some reason (hmm)
- referring to a `matrix_nginx_proxy_*` variable from within the
  `matrix-synapse` role is not ideal
- exposing always happened on `127.0.0.1`, which may not be good enough
  for some rarer setups (where the own webserver is external to the host)

> WARNING: Error loading config file: .dockercfg: $HOME is not defined

.. which appeared in Docker 20.10.