Merge branch 'spantaleev:master' into polychat-appservice

bin/ansible-all-hosts.sh

@@ -8,7 +8,7 @@
 #
 
 # set playbook root path
-root=$(dirname "$(readlink -f "$0")")/../..
+root=$(dirname "$(readlink -f "$0")")/..
 
 # set default tags or get from first argument if any
 tags="${1:-setup-all,start}"

docs/configuring-playbook-traefik.md

@@ -35,7 +35,7 @@ devture_traefik_dashboard_basicauth_user: YOUR_USERNAME_HERE
 devture_traefik_dashboard_basicauth_password: YOUR_PASSWORD_HERE
 ```
 
-**WARNING**: enabling the dashboard on a hostname you use for something else (like `matrix_server_fqn_matrix` in the configuration above) may cause conflicts. Enabling the Traefik Dashboard makes Traefik capture all `/dashboard` and `/api` requests and forward them to itself. If any of the services hosted on the same hostname requires any of these 2 URL prefixes, you will experience problems. So far, we're not aware of any playbook services which occupy these endpoints and are likely to cause conflicts.
+**WARNING**: Enabling the dashboard on a hostname you use for something else (like `matrix_server_fqn_matrix` in the configuration above) may cause conflicts. Enabling the Traefik Dashboard makes Traefik capture all `/dashboard` and `/api` requests and forward them to itself. If any of the services hosted on the same hostname requires any of these 2 URL prefixes, you will experience problems. So far, we're not aware of any playbook services which occupy these endpoints and are likely to cause conflicts.
 
 ## Additional configuration
 
@@ -48,3 +48,92 @@ devture_traefik_configuration_extension_yaml: |
   api:
     dashboard: true
 ```
+
+## Reverse-proxying another service behind Traefik
+
+The preferred way to reverse-proxy additional services behind Traefik would be to start the service as another container, configure the container with the corresponding Traefik [container labels](https://docs.docker.com/config/labels-custom-metadata/) (see [Traefik & Docker](https://doc.traefik.io/traefik/routing/providers/docker/)), and connect the service to the `traefik` network. Some services are also already available via the compatible [mash-playbook](https://github.com/mother-of-all-self-hosting/mash-playbook), but take a look at the minor [interoperability adjustments](https://github.com/mother-of-all-self-hosting/mash-playbook/blob/main/docs/interoperability.md).
+
+However, if your service does not run on a container or runs on another machine, the following configuration might be what you are looking for.
+
+## Reverse-proxying a remote HTTP/HTTPS service behind Traefik
+
+If you want to host another webserver would be reachable via `my-fancy-website.mydomain.com` from the internet and via `https://<internal webserver IP address>:<internal port>` from inside your network, you can make the playbook's integrated Traefik instance reverse-proxy the traffic to the correct host.
+
+Prerequisites: DNS and routing for the domain `my-fancy-website.mydomain.com` need to be set up correctly. In this case, you'd be pointing the domain name to your Matrix server - `my-fancy-website.mydomain.com` would be a CNAME going to `matrix.example.com`.
+
+First, we have to adjust the static configuration of Traefik, so that we can add additional configuration files:
+
+```yaml
+# We enable all config files in the /config/ folder to be loaded.
+# `/config` is the path as it appears in the Traefik container.
+# On the host, it's actually `/matrix/traefik/config` (as defined in `devture_traefik_config_dir_path`).
+devture_traefik_configuration_extension_yaml: |
+  providers:
+    file:
+      directory: /config/
+      watch: true
+      filename: ""
+```
+
+If you are using a self-signed certificate on your webserver, you can tell Traefik to trust your own backend servers by adding more configuration to the static configuration file. If you do so, bear in mind the security implications of disabling the certificate validity checks towards your back end.
+
+```yaml
+# We enable all config files in the /config/ folder to be loaded and
+devture_traefik_configuration_extension_yaml: |
+  providers:
+    file:
+      directory: /config/
+      watch: true
+      filename: ""
+  serversTransport:
+    insecureSkipVerify: true
+```
+
+
+Next, you have to add a new dynamic configuration file for Traefik that contains the actual information of the server using the `aux_file_definitions` variable. In this example, we will terminate SSL at the Traefik instance and connect to the other server via HTTPS. Traefik will now take care of managing the certificates. 
+
+```yaml
+aux_file_definitions:
+  - dest: "{{ devture_traefik_config_dir_path }}/provider_my_fancy_website.yml"
+    content: |
+      http:
+        routers:
+          webserver-router:
+            rule: Host(`my_fancy_website.mydomain.com`)
+            service: webserver-service
+            tls:
+              certResolver: default
+        services:
+          webserver-service:
+            loadBalancer:
+              servers:
+                - url: "https://<internal webserver IP address>:<internal port>"
+```
+Changing the `url` to one with an `http://` prefix would allow to connect to the server via HTTP.
+
+## Reverse-proxying another service behind Traefik without terminating SSL
+
+If you do not want to terminate SSL at the Traefik instance (for example, because you're already terminating SSL at other webserver), you need to adjust the static configuration in the same way as in the previous chapter in order to be able to add our own dynamic configuration files. Afterwards, you can add the following configuration to your `vars.yml` configuration file:
+
+```yaml
+aux_file_definitions:
+  - dest: "{{ devture_traefik_config_dir_path }}/providers_my_fancy_website.yml"
+    content: |
+      tcp:
+        routers:
+          webserver-router:
+            rule: Host(`my_fancy_website.mydomain.com`)
+            service: webserver-service
+            tls:
+              passthrough: true
+        services:
+          webserver-service:
+            loadBalancer:
+              servers:
+                - url: "https://<internal webserver IP address>:<internal port>"
+```
+Changing the `url` to one with an `http://` prefix would allow to connect to the server via HTTP.
+
+With these changes, all TCP traffic will be reverse-proxied to the target system. 
+
+**WARNING**: This configuration might lead to problems or need additional steps when a [certbot](https://certbot.eff.org/) behind Traefik also tries to manage [Let's Encrypt](https://letsencrypt.org/) certificates, as Traefik captures all traffic to ```PathPrefix(`/.well-known/acme-challenge/`)```. 

group_vars/matrix_servers

@@ -228,6 +228,12 @@ matrix_homeserver_sliding_sync_url: "{{ matrix_sliding_sync_base_url if matrix_s
 #     (see `matrix_playbook_internal_matrix_client_api_traefik_entrypoint_enabled`)
 # - core services (the homeserver) get a level of ~1000
 # - services that the homeserver depends on (database, Redis, ntfy, Coturn, etc.) get a lower level - between 500 and 1000
+# - Coturn gets a higher level if `devture_systemd_service_manager_service_restart_mode == 'one-by-one'` to intentionally delay it, because:
+#  - starting services one by one means that the service manager role waits for each service to fully start before proceeding to the next one
+#  - if Coturn has a lower priority than the homeserver, it would be started before it
+#  - since Coturn is started before the homeserver, there's no container label telling Traefik to get a `matrix.DOMAIN` certificate
+#  - thus, Coturn would spin and wait for a certificate until it fails. We'd get a playbook failure due to it, but service manager will proceed to start all other services anyway.
+#  - only later, when the homeserver actually starts, would that certificate be fetched and dumped
 # - reverse-proxying services get level 3000
 # - Matrix utility services (bridges, bots) get a level of 2000/2200, so that:
 #   - they can start before the reverse-proxy
@@ -336,7 +342,7 @@ devture_systemd_service_manager_services_list_auto: |
     +
     ([{'name': 'matrix-corporal.service', 'priority': 1500, 'groups': ['matrix', 'corporal']}] if matrix_corporal_enabled else [])
     +
-    ([{'name': 'matrix-coturn.service', 'priority': 900, 'groups': ['matrix', 'coturn']}] if matrix_coturn_enabled else [])
+    ([{'name': 'matrix-coturn.service', 'priority': (900 if devture_systemd_service_manager_service_restart_mode == 'clean-stop-start' else 1500), 'groups': ['matrix', 'coturn']}] if matrix_coturn_enabled else [])
     +
     ([{'name': 'matrix-rageshake.service', 'priority': 4000, 'groups': ['matrix', 'rageshake']}] if matrix_rageshake_enabled else [])
     +
@@ -1056,7 +1062,7 @@ matrix_mautrix_facebook_login_shared_secret: "{{ matrix_synapse_ext_password_pro
 
 matrix_mautrix_facebook_bridge_presence: "{{ matrix_synapse_presence_enabled if matrix_synapse_enabled else true }}"
 
-matrix_mautrix_facebook_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_mautrix_facebook_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_mautrix_facebook_metrics_proxying_enabled: "{{ matrix_mautrix_facebook_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_mautrix_facebook_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -1125,7 +1131,7 @@ matrix_mautrix_googlechat_homeserver_token: "{{ '%s' | format(matrix_homeserver_
 
 matrix_mautrix_googlechat_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}"
 
-matrix_mautrix_googlechat_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_mautrix_googlechat_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_mautrix_googlechat_metrics_proxying_enabled: "{{ matrix_mautrix_googlechat_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_mautrix_googlechat_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -1193,7 +1199,7 @@ matrix_mautrix_hangouts_homeserver_token: "{{ '%s' | format(matrix_homeserver_ge
 
 matrix_mautrix_hangouts_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}"
 
-matrix_mautrix_hangouts_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_mautrix_hangouts_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_mautrix_hangouts_metrics_proxying_enabled: "{{ matrix_mautrix_hangouts_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_mautrix_hangouts_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -1259,7 +1265,7 @@ matrix_mautrix_instagram_login_shared_secret: "{{ matrix_synapse_ext_password_pr
 
 matrix_mautrix_instagram_bridge_presence: "{{ matrix_synapse_presence_enabled if matrix_synapse_enabled else true }}"
 
-matrix_mautrix_instagram_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_mautrix_instagram_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_mautrix_instagram_metrics_proxying_enabled: "{{ matrix_mautrix_instagram_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_mautrix_instagram_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -1324,7 +1330,7 @@ matrix_mautrix_signal_appservice_token: "{{ '%s' | format(matrix_homeserver_gene
 
 matrix_mautrix_signal_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}"
 
-matrix_mautrix_signal_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_mautrix_signal_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_mautrix_signal_metrics_proxying_enabled: "{{ matrix_mautrix_signal_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_mautrix_signal_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -1395,7 +1401,7 @@ matrix_mautrix_telegram_homeserver_token: "{{ '%s' | format(matrix_homeserver_ge
 
 matrix_mautrix_telegram_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}"
 
-matrix_mautrix_telegram_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_mautrix_telegram_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_mautrix_telegram_metrics_proxying_enabled: "{{ matrix_mautrix_telegram_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_mautrix_telegram_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -1458,7 +1464,7 @@ matrix_mautrix_twitter_homeserver_token: "{{ '%s' | format(matrix_homeserver_gen
 
 matrix_mautrix_twitter_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}"
 
-matrix_mautrix_twitter_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_mautrix_twitter_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_mautrix_twitter_metrics_proxying_enabled: "{{ matrix_mautrix_twitter_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_mautrix_twitter_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -1519,7 +1525,7 @@ matrix_mautrix_gmessages_homeserver_token: "{{ '%s' | format(matrix_homeserver_g
 
 matrix_mautrix_gmessages_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}"
 
-matrix_mautrix_gmessages_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_mautrix_gmessages_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_mautrix_gmessages_metrics_proxying_enabled: "{{ matrix_mautrix_gmessages_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_mautrix_gmessages_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -1632,7 +1638,7 @@ matrix_mautrix_whatsapp_homeserver_token: "{{ '%s' | format(matrix_homeserver_ge
 
 matrix_mautrix_whatsapp_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}"
 
-matrix_mautrix_whatsapp_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_mautrix_whatsapp_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_mautrix_whatsapp_metrics_proxying_enabled: "{{ matrix_mautrix_whatsapp_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_mautrix_whatsapp_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -1774,8 +1780,7 @@ matrix_hookshot_container_labels_traefik_tls_certResolver: "{{ devture_traefik_c
 
 matrix_hookshot_provisioning_enabled: "{{ matrix_hookshot_provisioning_secret and matrix_dimension_enabled }}"
 
-# We only enable metrics (locally, in the container network) for the bridge if Prometheus is enabled.
-matrix_hookshot_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_hookshot_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_hookshot_metrics_proxying_enabled: "{{ matrix_hookshot_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_hookshot_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -3669,8 +3674,7 @@ devture_postgres_backup_databases: "{{ devture_postgres_managed_databases | map(
 # Most people don't need their own push-server, because they also need their own app to utilize it from.
 matrix_sygnal_enabled: false
 
-# If someone instals Prometheus via the playbook, they most likely wish to monitor Sygnal.
-matrix_sygnal_metrics_prometheus_enabled: "{{ prometheus_enabled }}"
+matrix_sygnal_metrics_prometheus_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_sygnal_hostname: "{{ matrix_server_fqn_sygnal }}"
 
@@ -4000,8 +4004,7 @@ matrix_synapse_tls_private_key_path: ~
 
 matrix_synapse_federation_port_openid_resource_required: "{{ not matrix_synapse_federation_enabled and (matrix_dimension_enabled or matrix_ma1sd_enabled or matrix_user_verification_service_enabled) }}"
 
-# If someone instals Prometheus via the playbook, they most likely wish to monitor Synapse.
-matrix_synapse_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_synapse_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_synapse_metrics_proxying_enabled: "{{ matrix_synapse_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_synapse_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"
@@ -4668,7 +4671,7 @@ matrix_dendrite_container_labels_public_federation_api_traefik_tls: "{{ matrix_f
 matrix_dendrite_container_labels_public_metrics_middleware_basic_auth_enabled: "{{ matrix_metrics_exposure_http_basic_auth_enabled }}"
 matrix_dendrite_container_labels_public_metrics_middleware_basic_auth_users: "{{ matrix_metrics_exposure_http_basic_auth_users }}"
 
-matrix_dendrite_metrics_enabled: "{{ prometheus_enabled }}"
+matrix_dendrite_metrics_enabled: "{{ prometheus_enabled or matrix_metrics_exposure_enabled }}"
 
 matrix_dendrite_metrics_proxying_enabled: "{{ matrix_dendrite_metrics_enabled and matrix_metrics_exposure_enabled }}"
 matrix_dendrite_metrics_proxying_hostname: "{{ matrix_metrics_exposure_hostname }}"

requirements.yml

@@ -22,7 +22,7 @@
   version: v4.97-r0-0-1
   name: exim_relay
 - src: git+https://gitlab.com/etke.cc/roles/grafana.git
-  version: v10.2.3-0
+  version: v10.3.1-0
   name: grafana
 - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-jitsi.git
   version: v9111-1
@@ -31,7 +31,7 @@
   version: v2.8.0-1
   name: ntfy
 - src: git+https://github.com/devture/com.devture.ansible.role.playbook_help.git
-  version: c1f40e82b4d6b072b6f0e885239322bdaaaf554f
+  version: 201c939eed363de269a83ba29784fc3244846048
   name: playbook_help
 - src: git+https://github.com/devture/com.devture.ansible.role.playbook_runtime_messages.git
   version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6
@@ -49,10 +49,10 @@
   version: v2.49.1-0
   name: prometheus
 - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-prometheus-node-exporter.git
-  version: v1.7.0-2
+  version: v1.7.0-3
   name: prometheus_node_exporter
 - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-prometheus-postgres-exporter.git
-  version: v0.14.0-3
+  version: v0.14.0-4
   name: prometheus_postgres_exporter
 - src: git+https://gitlab.com/etke.cc/roles/redis.git
   version: v7.2.3-2

roles/custom/matrix-bot-draupnir/templates/systemd/matrix-bot-draupnir.service.j2

@@ -29,7 +29,8 @@ ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \
 			{% for arg in matrix_bot_draupnir_container_extra_arguments %}
 			{{ arg }} \
 			{% endfor %}
-			{{ matrix_bot_draupnir_docker_image }}
+			{{ matrix_bot_draupnir_docker_image }} \
+			bot --draupnir-config /data/config/production.yaml 
 
 {% for network in matrix_bot_draupnir_container_additional_networks %}
 ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-bot-draupnir

roles/custom/matrix-bot-mjolnir/templates/systemd/matrix-bot-mjolnir.service.j2

@@ -29,7 +29,8 @@ ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create \
 			{% for arg in matrix_bot_mjolnir_container_extra_arguments %}
 			{{ arg }} \
 			{% endfor %}
-			{{ matrix_bot_mjolnir_docker_image }}
+			{{ matrix_bot_mjolnir_docker_image }} \
+			bot --mjolnir-config /data/config/production.yaml 
 
 {% for network in matrix_bot_mjolnir_container_additional_networks %}
 ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} matrix-bot-mjolnir

roles/custom/matrix-cactus-comments-client/defaults/main.yml

@@ -13,7 +13,7 @@ matrix_cactus_comments_client_public_path: "{{ matrix_cactus_comments_client_bas
 matrix_cactus_comments_client_public_path_file_permissions: "0644"
 
 # renovate: datasource=docker depName=joseluisq/static-web-server
-matrix_cactus_comments_client_version: 2.24.2
+matrix_cactus_comments_client_version: 2.25.0
 
 matrix_cactus_comments_client_container_image: "{{ matrix_container_global_registry_prefix }}joseluisq/static-web-server:{{ matrix_cactus_comments_client_container_image_tag }}"
 matrix_cactus_comments_client_container_image_tag: "{{ 'latest' if matrix_cactus_comments_client_version == 'latest' else (matrix_cactus_comments_client_version + '-alpine') }}"

roles/custom/matrix-sliding-sync/defaults/main.yml

@@ -6,7 +6,7 @@
 matrix_sliding_sync_enabled: true
 
 # renovate: datasource=docker depName=ghcr.io/matrix-org/sliding-sync
-matrix_sliding_sync_version: v0.99.14
+matrix_sliding_sync_version: v0.99.15
 
 matrix_sliding_sync_scheme: https
 

roles/custom/matrix-static-files/defaults/main.yml

@@ -8,7 +8,7 @@ matrix_static_files_enabled: true
 matrix_static_files_identifier: matrix-static-files
 
 # renovate: datasource=docker depName=joseluisq/static-web-server
-matrix_static_files_version: 2.24.2
+matrix_static_files_version: 2.25.0
 
 matrix_static_files_base_path: "{{ matrix_base_data_path }}/{{ 'static-files' if matrix_static_files_identifier == 'matrix-static-files' else matrix_static_files_identifier }}"
 matrix_static_files_config_path: "{{ matrix_static_files_base_path }}/config"

setup.yml

@@ -6,6 +6,9 @@
   roles:
     # Most of the roles below are not distributed with the playbook, but downloaded separately using `ansible-galaxy` via the `just roles` command (see `justfile`).
     - role: galaxy/playbook_help
+      tags:
+        - setup-all
+        - install-all
 
     - role: galaxy/systemd_docker_base