Add support for experimental encryption in Hookshot

Squashed based on the work done in https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3042

commit 49932b8f3c
Author: Slavi Pantaleev <slavi@devture.com>
Date:   Sat Dec 16 09:21:31 2023 +0200

    Fix syntax in matrix-bridge-hookshot/tasks/reset_encryption.yml

    Also, this task always does work and side-effects, so it should always report changes
    (`changed_when: true`).

commit 6bdf7a9dcb
Author: Slavi Pantaleev <slavi@devture.com>
Date:   Sat Dec 16 09:12:41 2023 +0200

    Add Hookshot validation task to ensure queue settings are set when encryption is enabled

commit 8c531b7971
Author: Slavi Pantaleev <slavi@devture.com>
Date:   Sat Dec 16 09:10:17 2023 +0200

    Add missing variables rewiring in group_vars/matrix_servers for Hookshot

commit 7d26dabc2f
Author: Slavi Pantaleev <slavi@devture.com>
Date:   Sat Dec 16 09:08:19 2023 +0200

    Add defaults for matrix_hookshot_queue_host and matrix_hookshot_queue_port

commit 74f91138c9
Author: Slavi Pantaleev <slavi@devture.com>
Date:   Sat Dec 16 09:06:17 2023 +0200

    Fix syntax for connecting to additional networks for Hookshot

commit ca7b41f3f2
Author: Slavi Pantaleev <slavi@devture.com>
Date:   Sat Dec 16 09:05:28 2023 +0200

    Fix indentation and remove unnecessary if-statements

commit ac4a918d58
Author: Slavi Pantaleev <slavi@devture.com>
Date:   Sat Dec 16 09:04:44 2023 +0200

    Add missing --network for Hookshot

    This seems to have been removed by accident.

commit 6a81fa208f
Author: Slavi Pantaleev <slavi@devture.com>
Date:   Sat Dec 16 09:02:47 2023 +0200

    Make automatic Redis enabling safer, when Hookshot encryption enabled

    If we ever default encryption to enabled for Hookshot, we only wish to force-enable Redis if Hookshot is actually enabled.

commit 75a8e0f2a6
Author: Slavi Pantaleev <slavi@devture.com>
Date:   Sat Dec 16 09:01:10 2023 +0200

    Fix typo

commit 98ad182eac
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 22:37:40 2023 +0100

    Add defaults for Hookshot's encryption

commit 29fa9fab15
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 22:35:11 2023 +0100

    Improve wording of Hookshot's encryption section

commit 4f835e0560
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 22:28:52 2023 +0100

    use safer mount options for the container's files

commit 8c93327e25
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 22:26:01 2023 +0100

    fix filename

commit 03a7bb6e77
Merge: e55d7694 06047763
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 22:23:44 2023 +0100

    Merge branch 'HarHarLinks/hookshot-encryption' of https://github.com/real-joshua/matrix-docker-ansible-deploy into HarHarLinks/hookshot-encryption

commit 06047763bb
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 22:15:54 2023 +0100

    Update roles/custom/matrix-bridge-hookshot/templates/config.yml.j2

    change the if statement to not require a variable with a length > 0 and add a filter to json for the redis host

    Co-authored-by: Slavi Pantaleev <slavi@devture.com>

commit e55d769465
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 22:13:50 2023 +0100

    clarify that Redis is required, standardadise on Hookshot with an upper-case first letter for consistency

commit 66706e4535
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 22:08:20 2023 +0100

    Update roles/custom/matrix-bridge-hookshot/templates/config.yml.j2

    fix for a typo

    Co-authored-by: Slavi Pantaleev <slavi@devture.com>

commit f6aaeb9a16
Merge: e5d34002 869dd33f
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 00:22:34 2023 +0100

    Merge branch 'master' into HarHarLinks/hookshot-encryption

commit e5d34002fd
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Fri Dec 15 00:09:27 2023 +0100

    Add Jinja loop to allow adding multiple networks

commit 69f947782d
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Thu Dec 14 23:52:41 2023 +0100

    split if statements for the message queue and experimental encryption support into seperate statements

commit 4c13be1c89
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date:   Thu Dec 14 23:31:19 2023 +0100

    change variable name per spantaleev's suggestion (https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2979#discussion_r1379015551)

commit 9905309aa9
Author: HarHarLinks <kim.brose@rwth-aachen.de>
Date:   Wed Nov 1 16:14:04 2023 +0100

    amend docs

commit 94abf2d5bd
Author: HarHarLinks <kim.brose@rwth-aachen.de>
Date:   Wed Nov 1 16:05:22 2023 +0100

    draft encryption support for hookshot

docs/configuring-playbook-bridge-hookshot.md

@@ -23,6 +23,11 @@ Other configuration options are available via the `matrix_hookshot_configuration
 
 Finally, run the playbook (see [installing](installing.md)).
 
+### End-to-bridge encryption
+
+You can enable [experimental encryption](https://matrix-org.github.io/matrix-hookshot/latest/advanced/encryption.html) for Hookshot by adding `matrix_hookshot_experimental_encryption_enabled: true` to your configuration (`vars.yml`) and [executing the playbook](installing.md) again.
+
+Should the crypto store be corrupted, you can reset it by executing this Ansible playbook with the tag `reset-hookshot-encryption` added, for example `ansible-playbook -i inventory/hosts setup.yml -K --tags=reset-hookshot-encryption`).
 
 ## Usage
 

group_vars/matrix_servers

@@ -1385,6 +1385,16 @@ matrix_hookshot_systemd_wanted_services_list: |
     (['matrix-' + matrix_homeserver_implementation + '.service'])
     +
     (['matrix-nginx-proxy.service'] if matrix_nginx_proxy_enabled else [])
+    +
+    ([(redis_identifier + '.service')] if redis_enabled and matrix_hookshot_queue_host == redis_identifier else [])
+  }}
+
+# Hookshot's experimental encryption feature (and possibly others) may benefit from Redis, if available.
+matrix_hookshot_queue_host: "{{ redis_identifier if redis_enabled else '' }}"
+
+matrix_hookshot_container_additional_networks_auto: |
+  {{
+    ([redis_container_network] if redis_enabled and matrix_hookshot_queue_host == redis_identifier else [])
   }}
 
 matrix_hookshot_container_http_host_bind_ports_defaultmapping:
@@ -3359,7 +3369,7 @@ ntfy_visitor_request_limit_exempt_hosts_hostnames_auto: |
 #
 ######################################################################
 
-redis_enabled: "{{ matrix_synapse_workers_enabled }}"
+redis_enabled: "{{ matrix_synapse_workers_enabled or (matrix_hookshot_enabled and matrix_hookshot_experimental_encryption_enabled) }}"
 
 redis_identifier: matrix-redis
 

roles/custom/matrix-bridge-hookshot/defaults/main.yml

@@ -10,6 +10,11 @@ matrix_hookshot_container_image_self_build: false
 matrix_hookshot_container_image_self_build_repo: "https://github.com/matrix-org/matrix-hookshot.git"
 matrix_hookshot_container_image_self_build_branch: "{{ 'main' if matrix_hookshot_version == 'latest' else matrix_hookshot_version }}"
 
+# Specifies additional networks for the Hookshot container to connect with
+matrix_hookshot_container_additional_networks: "{{ matrix_hookshot_container_additional_networks_auto + matrix_hookshot_container_additional_networks_custom }}"
+matrix_hookshot_container_additional_networks_auto: []
+matrix_hookshot_container_additional_networks_custom: []
+
 # renovate: datasource=docker depName=halfshot/matrix-hookshot
 matrix_hookshot_version: 4.7.0
 
@@ -30,6 +35,17 @@ matrix_hookshot_public_endpoint: /hookshot
 matrix_hookshot_appservice_port: 9993
 matrix_hookshot_appservice_endpoint: "{{ matrix_hookshot_public_endpoint }}/_matrix/app"
 
+# The variables below control the queue parameters and may optionally be pointed to a Redis instance.
+# These are required when experimental encryption is enabled (`matrix_hookshot_experimental_encryption_enabled`).
+matrix_hookshot_queue_host: ''
+matrix_hookshot_queue_port: 6739
+
+# Controls whether the experimental end-to-bridge encryption support is enabled.
+# This requires that:
+# - support to also be enabled in the homeserver, see the documentation of Hookshot.
+# - Hookshot to be pointed at a Redis instance via the `matrix_hookshot_queue_*` variables.
+matrix_hookshot_experimental_encryption_enabled: false
+
 # Controls whether metrics are enabled in the bridge configuration.
 # Enabling them is usually enough for a local (in-container) Prometheus to consume them.
 # If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_hookshot_metrics_proxying_enabled`.
@@ -41,7 +57,7 @@ matrix_hookshot_metrics_enabled: false
 matrix_hookshot_metrics_proxying_enabled: false
 
 # There is no need to edit ports.
-# Read the documentation to learn about using hookshot metrics with external Prometheus
+# Read the documentation to learn about using Hookshot metrics with external Prometheus
 # If you still want something different, use matrix_hookshot_container_http_host_bind_ports below to expose ports instead.
 matrix_hookshot_metrics_port: 9001
 

roles/custom/matrix-bridge-hookshot/tasks/main.yml

@@ -9,6 +9,12 @@
     - when: matrix_hookshot_enabled | bool
       ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml"
 
+- tags:
+    - reset-hookshot-encryption
+  block:
+    - when: matrix_hookshot_enabled | bool
+      ansible.builtin.include_tasks: "{{ role_path }}/tasks/reset_encryption.yml"
+
 - tags:
     - setup-all
     - setup-hookshot

roles/custom/matrix-bridge-hookshot/tasks/reset_encryption.yml

@@ -0,0 +1,14 @@
+---
+
+- name: Resetting Hookshot's crypto store
+  ansible.builtin.command:
+    cmd: |
+      {{ devture_systemd_docker_base_host_command_docker }} run
+      --rm
+      --name={{ matrix_hookshot_container_ident }}-reset-crypto
+      --user={{ matrix_user_uid }}:{{ matrix_user_gid }}
+      --cap-drop=ALL
+      --mount type=bind,src={{ matrix_hookshot_base_path }}/config.yml,dst=/config.yml
+      {{ matrix_hookshot_docker_image }}
+      yarn start:resetcrypto
+  changed_when: true

roles/custom/matrix-bridge-hookshot/tasks/validate_config.yml

@@ -87,6 +87,12 @@
   with_items:
     - "matrix_hookshot_provisioning_secret"
 
+- name: Fail if no Redis queue enabled when Hookshot encryption is enabled
+  ansible.builtin.fail:
+    msg: >-
+      You need to define a required configuration setting (`{{ item }}`) to enable Hookshot encryption.
+  when: "matrix_hookshot_experimental_encryption_enabled and matrix_hookshot_queue_host == ''"
+
 - name: (Deprecation) Catch and report old metrics usage
   ansible.builtin.fail:
     msg: >-

roles/custom/matrix-bridge-hookshot/templates/config.yml.j2

@@ -107,6 +107,16 @@ metrics:
   # (Optional) Prometheus metrics support
   #
   enabled: {{ matrix_hookshot_metrics_enabled | to_json }}
+{% if matrix_hookshot_queue_host != '' %}
+queue:
+  monolithic: true
+  port: {{ matrix_hookshot_queue_port }}
+  host: {{ matrix_hookshot_queue_host | to_json }}
+{% endif %}
+{% if matrix_hookshot_experimental_encryption_enabled %}
+experimentalEncryption:
+  storagePath: /data/encryption
+{% endif %}
 logging:
   # (Optional) Logging settings. You can have a severity debug,info,warn,error
   #

roles/custom/matrix-bridge-hookshot/templates/registration.yml.j2

@@ -28,3 +28,9 @@ namespaces:
 sender_localpart: hookshot
 url: "http://{{ matrix_hookshot_container_url }}:{{ matrix_hookshot_appservice_port }}" # This should match the bridge.port in your config file
 rate_limited: false
+
+{% if matrix_hookshot_experimental_encryption_enabled %}
+de.sorunome.msc2409.push_ephemeral: true
+push_ephemeral: true
+org.matrix.msc3202: true
+{% endif %}

roles/custom/matrix-bridge-hookshot/templates/systemd/matrix-hookshot.service.j2

@@ -13,10 +13,9 @@ DefaultDependencies=no
 [Service]
 Type=simple
 Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}"
-ExecStartPre=-{{ devture_systemd_docker_base_host_command_docker }} stop --time={{ devture_systemd_docker_base_container_stop_grace_time_seconds }} {{ matrix_hookshot_container_url }}
-ExecStartPre=-{{ devture_systemd_docker_base_host_command_docker }} rm {{ matrix_hookshot_container_url }}
-
-ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_hookshot_container_url }} \
+ExecStartPre=-{{ devture_systemd_docker_base_host_command_docker }} stop --time={{ devture_systemd_docker_base_container_stop_grace_time_seconds }} {{ matrix_hookshot_container_ident }}
+ExecStartPre=-{{ devture_systemd_docker_base_host_command_docker }} rm {{ matrix_hookshot_container_ident }}
+ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} create --rm --name {{ matrix_hookshot_container_ident }} \
           --log-driver=none \
           --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
           --cap-drop=ALL \
@@ -30,11 +29,18 @@ ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name
           {% endfor %}
           {{ matrix_hookshot_docker_image }}
 
-ExecStop=-{{ devture_systemd_docker_base_host_command_docker }} stop --time={{ devture_systemd_docker_base_container_stop_grace_time_seconds }} {{ matrix_hookshot_container_url }}
-ExecStop=-{{ devture_systemd_docker_base_host_command_docker }} rm {{ matrix_hookshot_container_url }}
+{% for network in matrix_hookshot_container_additional_networks %}
+ExecStartPre={{ devture_systemd_docker_base_host_command_docker }} network connect {{ network }} {{ matrix_hookshot_container_ident }}
+{% endfor %}
+
+ExecStart={{ devture_systemd_docker_base_host_command_docker }} start --attach {{ matrix_hookshot_container_ident }}
+
+ExecStop=-{{ devture_systemd_docker_base_host_command_docker }} stop --time={{ devture_systemd_docker_base_container_stop_grace_time_seconds }} {{ matrix_hookshot_container_ident }}
+ExecStop=-{{ devture_systemd_docker_base_host_command_docker }} rm {{ matrix_hookshot_container_ident }}
+
 Restart=always
 RestartSec=30
-SyslogIdentifier={{ matrix_hookshot_container_url }}
+SyslogIdentifier={{ matrix_hookshot_container_ident }}
 
 [Install]
 WantedBy=multi-user.target