Eight scenarios in, the same few things keep deciding whether a scenario is
worth having, and none of them were obvious beforehand:
- Falsify every assertion, not just enough to see red. One control asserted that
a component emitted no DEBUG records from a module and passed just as happily
with that module set to `debug`, because it emits none on a first run either
way. Green for the wrong reason, and only breaking it deliberately showed that.
- `ActiveState == active` with `NRestarts == 0` can both hold while a component
is completely broken, when it catches its errors and retries rather than
exiting. Two of the eight behave that way.
- Startup lines are the oldest in the journal, so tailing loses them. Grep, and
strip ANSI first.
- Parse rendered configuration and assert on structure, so a value under the
wrong key cannot pass.
- `molecule converge` on a running instance does not restart the container, so a
falsification can pass for that reason alone.
Also documents that a role's scenario obliges it to join the automerge list, and
that prek enforces the two staying in step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT
`group_vars/matrix_servers` selects postgres whenever postgres is enabled, which
is the default, so postgres is what essentially every deployment runs. The
scenarios were testing sqlite - a path almost nobody is on.
How little that path is used is not a guess: the mautrix-meta bridges could not
start at all under sqlite, and nobody reported it. Testing the engine users are
actually on is worth more than keeping coverage of the one they are not, so no
scenario is left behind on sqlite.
Four of the eight scenarios have a database and are converted; the other four
have none and are untouched.
molecule-shared/tasks/postgres.yml stands Postgres up on the scenario's network,
with the data directory on a tmpfs since it is thrown away with the container.
The image is pinned at the major the postgres role deploys to new installations
and left to Renovate: when a new major lands, the PR bumping that pin runs every
scenario against it, which is the earliest warning we get that a component does
not cope.
Each scenario gives its database and user names that differ from the role's
defaults, so the component reaching the database proves the role built its
connection string out of them. The assertions moved from "a file appeared at the
path we configured" to "these tables exist", which is strictly stronger: tables
can only appear once the component has resolved the hostname, authenticated with
the credentials the role rendered, and run its migrations to completion.
Costs about 10 seconds per affected scenario (115s to 125s locally for
mautrix-whatsapp), on jobs that run in parallel.
Gotcha worth recording: since Postgres 18 the image puts PGDATA in a versioned
subdirectory and refuses to start if it finds a mount at the old
/var/lib/postgresql/data, so the tmpfs is mounted at /var/lib/postgresql.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT
`just molecule` leaves a shared virtualenv of over 500 MB under var/, plus a
~7 MB Ansible home per role that has a scenario. Neither is reclaimed by
anything today.
`--idle-days N` limits it to what has not been touched recently, so it can be
run unattended without taking the cache out from under a scenario being worked
on right now.
The two directories are named explicitly rather than globbed: var/ holds other
things and must never be removed wholesale.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT
Scenarios install their Galaxy dependencies with `force: true`, so two roles
running at once re-extract the same collections and roles into ~/.ansible and
pull them out from under each other mid-play. It surfaces as a collection that
was working moments earlier going missing:
the connection plugin 'community.docker.docker' was not found
Found while running five scenarios in parallel, where it cost a run.
ANSIBLE_HOME relocates both `collections/` and `roles/`, so one variable covers
both halves; the scenarios' ANSIBLE_ROLES_PATH workaround now follows it rather
than hardcoding ~/.ansible/roles. Left alone if already set, and unset in CI,
where each role runs in its own job and has nothing to collide with.
Verified by removing var/molecule-ansible-home entirely and running
matrix-alertmanager-receiver from cold: green through idempotence, with the
collections and roles landing under the per-role directory - which also shows
nothing was quietly relying on the shared ~/.ansible being populated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT
Molecule checks for a requirements file at the scenario's default path
before it installs anything, and warns "Missing roles requirements file"
when it is absent - so `requirements-file` pointing elsewhere was silently
ignored and the galaxy roles were never installed. CI failed on a missing
ansible-role-docker; the local run passed only because those roles were
already in ~/.ansible/roles from earlier work, which is exactly the kind of
difference a clean CI run exists to expose.
A symlink satisfies Molecule's check while keeping one copy of the pins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three things that would not have scaled to 70 roles:
- The Python and Ansible dependency pins were about to be copied into every
role. They now live once in molecule-shared/, which scenarios reference
relatively, so they cannot drift apart.
- The helper container images used for probing were hardcoded inline. They
are pinned once in molecule-shared/vars.yml, carry `# renovate:`
annotations, and a custom manager in .github/renovate.json keeps them
current - verified with a local Renovate dry run, which offers
curl 8.11.1 -> 8.21.0 and python 3.13 -> 3.14-alpine. Seventy invisible
hardcodes is the blindness class we have been removing elsewhere.
- Running a scenario meant knowing the venv and cd incantation. `just
molecule <role>` does it, and with no argument lists the roles that have
a scenario.
Molecule is deliberately not wired into prek: a run takes minutes, pulls
images and needs Docker, which is fine on request and not fine per commit.
docs/molecule-testing.md covers how to run and write these, including the
four things a role here needs that a standalone role does not. AGENTS.md
points at it rather than carrying the detail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous image versions set ANSIBLE_WORKER_SESSION_ISOLATION=False in an
attempt to restore SSH prompts on Ansible 2.21. That setting does not bring
the prompts back — it makes Ansible hang silently at the first SSH connection
whenever a prompt would have been needed. See
https://github.com/devture/docker-ansible/issues/6
The new image auto-accepts the SSH host keys of previously unknown hosts
instead. The docs stop recommending ANSIBLE_WORKER_SESSION_ISOLATION=False
for direct (non-Docker) runs for the same reason, and now explain how to use
an ssh-agent for passphrase-protected SSH keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Protected rooms, watched policy lists, protection settings and the set of
users allowed to command a bot all belong to a management room rather than
to the bot, and a room can be protected from only one of them. That is what
makes it possible to delegate a single room to another moderator without
handing over power in every other protected room, so it is worth spelling
out, along with what such a split does not cover: the variables naming one
management room, and policy lists shared through subscriptions.
Adding a second management room also means declaring the rooms of a bot
which had been auto-creating its own, which is a step with a sharp edge:
once the bot is no longer auto-creating, any of its rooms missing from the
declaration is unregistered. Document that path and warn about it.
Also drop a note describing "!lists subscribe --insert-before" as a future
release. The role has shipped a version carrying it since v0.2608.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This document and docs/configuring-playbook-livekit-jwt-service.md were both
added in 9970603f1 (Add Element Call support), which squashed a branch that
renamed the role along the way (matrix-jwt-service ->
matrix-livekit-jwt-service). The document belonging to the old name survived
the squash, was never linked to from anywhere and was never touched again,
while the other one is the one referenced by the documentation index, the
README and the Matrix RTC and Element Call documents.
Its remaining content is covered elsewhere, and it was the only place telling
people to enable matrix_livekit_jwt_service_enabled by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The org.matrix.msc4143.rtc_foci property has been dropped from MSC4143 and
Element Call v0.24.0 no longer reads it, so the homeserver's own RTC
transports API is left as the single source of truth. Synapse, continuwuity
and tuwunel all serve it at the unstable path which clients request.
Keeping both around also made Element Web list the same LiveKit transport
twice, as it merges the two sources without deduplicating them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The homeserver only reads appservice registrations on startup, and the
playbook only restarts it at the very end of a run. On the run which
first enables Meowlnir, bot provisioning therefore talked to a
homeserver still running without Meowlnir's registration, hitting
M_UNKNOWN_TOKEN and hanging indefinitely.
Provisioning now verifies the token upfront and restarts the homeserver
itself when needed, so a single run completes even when Meowlnir was
just enabled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Asks the homeserver who Meowlnir's appservice token belongs to.
A 401 (M_UNKNOWN_TOKEN) response means the homeserver is running
without Meowlnir's appservice registration, which is the usual cause
of Meowlnir's "Failed to connect to homeserver" log messages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Since mautrix-meta v26.07, Instagram is a separate bridge upstream (its own binary, container image and configuration schema). As of v0.2608.0, the bridge no longer has a `mode` configuration option at all, so the premise of auto-generating the Instagram role from the Messenger one (one binary, one schema, a mode switch) is gone.
- Drop the role generator (`bin/rebuild-mautrix-meta-instagram.sh` and the `just rebuild-mautrix-meta-instagram` recipe). The two roles are now maintained independently.
- Drop `matrix_bridge_mautrix_meta_instagram_meta_mode` and all mode-derived lookups from the Instagram role. Computed values are unchanged.
- Sync both config templates with the v0.2608.0 example configs: the Instagram template loses `mode` and gains the Instagram-only options; the Messenger template replaces `mode` with the new `tor` option (still derived from `matrix_bridge_mautrix_meta_messenger_meta_mode: facebook-tor`) and loses the Instagram-only `disable_xma_*` options.
- Fix self-building for the Instagram role: build via `Dockerfile.ig` (producing the actual Instagram binary at `/usr/bin/mautrix-instagram`) instead of the Messenger `Dockerfile`. A new `matrix_bridge_mautrix_meta_instagram_container_binary_path` variable points the systemd service at the right binary for self-built and prebuilt images.
- Rename the systemd service templates to role-specific file names.
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/5510
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Draupnir stores each protection's settings as custom state events in the
management room, which requires a power level of 50 by default. A bot left
at the default power level of 0 cannot save any protection settings.
The failure is silent and has an expensive symptom: the
PolicyChangeNotification protection cannot remember its notifications room,
so it creates a brand new one on every restart of the bot. With the
playbook restarting the bot on every installation run, this litters the
server with abandoned rooms (90 of them observed on one deployment over
6 months).
Zero Touch Deployment is unaffected, since the bot creates and owns the
management room itself.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hookshot's end-to-bridge encryption needs MSC2409 and MSC3202 enabled
on the homeserver, but the playbook only turned them on for Meowlnir.
Hookshot's encryption defaults to the playbook-wide
`matrix_bridges_encryption_enabled` toggle, so turning that on gave
Hookshot encryption against a Synapse which was not set up for it, and
Hookshot then struggled in encrypted rooms.
The playbook already wires up Hookshot's other encryption prerequisite
(Valkey) and fails the run when the Redis settings are missing, so
being only half-automatic here was the odd one out.
Synapse sends the extra data only to appservices which ask for it in
their registration file, so turning these on affects no other
component.
The documentation told Hookshot users to set the two Synapse variables
by hand, and argued they should be enabled deliberately. That
contradicted what we already do for Meowlnir, so it is gone; the note
now matches the Meowlnir one and points out that other homeserver
implementations still need arranging by hand.
Closes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/5506
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Meowlnir (https://github.com/maunium/meowlnir) is a Matrix moderation
bot which speaks the same policy-list protocol as Mjolnir and Draupnir,
but runs as an appservice and can override individual policies coming
from ban lists you do not control.
Bots and their management rooms live only in Meowlnir's own database —
nothing in its configuration file can declare one — so the role
provisions them through the management API from a declarative roster
(matrix_bot_meowlnir_bots_custom), applied under the
ensure-matrix-users-created tag. Management rooms may be declared or
created for you; bots and rooms no longer declared get pruned.
Wrapper scripts for driving the management API by hand are installed
to /matrix/meowlnir/bin.
Meowlnir re-runs its configuration upgrader in memory on every start,
so a literal `generate` value yields a new secret per restart. All
secrets are therefore rendered explicitly, validation rejects
`generate`, and the configuration directory is mounted read-only.
Draupnir and Meowlnir both want synapse-http-antispam, which the
playbook wires up to a single consumer. The wiring prefers Draupnir,
and both roles fail the run when each claims it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Jitsi migrated image publishing from Docker Hub to the GitHub Container
Registry, and the new registry carries no tag history, so this also moves
Jitsi to stable-11146-1. That release makes the containers rootless and runs
them with a read-only root filesystem.
`jitsi_jvb_container_colibri_ws_host_bind_port` goes away with it. Colibri
WebSocket support was removed upstream in favour of SCTP data channels, so
the Jitsi Video Bridge no longer serves HTTP at all and needs no
reverse-proxy routing.
Logging in does not involve approving a session from the LINE mobile app.
The bridge shows a PIN code, which needs to be entered into the LINE
mobile app instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Its upstream repository has been archived and has not seen a change
since March 2025. mautrix-linkedin now covers the same use case.
The bridge remains installable, but as with other deprecated and
unmaintained components, it is no longer listed in the README and has
been moved to the deprecated services section of the documentation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the question-and-answer troubleshooting format (which no other
documentation page uses) with regular subsections, and clarifies why the
request needs to come from a Chrome-based browser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The playbook exposes service metrics under a single endpoint
(`https://matrix.example.com/metrics/*`), controlled by
`matrix_metrics_exposure_enabled` and friends. synapse-usage-exporter was
not wired into this at all, so its metrics could only be scraped by the
integrated Prometheus and were unreachable for an external one.
The role had no metrics router whatsoever. Its only Traefik router (for
the `/report-usage-stats/push` ingestion endpoint) was gated on
`matrix_synapse_usage_exporter_container_labels_traefik_enabled`, which
defaulted to `matrix_synapse_usage_exporter_proxying_enabled`. Adding a
metrics router under that same switch would have meant that exposing
metrics also publishes the ingestion endpoint, so the master switch now
defaults to `true` and each router carries its own conditional.
Despite its generic name, `matrix_synapse_usage_exporter_proxying_enabled`
only ever concerned the endpoint that Synapse pushes usage statistics
**to**, never the metrics endpoint that Prometheus reads **from**. The
documentation now says so explicitly.
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/5467
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Since Ansible 2.21, forked workers call `setsid()` and thus lose the
controlling terminal, so SSH can no longer ask about unknown host keys or
prompt for the passphrase of an SSH key.
Reported in https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/5472
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The new image restores the ability of SSH to ask about unknown host keys
and to prompt for the passphrase of an SSH key, which Ansible 2.21 broke
by having forked workers run session-isolated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The playbook exposes service metrics under a single endpoint
(`https://matrix.example.com/metrics/*`), controlled by
`matrix_metrics_exposure_enabled` and friends. ntfy was not wired into
this at all, so enabling metrics exposure did not expose ntfy's metrics
and enabling Basic Authentication did not protect them.
We have ntfy serve its metrics on a dedicated port, instead of on its
regular HTTP port. Serving them on the regular HTTP port also makes them
reachable at `https://ntfy.example.com/metrics`, where nothing protects
them, as ntfy does not apply its own access-control rules to the metrics
endpoint.
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/5468
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some web servers (notably shared hosting) serve extensionless files
as plain text or offer them as downloads, which breaks stricter
clients such as Element X.
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/4763
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The /_tuwunel namespace carries Tuwunel's ad-hoc routes (server_version,
local_user_count) and its native OpenID Connect provider endpoints
(/_tuwunel/oidc/...), which a reverse proxy must route for OIDC login
to work. Add public (enabled by default, like the client API) and
internal Traefik label groups.
Tuwunel serves the Synapse administration API under /_synapse/admin,
but the role only routed /_matrix, so admin dashboards and moderation
bots could not reach it. Add public and internal Traefik label groups
mirroring the matrix-synapse role, and enable them automatically when
Ketesa, Element Admin, or Draupnir is installed.
Codifies the conventions the playbook and its documentation already follow:
language and Markdown style, placeholder values for examples, the structure
of component documentation pages, and the checklist for adding a new
component (including where a component should live: this playbook, an
external MASH role, or mash-playbook).
Fixes#3631
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Renames the matrix_mautrix_SERVICENAME_* placeholders in the common
mautrix bridges guide to matrix_bridge_mautrix_SERVICENAME_*, and fixes
prose in the matrix-bridge-steam role that referred to the old variable
prefix and to a misspelled (underscore-styled) systemd service name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part of adopting a uniform naming policy for bridge variables,
where the variable prefix matches the role directory name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part of adopting a uniform naming policy for bridge variables,
where the variable prefix matches the role directory name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part of adopting a uniform naming policy for bridge variables,
where the variable prefix matches the role directory name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part of adopting a uniform naming policy for bridge variables,
where the variable prefix matches the role directory name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part of adopting a uniform naming policy for bridge variables,
where the variable prefix matches the role directory name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part of adopting a uniform naming policy for bridge variables,
where the variable prefix matches the role directory name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part of adopting a uniform naming policy for bridge variables,
where the variable prefix matches the role directory name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part of adopting a uniform naming policy for bridge variables,
where the variable prefix matches the role directory name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slavi Pantaleev
Kim Brose
svierne
Suguru Hirahara
Aine
Co van Leeuwen
Jason Volk