From 31a9d21ca7f1ba9a8174272884befa076dde7a42 Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Thu, 27 Aug 2026 18:25:26 +0300 Subject: [PATCH] Write down what the first wave of Molecule scenarios taught us Eight scenarios in, the same few things keep deciding whether a scenario is worth having, and none of them were obvious beforehand: - Falsify every assertion, not just enough to see red. One control asserted that a component emitted no DEBUG records from a module and passed just as happily with that module set to `debug`, because it emits none on a first run either way. Green for the wrong reason, and only breaking it deliberately showed that. - `ActiveState == active` with `NRestarts == 0` can both hold while a component is completely broken, when it catches its errors and retries rather than exiting. Two of the eight behave that way. - Startup lines are the oldest in the journal, so tailing loses them. Grep, and strip ANSI first. - Parse rendered configuration and assert on structure, so a value under the wrong key cannot pass. - `molecule converge` on a running instance does not restart the container, so a falsification can pass for that reason alone. Also documents that a role's scenario obliges it to join the automerge list, and that prek enforces the two staying in step. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT --- docs/molecule-testing.md | 46 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/docs/molecule-testing.md b/docs/molecule-testing.md index 53bd9ebb5..57e88fd30 100644 --- a/docs/molecule-testing.md +++ b/docs/molecule-testing.md @@ -30,6 +30,16 @@ Molecule is deliberately **not** part of the `prek` hooks. A run is far too slow When the diff base cannot be determined (a new branch, a force push), it falls back to running every scenario, which errs toward testing too much rather than too little. `workflow_dispatch` accepts an optional role name. +## Automerge + +A role that has a scenario is listed in the Molecule automerge rule in `.github/renovate.json`, so +patch bumps of its component merge on their own once the scenario has passed on them. + +**Add your role to that list when you add its scenario.** `bin/check-molecule-automerge-list.py` +runs from prek and fails the commit if the list and the scenarios have drifted apart. The direction +that matters is a role staying in the list after losing its scenario, since its bumps would then +merge with nothing exercising them. + ## Writing a scenario Start from `roles/custom/matrix-alertmanager-receiver/molecule/default/` — it is the reference. Four things differ from a standalone role's scenario, all of them consequences of these roles living inside a playbook: @@ -70,6 +80,42 @@ Give the scenario values that differ from both the role's defaults and the compo Then try to break it. If a scenario cannot be made to fail by deliberately breaking the thing it checks, it is not testing that thing. +Falsify **every** assertion, not just enough of them to see the scenario go red. An assertion that +passes is not necessarily an assertion that works: one control here asserted that a component +emitted no DEBUG records from a particular module, and it passed just as happily with that module +set to `debug`, because the module emits none on a first run either way. It was green for the wrong +reason, and only breaking it deliberately exposed that. + +Two traps make a falsification pass when it should fail: + +- `molecule converge` against an already-running instance rewrites the configuration but only does + `state: started`, so the container keeps the old one. Full `molecule test` is unaffected - this + bites the local iterate-with-converge loop, which is where falsifications get run. +- The failure must land on the assertion you aimed at. If it fails at an earlier gate, you have + proved something about that gate instead. + +### Work out whether the component crashes or retries + +Some components exit when their configuration is wrong; others catch everything and retry forever. +For the second kind, `ActiveState == active` and `NRestarts == 0` **both stay true while the +component is completely broken** - matrix-reminder-bot and baibot both behave this way, retrying a +failed login or profile step indefinitely. There the unit assertions prove nothing on their own, and +something the component says about itself has to carry the scenario. + +Establish which kind yours is before deciding what the weight-bearing assertion is. + +### Reading the journal + +Grep the whole journal rather than tailing it. Startup lines are the **oldest** entries, and a +component that syncs can bury them under thousands of lines within a minute, so `--lines=N` loses +exactly what you were looking for. Strip ANSI escapes too - some components colour their output, and +a plain substring match against raw journal text then fails silently. + +### Assert against parsed documents + +Where a scenario reads a rendered configuration, parse it and assert on the structure rather than +matching substrings. A value landing under the wrong key cannot then pass. + ## Running more than one scenario at once `bin/molecule.sh` points `ANSIBLE_HOME` at `var/molecule-ansible-home//`, so each role gets