Skip to content

T1 · Golden fixtures for the synthetic pipelines, per engine version

Built 2026-10-27 on autopilot/T1 · 11 of 11 scenarios pass on Nextflow 26.10.4, 25.10.9, 25.04.8 and Snakemake 9.27.1 · 44 golden fixtures, checked in CI · 3 fixture faults fixed · resume identity recorded for inbox #32 · rnaseq and sarek gate check not run, moved to the nightly corpus (inbox #47) · review-code approved · doc-review approve 14/14

T1 figure: the fixture loop. A scenario runs on the local executor, its Nextflow trace or Snakemake log is normalised, and one TSV per scenario is written under golden/<engine>-<version>/; CI re-runs the scenario and compares. The time-limit exit code differs per version: 143 on Nextflow 26.10.4, none (shown as -) on 25.10.9 and 25.04.8, 124 on Snakemake 9.27.1. A tool error in COMPARE gets 1 attempt on Nextflow and 3 on Snakemake, which retries by count. Resume identity: a Nextflow resume gets a new run name with the same session ID, in the same launch directory only; every Snakemake rerun gets a new random workflow_id and nothing links the runs, so Spectio records the parent link itself T1 figure: the fixture loop. A scenario runs on the local executor, its Nextflow trace or Snakemake log is normalised, and one TSV per scenario is written under golden/<engine>-<version>/; CI re-runs the scenario and compares. The time-limit exit code differs per version: 143 on Nextflow 26.10.4, none (shown as -) on 25.10.9 and 25.04.8, 124 on Snakemake 9.27.1. A tool error in COMPARE gets 1 attempt on Nextflow and 3 on Snakemake, which retries by count. Resume identity: a Nextflow resume gets a new run name with the same session ID, in the same launch directory only; every Snakemake rerun gets a new random workflow_id and nothing links the runs, so Spectio records the parent link itself

The exit codes in the figure come from the timeout_fatal fixtures. The resume identities come from the research note.

What changed

  • Golden fixtures, new: tests/pipelines/synthetic/golden/<engine>-<version>/<scenario>.tsv.
    • Four directories: nextflow-26.10.4, nextflow-25.10.9, nextflow-25.04.8 and snakemake-9.27.1.
    • 11 files each, one per PR scenario: 44 files, 512 lines.
    • Each file holds the exit code of every run, then the task outcomes with the columns run, process, status, exit, attempt, tag, count.
    • The format and the normalisation rules are in the research note. This page does not repeat them.
  • tests/pipelines/synthetic/run_scenarios.py:
    • --golden record writes the fixtures from a run; --golden check compares a run with them. On a difference, the scenario fails and golden.actual.tsv is written next to the run.
    • --golden check fails the whole run when the engine version does not read as X.Y.Z, when the version has no fixture directory, or when a PR scenario has no fixture. Only nightly scenarios may have none.
    • --golden refuses extra executor arguments: fixtures come from the local executor only.
    • nf_rows() reads the Nextflow trace after each run (the previous trace is deleted first). smk_rows() reads the Snakemake log.
    • The Snakemake align gate now stops with --until qc_report align_done, not --until align_done (finding 1 below).
  • tests/pipelines/synthetic/test_golden.py, new: 8 unit tests. They cover the normaliser on sample trace and log text, the target filter, the whole-token sample match (S1 does not match S10), the version and missing-directory errors, and one fixture per PR scenario for every engine version.
  • tests/pipelines/synthetic/nextflow/nextflow.config: the docker and k8s profiles use python:3.12 pinned by digest, not python:3.12-slim, because Nextflow needs ps in the container. The docker profile mounts the shared bin/ read-only (finding 3).
  • tests/pipelines/synthetic/scenarios.yaml: gate_align_then_resume now expects multiqc/general_stats.tsv after the first run. The tool_error_then_fixed_resume title says that Snakemake retries the tool error.
  • tests/pipelines/synthetic/snakemake/workflow/Snakefile: its header names the new align gate targets.
  • .github/workflows/pipelines-synthetic.yml (spec §11):
    • the Nextflow 26.10 leg moves from 26.10.3 to 26.10.4, the pin in dev/versions.env;
    • a new step runs test_golden.py;
    • the scenario step runs with --golden check;
    • golden.actual.tsv is uploaded with the logs.
  • Spec: dev-environment-and-testing.md:
    • §3: the rnaseq and sarek gate check is marked pending (inbox #47);
    • §10.1: the Snakemake gate targets, the time-limit exit code per engine version, and the Snakemake retry of a tool error;
    • §11: the pipelines-synthetic.yml row (26.10.4, --golden check, nf-prov 1.8.x with 26.10.x).
  • Research note, new: Spike T1: fixture workflows, golden fixtures and resume identity. It holds the per-version table of failure modes, the four findings and the resume-identity sources. It is in the nav under Research.
  • Inbox: #47 and #48 are new. #42 has an update line: the matrix is on 26.10.4, and no branch-protection change is needed, because of the pipeline checks, only the aggregate ci-ok is required (autopilot §8).
  • Review record: docs/reviews/code/T1.md.

Why

Prompt: next-phase.md · T1. It asks for three things:

  1. prove each failure mode of the synthetic pipelines on the local executor, and record the engine outputs as golden fixtures named with the tool version;
  2. run the gate stop-and-resume check on the nf-core/rnaseq and nf-core/sarek test profiles;
  3. record, for inbox #32, what each engine does with the run's identity on a resume.

Items 1 and 3 are done. Item 2 is not: see Findings. No decision-log entry: the step decided no behaviour. The one rule change it found (#48) waits for /apply-findings.

How to test it

The step test, from the repository root:

python3 -m unittest tests/pipelines/synthetic/test_golden.py && \
  python3 tests/pipelines/synthetic/run_scenarios.py --engine snakemake --golden check

For a Nextflow version, add --engine nextflow --nextflow <binary>, for example --nextflow nextflow-25.10.9. The engines come from dev/bootstrap.sh at the pins of dev/versions.env. To see a check fail, change one count in a fixture (for example ALIGN 24 → 23 in golden/<engine>-<version>/happy.tsv): the scenario fails and golden.actual.tsv shows the real outcome.

Evidence

Unit tests, run for this page on 828e57f:

$ python3 -m unittest -v tests/pipelines/synthetic/test_golden.py
test_fixtures_cover_every_pr_scenario (…FixturesTest…) ... ok
test_an_unreadable_version_is_rejected (…GoldenDirTest…) ... ok
test_check_fails_when_the_version_has_no_fixtures (…GoldenDirTest…) ... ok
test_first_attempt_successes_are_counted_and_the_rest_listed (…NormaliseTest…) ... ok
test_nextflow_trace_rows (…NormaliseTest…) ... ok
test_sample_matches_a_whole_tag_token (…NormaliseTest…) ... ok
test_snakemake_log_rows (…NormaliseTest…) ... ok
test_target_filter_keeps_only_the_injected_task (…NormaliseTest…) ... ok
Ran 8 tests in 0.015s
OK

The step test on Snakemake 9.27.1, run again for this page on 828e57f in a fresh workdir:

$ (time python3 tests/pipelines/synthetic/run_scenarios.py --engine snakemake --workdir /tmp/t1-docs --golden check) > smk-check.log 2>&1; echo "exit=$?" >> smk-check.log
$ cat smk-check.log
PASS  snakemake happy                              4.3s
PASS  snakemake gate_qc_then_resume                7.1s
PASS  snakemake gate_align_then_resume             6.8s
PASS  snakemake exclude_sample_and_resume          7.4s
PASS  snakemake flaky_retry                        4.5s
PASS  snakemake oom_once_retry                     4.7s
PASS  snakemake oom_fatal                          3.3s
PASS  snakemake timeout_fatal                     26.1s
PASS  snakemake tool_error_then_fixed_resume       7.0s
PASS  snakemake dynamic_fanout                     5.0s
PASS  snakemake fanout_medium                     11.6s

real    1m29.165s
exit=0

Nextflow 26.10.4, after the review fixes. The code-simplifier re-ran the suite on 7202d9f (nextflow -version: 26.10.4 build 14435). Its report.json, read for this page:

$ python3 -c "import json; r=json.load(open('/tmp/t1n/nextflow/report.json')); print(sum(x['ok'] for x in r), '/', len(r))"
11 / 11

All four engine versions. In the step, each suite ran twice: --golden record, then --golden check in a fresh workdir. All 11 PR scenarios passed both times on Nextflow 26.10.4, 25.10.9 and 25.04.8 and on Snakemake 9.27.1, and every check exited 0. The commands are in the note, Setup. The coordinator later re-ran --engine snakemake --golden check on its own: 11/11 PASS, exit 0. A fixture altered on purpose (happy, ALIGN 24 → 23) failed the check; the restored one passed.

Time-limit exit codes, the one outcome that differs between engine versions (timeout_fatal, injected in QC_STATS S05):

Fixture Run exit QC_STATS exit, attempts 1 to 3
nextflow-26.10.4 1 143, 143, 143
nextflow-25.10.9 1 -, -, - (no exit code)
nextflow-25.04.8 1 -, -, - (no exit code)
snakemake-9.27.1 1 124, 124, 124

A tool error is retried on Snakemake only (tool_error_then_fixed_resume, COMPARE treatA, run 1):

$ grep FAILED tests/pipelines/synthetic/golden/nextflow-26.10.4/tool_error_then_fixed_resume.tsv
1   COMPARE FAILED  1   1   treatA_vs_ctrl  1
$ grep FAILED tests/pipelines/synthetic/golden/snakemake-9.27.1/tool_error_then_fixed_resume.tsv
1   COMPARE FAILED  1   1   group=treatA    1
1   COMPARE FAILED  1   2   group=treatA    1
1   COMPARE FAILED  1   3   group=treatA    1

Resume identity (inbox #32), observed on Nextflow 26.10.4 and 25.10.9 and on Snakemake 9.27.1 with gate_qc_then_resume. Sources for each row are in the note.

Engine Run identity on a resume Launch directory
Nextflow a new run name (sharp_meninsky, then dreamy_mclean on 26.10.4) and the same session ID (2816c414-… on both rows of nextflow log). Reusing a run name fails with exit 1. must be the same: from another directory, -resume <session-id> fails with "Can't find a run with the specified id", exit 1
Snakemake a new random workflow_id on every invocation (486ceb8b-…, then 8e78c667-…). Nothing links it to the previous one. the only link between runs: its outputs and .snakemake/metadata/ decide what reruns

Docs gate, on 828e57f before this page: python3 tools/docs/check.py gave docs gate: 0 failure(s), and mkdocs build --strict exited 0. The results on this page's commit are in the PR checks.

Review (the review record):

CODE  step=T1  review=approve b/M/m=0/0/4  security=approve b/M/m=0/0/5  rounds=2  simplify=nochange -0/+0  tests=pass
  • code-reviewer: changes in round 1 (1 blocker, 2 majors, 6 minors, on e2064ff), then approve in round 2 (0/0/4, on 7202d9f).
  • security-reviewer: approve in round 1 (0/0/7), and approve on the fix delta in round 2.
  • The round-1 fixes are in 7202d9f. CR-10 (the nf-prov row of §11) is fixed in 828e57f.
  • The round-1 blocker, CR-01, was that the rnaseq and sarek check was neither run nor tracked. It became a minor once the deferral was made explicit in inbox #47 and in dev-env §3.
  • code-simplifier: no change. It re-ran Snakemake and Nextflow 26.10.4, 11/11 each.

Figures and media

  • t1-golden-fixtures sketch, light and dark, at the top of this page. Drawn by diagram-artist: python3 tools/diagrams/t1_golden_fixtures.py.
  • No cast or screenshot, an exemption from §3.1: run_scenarios.py is a test harness for contributors, not a user CLI flow, the step adds no screen, and How to test it shows both the passing and the failing check.

Findings and open questions

Findings. Four, with their causes and sources, are in the note, Findings and fixes:

  1. The Snakemake align gate skipped the qc stage, because --until runs only the target's own dependencies. Fixed in the harness. The generation rule belongs to checking-scenarios §12.3, which needs a change: inbox #48. Nothing is assumed; §12.3 is unchanged until /apply-findings.
  2. Snakemake --retries restarts every failed job, a tool error included; Nextflow's errorStrategy here (nextflow.config line 17) retries exit codes 137, 140, 143 and Integer.MAX_VALUE, and any failure when params.fail_mode == 'flaky'. So flaky_retry (exit 1) is retried on Nextflow, and the exit-1 tool error is not. Spec §10.1 is corrected.
  3. The docker profile could not run any task (ps missing in -slim, and a dangling link to bin/). Fixed; oom_real then gave exit 137 under Docker's memory limit, its first proof.
  4. Nextflow 25.04.8 and 25.10.9 report no exit code on a time limit; only 26.10.4 reports 143. Spec §10.1 now gives the code per version.

For Thomas:

  1. The rnaseq and sarek gate check moves to the nightly corpus (inbox #47) (assumed — pending Thomas). T1 did not run it. Docker works in the session; the blocker is cost: on CI, rnaseq used about 23 GB of disk and 18 to 23 min per full run, against 26 GB free in the session. Proposed: a per-entry gate key in run_corpus.py, used by the nightly rnaseq and sarek entries. CH-016 stays open until its first nightly run.
  2. Gate targets on Snakemake (inbox #48). Recommended: a gate after stage N passes --until with the last targets of every stage up to N.
  3. Inbox #32, resume. The observations above fit option B, a child run per resume: Spectio records the Nextflow run name and session ID, and on Snakemake the parent link and the shared directory. The decision stays with Thomas (inbox #32).

Open review findings (review record):

  • CR-08: two parsers read the same engine outputs with different rules. It stays open: merging them changes the Snakemake exit codes.
  • CR-07, SEC-T1-01, SEC-T1-02, SEC-T1-03 and SEC-T1-07 are fixed since 2026-10-28 (commits in the review record):
    • CR-07: smk_rows gives each failed job the exit code of its own shell command, so two jobs that fail at once keep their own codes (unit test test_two_jobs_failing_at_once_keep_their_own_exit_codes).
    • SEC-T1-01: pipelines-synthetic.yml passes the matrix values to run: through env:.
    • SEC-T1-02: the docker profile runs tasks as the invoking user (-u $(id -u):$(id -g)) and quotes the bin/ mount path.
    • SEC-T1-03: the python:3.12 digest is set once, in params.python_image, with its pin date and update command.
    • SEC-T1-07: the workflow pins pyyaml==6.0.3, the same pin as claude-config.yml.
  • SEC-T1-06 (the harness deleted and reused the predictable shared workdir /tmp/spectio-scenarios) is fixed since 2026-10-28: without --workdir, both runners make a private directory, and an explicit --workdir is checked (review record).

Open items of the step are in the note, Open items: job arrays in the slurm profile (T3), fanout_stress (nightly, no fixture), a CI leg for Nextflow 25.04.8, oom_real on Snakemake (T3, T4), and the k8s profile's bin/ mount (T4).