T7 · The real-pipeline corpus runs green in CI

Built 2026-10-27 on autopilot/T7 · PR tier 7 of 7 green, nightly 15 of 18, both on fc4101c · three causes found, none in Spectio's code · 6 catalog entries now manual · pipelines-real gates ci-ok again, PR tier cut to 2 light entries · review-code approved · doc-review approve 11/14

Every number in the figure comes from the research note: the runner measurements and the tiers of corpus.yaml.

What changed

• .github/workflows/pipelines-real.yml (spec §11). Each corpus job now:
  • frees runner disk first, by removing toolchains it never uses and the preloaded Docker images, with plain rm and no third-party action. This step runs only on GitHub-hosted runners (if: runner.environment == 'github-hosted');
  • samples free memory and disk every 5 s, and writes the start and minimum values to the job summary and to a resources.json artifact;
  • in the nf-core job, prints the last 80 lines of run.log when the run fails or is cancelled, and stays green when there is no log;
  • lets a catalog entry name its own snakefile.
  • Event values reach the shell through env: (NAME, LIMITS), not through ${{ }} inside run:.
  • The matrix job runs the unit tests of run_corpus.py first, then builds the matrix with run_corpus.py --matrix <tier>, which validates the corpus before printing it.
  • astral-sh/setup-uv runs with enable-cache: false, so no shared cache is saved from a job that runs third-party code.
  • In the Snakemake job, the workflow's own checkout replaces the workspace. A sparse checkout of runner_probe.sh comes first, and the script is copied to $RUNNER_TEMP.
• tests/pipelines/real/runner_probe.sh, new: the free, sample and summary steps, shared by both jobs (runner_probe.sh free | sample | summary NAME). summary exits cleanly with "no samples" when the job stopped before sampling started.
• tests/pipelines/real/run_corpus.py:
  • a clone() helper also accepts a full commit SHA, for repositories with no release tag. If a clone by SHA fails, it removes the half-made checkout before re-raising;
  • per-entry env and args;
  • run.log is streamed to disk, so a run killed by a full disk or a timeout still leaves its log;
  • validate() runs on every invocation. Every ref must be a release tag or a 40-hex commit SHA, never a branch. repo, directory, snakefile and each args token must be plain tokens. An error names the entry;
  • --matrix <tier> prints the CI matrix lines for GITHUB_OUTPUT. The workflow's inline Python is gone.
• tests/pipelines/real/test_run_corpus.py, new: 5 unit tests: clone cleanup, validation of the current corpus, branch refs and unsafe values rejected, and matrix names.
• tests/pipelines/real/corpus.yaml:
  • viralrecon 3.0.0 and methylseq 4.2.0 run with NXF_SYNTAX_PARSER=v1;
  • ampliseq 3.0.0 gets --config_profile_contact none --config_profile_url none;
  • dna-seq-varlociraptor gets its upstream --configfile, single-cell-rna-seq its root Snakefile;
  • chipseq is pinned to commit 1345df8 of its default branch, master (it was tag: main, which does not exist);
  • rna-seq-star-deseq2 and rna-seq-kallisto-sleuth get --retries 2 and network: [ensembl];
  • six catalog entries move to manual, each with a note that says why;
  • the runner_limits comment now gives the measured values; private stays 2 CPU / 6 GB / 2 h.
• .github/workflows/ci.yml: pipelines-real is back in ci-ok's needs (ea0f6bf). D4 had left it out while the corpus was red (D4).
• Spec: dev-environment-and-testing.md:
  • §10.2: the tier table with a manual column, the new optional keys, corpus validation, the parser and runner constraints, and the re-run rule for entries tagged network:;
  • §11: ci.yml and pipelines-real.yml as built.
• Research note: Real-pipeline corpus on GitHub runners, T7 results. It holds the runner measurements, the cause and decision for every failing entry, and the per-entry durations, peak memory and disk. This page does not repeat those tables.
• F3: its sentence on chipseq now gives the pinned commit.

Why

pipelines-real.yml was red: 18 of 23 jobs failed in the nightly run on main (run 40463254070), and 5 of 9 in the PR tier on weber8thomas/spectio#16 (run 40535210648). While it was red, it could not gate ci-ok.

Prompt: next-phase.md · T7. For each failure, the prompt asks for one of three decisions: a runner limit, an upstream issue, or a corpus bug. No new decision-log entry; the gate design is D4's (ci.yml, inbox #26).

Three causes, none in Spectio's code (details in the note, §2):

Cause	Entries	Decision
Runner disk: No space left on device	9 failures across 7 nf-core entries (rnaseq and molkart failed in both runs)	runner limit: free disk first, 15 GB → about 46 GB
Strict config parser rejects an include missing from the tag	viralrecon 3.0.0, methylseq 4.2.0	corpus fix: NXF_SYNTAX_PARSER=v1 (nf-core/viralrecon#606, nf-core/methylseq#607)
Upstream breakage in catalog workflows (conda pins, a dead URL, a missing package, Ensembl timeouts)	dna-seq-gatk-variant-calling, chipseq, cyrcular-calling, ONT-AmpSeq, rna-seq-kallisto-sleuth	upstream: moved to manual, issue linked where one exists

Three corpus bugs were also fixed: chipseq's tag: main, varlociraptor's missing --configfile, and single-cell-rna-seq's Snakefile path. ampliseq needed two arguments under Nextflow 26.10's typed params.

Evidence

The two runs below were taken on fc4101c, before the review fixes (1a14299 and later). They show the corpus fixes, not the refactored workflow.

PR tier, after the corpus fixes. Run 40555936925 (workflow_dispatch on autopilot/T7, tier pr, head fc4101c), read with gh run view on 2026-10-27 (output wrapped at job boundaries):

$ gh run view 40555936925 -R weber8thomas/spectio --json conclusion,headSha,event,jobs \
    --jq '{c:.conclusion,sha:.headSha[0:7],ev:.event,jobs:[.jobs[]|"\(.name)=\(.conclusion)"]}'
{"c":"success","ev":"workflow_dispatch","jobs":["matrix=success",
 "nf-core/molkart 1.2.0=success","nf-core/demo 1.2.0=success","nf-core/rnaseq 3.27.0=success",
 "nf-core/viralrecon 3.0.0=success","nf-core/methylseq 4.2.0=success",
 "MPUSP/snakemake-ms-proteomics v1.1.1=success","snakemake-workflows/rna-seq-star-deseq2 v3.1.2=success"],"sha":"fc4101c"}

7 of 7 entries pass. The longest are viralrecon (41 min) and rnaseq (23 min), job time including disk freeing.

Nightly tier. Run 40562495868 (tier nightly, head fc4101c):

$ gh run view 40562495868 -R weber8thomas/spectio --json conclusion,headSha,event,jobs \
    --jq '{c:.conclusion,sha:.headSha[0:7],ev:.event,jobs:[.jobs[]|"\(.name)=\(.conclusion)"]}'
{"c":"failure","ev":"workflow_dispatch","jobs":["matrix=success",
 "snakemake-workflows/chipseq 1345df8a53e59772486dcd9a6dde4437a97ae6ed=failure",
 "snakemake-workflows/dna-seq-varlociraptor v6.11.3=success",
 "snakemake-workflows/rna-seq-kallisto-sleuth v4.1.1=failure",
 "nf-core/fetchngs 1.13.0=success","nf-core/sarek 3.10.0=success","nf-core/taxprofiler 2.0.1=success",
 "nf-core/viralrecon 3.0.0=success","nf-core/mhcquant 3.3.0=success","nf-core/demo 1.2.0=success",
 "MPUSP/snakemake-ms-proteomics v1.1.1=success","nf-core/mag 5.5.0=success",
 "snakemake-workflows/rna-seq-star-deseq2 v3.1.2=failure","nf-core/scrnaseq 4.2.0=success",
 "nf-core/rnaseq 3.27.0=success","nf-core/ampliseq 3.0.0=success","nf-core/molkart 1.2.0=success",
 "nf-core/methylseq 4.2.0=success","nf-core/differentialabundance 2.0.0=success"],"sha":"fc4101c"}

• All 13 nf-core entries pass. ampliseq takes 72 min of the 120 min job timeout, mag about 50 min.
• Snakemake: ms-proteomics and dna-seq-varlociraptor pass.
• chipseq failed because cutadapt 2.10 can no longer be solved by conda. rna-seq-kallisto-sleuth failed on an Ensembl archive timeout. Both moved to manual in 09f9871, after this run.
• rna-seq-star-deseq2 failed on a conda-forge 502 Bad Gateway. It keeps --retries 2 and now runs nightly (inbox #45).

Tiers in the corpus file, after inbox #45:

$ uv run --with pyyaml python tests/pipelines/real/run_corpus.py --list | awk '$1=="snakemake"'
snakemake rna-seq-star-deseq2              v3.1.2   nightly  bulk-groups
snakemake dna-seq-gatk-variant-calling     v2.1.1   manual   variants
snakemake rna-seq-kallisto-sleuth          v4.1.1   manual   bulk-groups
snakemake dna-seq-varlociraptor            v6.11.3  nightly  variants-pairs
snakemake single-cell-rna-seq              v1.1.0   manual   cells
snakemake chipseq                          1345df8a manual   bulk-samples
snakemake cyrcular-calling                 v2.1.1   manual   long-reads
snakemake ONT-AmpSeq                       v1.1.2   manual   amplicon-taxa
snakemake snakemake-ms-proteomics          v1.1.1   pr       ms-runs

nf-core: demo in pr; rnaseq, viralrecon, methylseq and molkart moved to nightly, which now holds 12; mcmicro manual.

Runner measurements (private repository, ubuntu-latest resolved to the ubuntu-26.04 image): 2 CPUs, MemTotal 7.75 GB, 6.84 to 6.96 GB available before a run, and 15 GB free on / before freeing. Nextflow's local executor sees 7.7 GB. Sources and the per-entry peaks are in the note, §1 and §3.

The gate. ea0f6bf puts pipelines-real back into ci-ok's needs:

-    needs: [changes, security, pipelines-synthetic]
+    needs: [changes, security, pipelines-synthetic, pipelines-real]

After the review fixes. A run on 7263253 was cancelled before any pipeline finished (run 40575395924, ci, pull_request). It still exercised the new steps on the GitHub runner. The pipelines-real / matrix job succeeded, so the corpus validation passed. In both Snakemake jobs, "Copy the runner probe out of the workspace", "Free runner disk", "Sample free memory and disk every 5 s" and "Memory and disk summary" succeeded. The unit-test step came later (0fe5196), so this run did not include it.

$ gh run view 40575395924 -R weber8thomas/spectio --json jobs \
    --jq '.jobs[]|select(.name|startswith("pipelines-real"))|"\(.name)=\(.conclusion)"'
pipelines-real / matrix=success
pipelines-real / MPUSP/snakemake-ms-proteomics v1.1.1=cancelled
pipelines-real / nf-core/molkart 1.2.0=cancelled
pipelines-real / snakemake-workflows/rna-seq-star-deseq2 v3.1.2=cancelled
pipelines-real / nf-core/methylseq 4.2.0=cancelled
pipelines-real / nf-core/demo 1.2.0=cancelled
pipelines-real / nf-core/rnaseq 3.27.0=cancelled
pipelines-real / nf-core/viralrecon 3.0.0=cancelled

Local checks after the light-tier change, with the commands the review record names:

$ python3 -m unittest tests/pipelines/real/test_run_corpus.py
Ran 5 tests in 0.017s
OK
$ python3 tests/pipelines/real/run_corpus.py --matrix pr | cut -c1-60
nfcore=[{"name": "demo", "version": "1.2.0", "min_nf": "25.1
smk=[{"repo": "MPUSP/snakemake-ms-proteomics", "tag": "v1.1.

PR-tier result on the PR head: see weber8thomas/spectio#19. CI runs there on the PR head.

Not touched: the synthetic suite. The step changes no file under tests/pipelines/synthetic/ and does not change pipelines-synthetic.yml.

Review (the review record):

CODE-REVIEW  verdict=approve  blockers=0  majors=0  minors=2  simplify=0
SECURITY-REVIEW  verdict=approve  blockers=0  majors=0  minors=6

• code-reviewer: changes in round 1 (0/1/5, on ea0f6bf), then approve in round 2 (0/0/2, on 7263253).
• security-reviewer: approve in round 1 (0/0/6).
• The round-1 fixes are in 1a14299. The two round-2 minors are fixed in 0fe5196 (CR-7, unit tests in CI) and in this refresh of the page (CR-8).
• code-simplifier: no change.
• SEC-T7-01 (tag pins in a gating check) and SEC-T7-04 (unpinned conda setup inside snakemake-github-action) are deferred to inbox #46.

Figures and media

• t7-corpus-ci sketch, light and dark, at the top of this page: python3 tools/diagrams/t7_corpus_ci.py.
• No cast or screenshot: the step adds no CLI flow and no screen.

Findings and open questions

For Thomas:

1. A light PR tier (answered). The PR tier ran 7 entries in about 40 min on 2-CPU runners. On 2026-10-27, three jobs of its heavy entries failed on external services. Two were nightly failures of rna-seq-star-deseq2 (an Ensembl timeout, then a conda-forge 502). The third was a Docker registry reset that hit rnaseq in this PR's own run (run 40578574693). Thomas chose to keep only nf-core/demo and the ms-proteomics dry run in pr, about 5 min (inbox #45). The other five run nightly, which does not gate PRs.
2. Nightly failures are not watched yet. Nobody is notified when the nightly goes red. The runs only show up in the Actions tab.

3. Public runners are not measured. The repository is private, so runner_limits.public (4 CPU / 15 GB) stays [to be validated] until it goes public. Resolved 2026-10-28: the repository went public that day, and nightly run 40649538349 measured the public runners (4 CPU, MemTotal 16366644 kB, 14.66 GB available at start, highest peak 8.35 GB from viralrecon). The public limits are now 4 CPU / 13 GB (dev-env §10.2; review: T7-retry).

4. Nextflow pin. pipelines-real.yml still pins nf-core/setup-nextflow to 26.10.3, and corpus.yaml says 26.10.3, while dev/versions.env pins 26.10.4. Left as is; it belongs to inbox #42. Resolved 2026-10-28: pipelines-real.yml now reads NEXTFLOW_VERSION from dev/versions.env (26.10.4), and corpus.yaml no longer repeats the current version (SPK-T7-03, round "spikes"; review: T7-pin).

5. Corpus pins and the Snakemake action (SEC-T7-01, SEC-T7-04). Pin every entry by commit as well as tag, and replace snakemake-github-action with pinned conda setup steps? Inbox #46; nothing is assumed.

Open items (dropping NXF_SYNTAX_PARSER=v1 once fixed upstream, ampliseq's 72 min, viralrecon's FREYJA_BOOT memory) are in the note: §2 for the override, §1 and §3 for the measurements.