Skip to content

Real-pipeline corpus on GitHub runners — T7 results (2026-10-27)

Question. Why was pipelines-real.yml red (18 of 23 jobs failed in the nightly run on main, run 40463254070; 5 of 9 in the PR tier on weber8thomas/spectio#16, run 40535210648)? What does it take to make the PR tier pass, and what do the runners actually leave to the pipelines?

Answer. Three causes, none in Spectio's code, plus a few corpus fixes:

  1. Runner disk. 9 failures across 7 nf-core entries (rnaseq and molkart failed in both runs) were No space left on device. Freeing unused toolchains and Docker images before each job raises free space on / from 15 GB to 46 GB, and every nf-core entry now passes.
  2. Strict config parser. viralrecon 3.0.0 and methylseq 4.2.0 include config files that their release tags do not contain. Nextflow 26 rejects them, so they run with NXF_SYNTAX_PARSER=v1.
  3. Upstream breakage in catalog workflows. Six Snakemake entries move to manual (details in §2):
    • dna-seq-gatk-variant-calling and chipseq: old conda pins (samtools 1.10, cutadapt 2.10) no longer solve;
    • cyrcular-calling: its RepeatMasker download URL is dead (404);
    • ONT-AmpSeq: its conda env lacks pandas;
    • rna-seq-kallisto-sleuth: Ensembl archive timeouts;
    • single-cell-rna-seq: unmaintained since 2019, never run.

Corpus fixes: chipseq's tag: main, varlociraptor's missing --configfile, single-cell-rna-seq's Snakefile path, and two arguments ampliseq needs under Nextflow 26.10's typed params (§2).

After the fixes, the PR tier is green (7 of 7, run 40555936925). In the nightly tier (run 40562495868), all 13 nf-core entries passed. Of the Snakemake entries, 2 passed and 3 failed: two now sit in manual, and one, rna-seq-star-deseq2, hit a conda-forge 502 error. Afterwards, the PR tier was cut to two light entries; the five heavier ones run nightly (§3). The corpus file is tests/pipelines/real/corpus.yaml; the owning spec is dev environment §10.2.

1. Runner measurements

Runner: ubuntu-latest, which resolved to the ubuntu-26.04 image (version 20261025.179.1, from the job log's "Runner Image" group), for this private repository. The figures below come from the "Free runner disk" and "Sample free memory and disk" steps of pipelines-real.yml and from Nextflow's .nextflow.log. They are identical across the 25 jobs of both runs, to within 0.1 GB.

Quantity Value Source
CPUs 2 nproc
MemTotal 7.75 GB (8 122 096 kB) /proc/meminfo
MemAvailable before the run 6.84–6.96 GB /proc/meminfo, first sample
Swap 3 GB SwapTotal
Memory Nextflow's local executor allows 7.7 GB .nextflow.log: Creating local task monitor for executor 'local' > cpus=2; memory=7.7 GB
Free disk on / at job start 15 GB of 72 GB df -h /
Free disk on / after freeing 45.5–47 GB df -h /
Time spent freeing 115–233 s step duration, run 40555936925

Why 7.7 GB and not the free memory. The local executor takes its memory capacity from OperatingSystemMXBean.getTotalPhysicalMemorySize() unless executor.memory is set (LocalPollingMonitor.configMem, nextflow v26.10.3, lines 119-120, read 2026-10-27). So it accepts a 7 GB task even when only 6.9 GB is free. The earlier comment in corpus.yaml ("avail: 5.8 GB") came from this cloud session, not from a runner.

runner_limits.private stays at 2 CPU / 6 GB / 2 h. 6 GB is under the 6.9 GB that is free before a run, which leaves room for the Nextflow JVM and the Docker daemon. Because the executor sums task requests against 7.7 GB, two 6 GB tasks never run at the same time. viralrecon's FREYJA_BOOT reached a 6.7 GB peak RSS, above the 6 GB cap, and still finished. Whether Docker enforced any memory limit on that container is [to be validated].

runner_limits.public (4 CPU / 15 GB) is not measured. This repository is private, so no public runner ran. The value stays [to be validated].

Note, 2026-10-28: public runners measured. The repository went public on 2026-10-28. Nightly run 40649538349 on main ran on public runners: 4 CPU, MemTotal 16366644 kB (15.6 GB, against 7.75 GB above), 14.66 GB available at start (against 6.9 GB), and a highest run peak of 8.35 GB (viralrecon). runner_limits.public is now 4 CPU / 13 GB / 2 h (dev-env §10.2).

Disk use per job. The table in §3 gives each entry's minimum free disk. molkart dropped to 16.1 GB free and viralrecon to 12.3 GB, so each used about 30–34 GB. Neither would fit in the 15 GB the image leaves free. The freeing step removes:

  • /usr/local/lib/android, /usr/share/dotnet, /opt/ghc, /usr/local/.ghcup, /opt/hostedtoolcache/CodeQL, /usr/share/swift, /usr/local/share/boost, /usr/local/share/powershell, /usr/local/julia*, /opt/az and /opt/microsoft;
  • every preloaded Docker image (docker image prune -af).

The jobs use none of these. The step is plain rm, with no third-party action.

2. Failure analysis and decisions

Entry Symptom (run 40463254070 / 40535210648) Cause Decision
nf-core/rnaseq 3.27.0, molkart 1.2.0, differentialabundance 2.0.0, mag 5.5.0, mhcquant 3.3.0, taxprofiler 2.0.1, sarek 3.10.0 No space left on device (sarek: a fault occurred in a recent unsafe memory access operation, a JVM error that fits a full disk under a memory-mapped file [to be validated]) runner disk runner limit: free disk first; all pass
nf-core/methylseq 4.2.0 Invalid include source: conf/aws/batch/nextflow.config the tag has no conf/aws/ directory (git ls-tree of tag 4.2.0, commit 5aa5646); the strict parser rejects the include, the v1 parser does not corpus fix: env: {NXF_SYNTAX_PARSER: v1}. Upstream: nf-core/methylseq#607, closed on 2026-07-16 with a fix on dev that is not in 4.2.0 (comment of 2026-10-23); a later release should drop the override
nf-core/viralrecon 3.0.0 Invalid include source: conf/test_full_sispa.config the tag's conf/ has test_sispa.config but no test_full_sispa.config (nextflow.config:252) corpus fix: NXF_SYNTAX_PARSER=v1. Upstream: nf-core/viralrecon#606 reports the same error on 26.04.6, worked around the same way, closed by the reporter
nf-core/ampliseq 3.0.0 Parameter config_profile_contact is required but no value was provided main.nf:250-253 declares config_profile_contact: String in a typed params block, and nextflow.config:30 sets it to null. Nextflow 26.10 treats that as missing; the test profile sets only config_profile_name and config_profile_description. Reproduced locally with Nextflow 26.10.4 and -preview corpus fix: args: --config_profile_contact none --config_profile_url none. Whether ampliseq's own CI (26.04) passes is [to be validated]; no upstream issue was found
snakemake-workflows/dna-seq-gatk-variant-calling v2.1.1 LibMambaUnsatisfiableError on samtools ==1.10 (from bio/samtools/faidx wrapper 0.74.0): htslib 1.10 needs a libdeflate that conda-forge no longer offers alongside the other pins bit rot; v2.1.1 is the latest tag and equals main (git ls-remote, cffa77a) upstream: no issue found (search 2026-10-27). Moved to manual
snakemake-workflows/chipseq 1. tag: main: checkout failed, the default branch is master and there is no tag. 2. After the fix: LibMambaUnsatisfiableError on cutadapt ==2.10 (wrapper 0.64.0; xopen has no viable build) 1. corpus bug; 2. bit rot corpus fix: pinned to commit 1345df8 of master; run_corpus.py can clone a SHA. upstream: no issue found. Moved to manual
snakemake-workflows/dna-seq-varlociraptor v6.11.3 configfile config/config.yaml ... not present corpus bug: the tests live in .test/config-*; upstream CI passes --configfile .test/config-simple/config.yaml (.github/workflows/main.yml:52-62 at v6.11.3) corpus fix; passes in 31 min
snakemake-workflows/single-cell-rna-seq v1.1.0 Snakefile "workflow/Snakefile" not found corpus bug: the 2019 layout has a root Snakefile and Travis CI only. Its envs pin tensorflow =1.10, bioconductor-scran =1.10 (envs/*.yaml) corpus fix (snakefile: Snakefile), then moved to manual without a run: the pins are from 2019, and nf-core/scrnaseq covers the cells class
snakemake-workflows/cyrcular-calling v2.1.1 download_repeatmasker_annotation: 404 the RepeatMasker hg38 URL in .test/config/config.yaml:16 returns 404 (checked again 2026-10-27); unchanged on main (c7d792f) upstream: related to snakemake-workflows/cyrcular-calling#9 (open, "more generic repeat annotation retrieval"). Moved to manual
MathiasEskildsen/ONT-AmpSeq v1.1.2 fix_tax_blast: ImportError for numpy and pandas workflow/envs/generic.yml has only gzip and python=3.12.1, but the script imports pandas, which leaks in from Snakemake's own Python 3.14 environment upstream: MathiasEskildsen/ONT-AmpSeq#20 (open, same traceback). Moved to manual
snakemake-workflows/rna-seq-kallisto-sleuth v4.1.1 get_transcript_info: biomaRt Timeout was reached [jun2026.archive.ensembl.org], also after 2 retries an external service; it failed in both runs of 2026-10-27 upstream/external: related to snakemake-workflows/rna-seq-kallisto-sleuth#127 (open, Ensembl downloads). Moved to manual
snakemake-workflows/rna-seq-star-deseq2 v3.1.2 nightly on main: the same biomaRt timeout. T7 nightly: conda-forge 502 Bad Gateway transient external errors; it passed in both PR-tier runs (15:52 on #16, 18:00 here) moved to nightly (inbox #45), with --retries 2; network: [ensembl] records the dependency. When it fails on an external error, re-run that job once before investigating (dev environment §10.2)

3. Per-entry results

Run 40555936925 is the PR tier, run 40562495868 the nightly tier, both on autopilot/T7 at fc4101c. Each cell gives the job duration (including 2–4 min of disk freeing) and the pipeline's own duration (result.json seconds, nf-core only).

Peak memory. "Run peak" is MemAvailable before the run minus the lowest MemAvailable seen, sampled every 5 s. It covers Nextflow or Snakemake plus every container. "Task peak RSS" is the highest peak_rss in the Nextflow trace.

Disk. "Min free disk" is the lowest free space on / during the run.

Entry Tier (now) PR run Nightly run Run peak Task peak RSS Min free disk
nf-core/demo 1.2.0 pr pass, 6 min (119 s) pass, 4 min (116 s) 1.0–1.2 GB 0.9 GB MULTIQC 42.7 GB
nf-core/rnaseq 3.27.0 nightly pass, 23 min (1144 s) pass, 18 min (992 s) 2.1–2.4 GB 1.9 GB PICARD_MARKDUPLICATES 23.0 GB
nf-core/viralrecon 3.0.0 nightly pass, 41 min (2324 s) pass, 38 min (2061 s) 4.6–4.9 GB 6.7 GB FREYJA_BOOT 12.3 GB
nf-core/methylseq 4.2.0 nightly pass, 8 min (295 s) pass, 7 min (322 s) 1.3–1.5 GB 0.9 GB MULTIQC 39.3 GB
nf-core/molkart 1.2.0 nightly pass, 12 min (552 s) pass, 13 min (575 s) 1.2–1.7 GB 1.3 GB CELLPOSE 16.1 GB
nf-core/fetchngs 1.13.0 nightly — pass, 26 min (1503 s) 0.7 GB 0.1 GB 41.5 GB
nf-core/sarek 3.10.0 nightly — pass, 9 min (465 s) 3.6 GB 2.5 GB GATK4_MARKDUPLICATES 27.3 GB
nf-core/scrnaseq 4.2.0 nightly — pass, 16 min (715 s) 3.2 GB 2.0 GB STAR_ALIGN 37.1 GB
nf-core/mhcquant 3.3.0 nightly — pass, 23 min (1326 s) 2.5 GB 3.9 GB MS2RESCORE 26.3 GB
nf-core/differentialabundance 2.0.0 nightly — pass, 10 min (471 s) 1.8 GB 1.2 GB QUARTONOTEBOOK 29.2 GB
nf-core/mag 5.5.0 nightly — pass, 49 min (2859 s) 2.1 GB 1.1 GB SEMIBIN_SINGLEEASYBIN 20.2 GB
nf-core/ampliseq 3.0.0 nightly — pass, 72 min (4250 s) 5.7 GB 4.7 GB DADA2_TAXONOMY 18.1 GB
nf-core/taxprofiler 2.0.1 nightly — pass, 27 min (1509 s) 3.1 GB 2.4 GB FASTP_SINGLE 22.6 GB
rna-seq-star-deseq2 v3.1.2 nightly pass, 12 min fail, 4 min (conda-forge 502) 2.2 GB — 37.2 GB
snakemake-ms-proteomics v1.1.1 (dry-run) pr pass, 4 min pass, 4 min 1.8–2.1 GB — 43.7 GB
dna-seq-varlociraptor v6.11.3 nightly — pass, 31 min 1.9 GB — 26.6 GB
rna-seq-kallisto-sleuth v4.1.1 manual — fail, 22 min (Ensembl timeout) 4.5 GB — 31.1 GB
chipseq 1345df8 manual — fail, 5 min (cutadapt 2.10 unsolvable) 2.3 GB — 42.3 GB

rnaseq, viralrecon, methylseq, molkart and rna-seq-star-deseq2 ran in the PR tier in these runs. The PR tier's wall time was set by viralrecon, at about 40 min. Then the PR-tier run on 7a3fe4f (run 40578574693) lost rnaseq to a Docker registry error: docker: failed to copy: … connection reset by peer while pulling community.wave.seqera.io/library/bioconductor-dupradar:1.38.0--831da16eb40a64ab. It was the third failure from an external service that day. The other two were rna-seq-star-deseq2's Ensembl timeout and conda-forge 502. Thomas chose a light PR tier (inbox #45, 2026-10-27): demo and the ms-proteomics dry run, about 5 min. The other five run nightly.

ampliseq is the longest nightly entry: 72 min of the 120 min job timeout, with only 1.3 GB of memory left at its lowest point. It is the first candidate for manual if it grows.

4. Changes made in T7

The workflow, script and corpus changes are listed on the T7 Progress page. Two choices belong here:

  • One probe script. Both jobs call tests/pipelines/real/runner_probe.sh (free, sample, summary NAME). This replaces copies of the three steps in each job, which had already drifted apart, and avoids YAML anchors, whose support in zizmor is beta (zizmor usage docs).
  • Corpus validation. run_corpus.py --matrix TIER validates corpus.yaml first: refs must be release tags or 40-hex SHAs, so an entry cannot silently track a branch, and paths and args must be plain tokens, because these values end up in workflow inputs and command lines.

Not changed.

  • pipelines-real.yml still pins nf-core/setup-nextflow to 26.10.3, while dev/versions.env pins 26.10.4. Aligning them is inbox #42.
  • corpus.yaml versions.nextflow.current also says 26.10.3.
  • No required check was renamed.

Follow-ups (to the inbox).

  • SEC-T7-01: pin every corpus entry by commit SHA, kept current by Renovate.
  • SEC-T7-04: replace snakemake/snakemake-github-action, a composite action that wraps an unpinned setup-miniconda@v3 and the latest Miniforge.

5. Sources