Skip to content

Spike t3-ci: T3 on a GitHub-hosted runner (2026-10-28)

Question. Can pipeline step T3 (mini SLURM with sind, next-phase.md) be done CI-driven on GitHub-hosted runners, as option (b) of inbox #51 proposes? A Claude Code cloud session cannot run sind itself: its cgroup layout is hybrid v1 (spike D2). In order:

  1. Does a sind cluster shaped like slurm-site (two partitions, slurmdbd) come up in a GitHub Actions job?
  2. Do the synthetic pipelines run on it through run_scenarios.py: Nextflow -profile slurm and Snakemake --workflow-profile profiles/slurm, happy and one failure scenario per engine?
  3. Can squeue --json, sacct --json and sinfo --json of a failed job be printed in the job log, size-limited and fenced, and extracted back as fixtures?
  4. If time allows: NODE_FAIL by a power cut; a Snakemake head submitted with sbatch; a maintenance reservation and an association limit as pending reasons.

Answer in one line. Yes, all four. The fourth and last run is green in 14 min 32 s of wall-clock time on a free runner. One loop costs about 15 minutes for the whole set, or about 3 to 6 minutes for a single scenario. The fixtures come back through the log byte-exact: each block's sha256 is checked. The spike also found 7 differences with the specs and the harness (below).

Step page: none. This is a spike for inbox #51, not T3: no step is marked done, and pipeline.yaml, docs/specs/ and the inbox are unchanged. Branch spike/t3-ci only. Time box: about 90 minutes, 4 push iterations.

Setup

Fact Value Evidence
Runner ubuntu-24.04, Ubuntu 24.04.5, kernel 6.17 / 7.0 (Azure), 4 CPUs, 15 983 MiB, uid 1001. GitHub documents 4 CPU and 16 GB for public repositories, 2 CPU and 8 GB for private ones (GitHub-hosted runners); this repository is public. step "Runner facts" of every run
cgroup v2, unified. /sys/fs/cgroup is cgroup2fs, mounted rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot. Controllers: cpuset cpu io memory hugetlb pids rdma misc dmem. Docker 28.0.4, cgroup v2, driver systemd. sind needs Docker ≥ 28 on cgroup v2 with nsdelegate (sind CI guide). same
Toolchain dev/bootstrap.sh --ci at the pins of dev/versions.env, with SPECTIO_HOME=/data/tools and UV_PYTHON_INSTALL_DIR=/data/tools/python: Nextflow 26.10.4, Snakemake 9.27.1 with snakemake-executor-plugin-slurm 2.8.0, sind 0.15.0, Python 3.13.14. Java is the runner's Temurin 21 (21.0.12.1 or 21.0.13 depending on the runner image; the patch level is not pinned, as in dev-env.yml), copied to /data/tools/jdk. step "Tools into /data"
Cluster sind 0.15.0, node image sind-node:v0.15.0 by digest, Slurm 26.05.5, data_parser/v0.0.45. spikes/t3-ci/slurm-site.yaml.tmpl: controller, db (MariaDB + slurmdbd), submitter, worker-[0-1] in partition htc (2 CPUs, 1.5 GiB each), worker-2 in bigmem (2 CPUs, 3 GiB). Accounts lab (default) and capped (MaxJobs=1), AccountingStorageEnforce=associations,limits, ConstrainRAMSpace=yes, SlurmdTimeout=45. step "Create the slurm-site cluster"
Data The runner directory /data is bind-mounted at the same path in every node (storage.dataStorage.type: hostPath), so the tools, the venv shebangs and the workdirs resolve identically on the runner, the submitter and the workers. The cluster user spx gets the runner's uid (1001), so files keep one owner on both sides. cluster.sh tools, cluster.sh up
Engines Run as spx on the submitter, as on a login node (dev environment §4.1), through the unchanged tests/pipelines/synthetic/run_scenarios.py. cluster.sh scenarios

Files (branch spike/t3-ci):

File What
.github/workflows/spike-t3-ci.yml One job, 60 min limit; triggers push to spike/t3-ci and workflow_dispatch; permissions: contents: read; no secrets; actions pinned by SHA (copied from dev-env.yml); no ${{ }} in run:.
spikes/t3-ci/cluster.sh One subcommand per workflow step: facts, tools, up, scenarios ENGINE IDS, failed-fixtures ENGINE, probes, heads, down.
spikes/t3-ci/slurm-site.yaml.tmpl The sind config above. Reusable as the start of the T3 slurm-site profile.
spikes/t3-ci/env.sh Sourced in the nodes: tools on PATH, JAVA_HOME, NXF_OPTS.
spikes/t3-ci/fixture.py Prints one Slurm JSON document as a fenced, size-limited block.
spikes/t3-ci/extract_fixtures.py Finds the blocks again in a downloaded job log and checks each sha256.

Runs

# Run Head Result Wall-clock (job) What changed after it
1 40661645177 0f388fb cancelled after 28 min — java: command not found in the nodes: $JAVA_HOME on the runner is a symlink into the tool cache, and cp -a copied the link. Snakemake happy passed but took 762.6 s (next section).
2 40665414538 faff83c failure 15 min 49 s Everything ran. The failed-job query found nothing (sacct -s FAILED -S <t>, finding 6); Snakemake oom_fatal failed one harness check (finding 1).
3 40667769713 616761e failure 17 min 45 s Failed-task fixtures for both engines. Same Snakemake harness check.
4 40670286528 9d6451a success 14 min 32 s The one known harness gap is accepted explicitly by cluster.sh (finding 1); every other check applies.

The commit that adds this note triggers a fifth run of the same workflow; it changes no spike file.

Time per step, runs 2 / 3 / 4 (seconds). Checkout and setup-java add about 5 s before them.

Step Run 2 Run 3 Run 4
Tools into /data (dev/bootstrap.sh --ci, images pre-pulled) 49 40 51
sind create cluster (6 nodes, accounts) 8 9 10
Nextflow happy / oom_fatal 75.8 / 35.9 73.3 / 44.1 72.0 / 52.0
Probes (OOM, timeout, pending reasons, NODE_FAIL) 98 91 99
Snakemake happy / oom_fatal 226.7 / 71.2 337.5 / 82.0 221.9 / 76.5
Heads in sbatch: Snakemake / Nextflow happy 241.9 / 75.5 257.3 / 78.3 196.7 / 71.8

Result

Item Works Evidence and limits
slurm-site-shaped cluster on the runner yes Partitions htc* (worker-[0-1]) and bigmem (worker-2), plus sind's own all. slurmdbd registered the cluster and sind created the associations (site\|capped\|spx\|1, site\|lab\|spx\|). Up in 8–10 s from pre-pulled images. Not built: partition gpu with fake GRES, the api node (slurm-rest), the 25.11 leg.
Nextflow -profile slurm, happy yes, 72–76 s 65 tasks; ALIGN as Slurm job arrays (e.g. 108_[0-3]). Unchanged harness checks pass.
Nextflow oom_fatal yes, 36–52 s Three QC_STATS (S05) jobs end FAILED 137:0, the trace has native_id = Slurm job id and exit 137 for attempts 1–3. Nextflow then cancels its pending array: CANCELLED by 1001 (finding 4).
Snakemake --workflow-profile profiles/slurm, happy yes, 222–338 s Needs --slurm-init-seconds-before-status-checks 5: with the plugin default of 40 s the same run took 762.6 s (finding 3).
Snakemake oom_fatal yes, except one harness check Three Slurm jobs FAILED 1:0, run exits non-zero, final_report.html absent, QC_STATS.failed=3. The check "exit code 137 seen" fails: the log says only "SLURM status is: 'FAILED'" (finding 1).
Failed-task fixtures (squeue --json --states=all, scontrol show job --json, sacct --json, sinfo --json) yes Printed right after each engine's scenarios. squeue --states=all still lists the finished job (within MinJobAge). 7–13 KB per job.
Real OOM (--mem=100M, 400 MiB touched) yes OUT_OF_MEMORY, exit_code.signal 125, status SIGNALED: this confirms on sind the "0:125 under SLURM" that schedulers-and-monitoring.md §3.5 marks [to be validated on sind].
Time limit (-t 1, sleep 600) yes Job TIMEOUT, exit 0:0; its batch step CANCELLED with signal 15 (TERM).
Association limit yes Second job of account capped pends with AssocMaxJobsLimit; after it ran, sacct still reports reason: AssocMaxJobsLimit on the COMPLETED record.
Maintenance reservation yes scontrol create reservation … Flags=MAINT,IGNORE_JOBS on worker-2: the bigmem job pends with state_reason ReqNodeNotAvail and state_description ReqNodeNotAvail, Reserved for maintenance (finding 5).
NODE_FAIL (sind power cut worker-2.site) yes, 65–74 s after the cut A --no-requeue job ends NODE_FAIL, exit 1:0 (status ERROR), its batch step CANCELLED. sinfo --json: worker-2 DOWN, NOT_RESPONDING, reason "Not responding". sind power on brings it back idle (ReturnToService=2). SlurmdTimeout=45 follows sind's node-failure advice.
Snakemake head submitted with sbatch yes, 3 of 3 runs sbatch --wait -p bigmem of the harness running Snakemake with the SLURM executor: happy passes, the head job ends COMPLETED 0:0. The plugin logs "You are running snakemake in a SLURM job context. This is not recommended…" and removes every SLURM_* variable before it submits (plugin v2.8.0 __init__.py:458-466, utils.py delete_slurm_environment). So the 0.0.1 head-job design works for this pipeline; the fallback of dev environment §15.3 is not needed here. Not tested: cancelling the head, a head at its walltime.
Nextflow head submitted with sbatch yes, 3 of 3 runs Same, happy in 72–78 s, head COMPLETED 0:0.

Fixtures through the log

  • fixture.py NAME --jobs IDS keeps only the listed jobs (by job_id or array job id), indents the JSON (short lines survive the log), caps it at 48 KiB by dropping list entries from the end (then sets _spike_truncated), and prints:

    ----- BEGIN SPECTIO FIXTURE name=sacct-probes bytes=40557 sha256=… -----
{ …indented JSON… }
----- END SPECTIO FIXTURE name=sacct-probes -----

  • extract_fixtures.py LOG OUT_DIR strips the timestamp GitHub puts on each log line, rebuilds each block, and writes it only if its sha256 matches. On the logs of runs 2, 3 and 4 (downloaded with gh api repos/{owner}/{repo}/actions/jobs/{id}/logs) every block came back intact: 3, 10 and 10 blocks, 7–41 KB each, none truncated.

  • Run 4 carries ten: nextflow-{squeue,scontrol-job,sacct}-failed-task, snakemake-{…}-failed-task, sinfo, sinfo-node-down, squeue-pending-reasons (three jobs: running, AssocMaxJobsLimit, ReqNodeNotAvail) and sacct-probes (OUT_OF_MEMORY, TIMEOUT, NODE_FAIL, the two pending-reason jobs).
  • For T3, the extraction would add the Slurm version to each file name (slurm-26.05.5/…), as the test-architect rule wants, and redact what is site- or run-specific (submit_line holds full paths, temp names and base64 resources).
  • No artifact and no write token are needed. The same blocks could also go to actions/upload-artifact for Thomas to download, which the GitHub tools of a cloud session cannot do.

Differences with the specs and the harness

  1. Snakemake on SLURM hides the task exit code. A task's Slurm job runs python -m snakemake … --executor slurm-jobstep, so a rule that exits 137 makes the job end FAILED 1:0 (steps batch and python both 1). The 137 is only in the task's Slurm log file. The engine log says "SLURM-job '224' failed, SLURM status is: 'FAILED'. Reason: Unknown." run_scenarios.py's Snakemake exit-code check therefore cannot pass on SLURM; cluster.sh accepts exactly that one error. For schedulers-and-monitoring.md §3.5, "Tool error: FAILED with an exit code" gives 1 for every Snakemake task, whatever the tool returned.
  2. The synthetic oom mode is not an OOM for Slurm. It exits 137 by itself, so Slurm records FAILED 137:0, not OUT_OF_MEMORY. Only oom_real (or the probe) produces the record §3.5 classifies as oom. The T3 scenarios need oom_real with ConstrainRAMSpace=yes to test classification.
  3. Snakemake SLURM status polling dominates small runs. Plugin 2.8.0 waits init_seconds_before_status_checks (default 40 s) before the first check and after every check that sees a job finish, up to 180 s when nothing finishes (__init__.py:244-251, 1611-1617). happy took 762.6 s with the default and 222–338 s with 5 s. profiles/slurm/config.yaml should set it for tests (a sind-specific profile, since a real site may not want 5 s polling).
  4. Engine cleanup looks like an external cancel. After a fatal failure, Nextflow cancels its own pending jobs; sacct shows CANCELLED by 1001, the submitting user's own uid. §3.5 maps CANCELLED by <uid> to cancelled_external "Cancelled by someone else": it must compare the uid with the run's user and the engine's cleanup moment.
  5. The pending reason in --json is split in two fields. squeue --json gives state_reason: "ReqNodeNotAvail" and state_description: "ReqNodeNotAvail, Reserved for maintenance". The text output (%r) shows the long form, which §3.5's table quotes. A JSON parser must read state_description to tell a maintenance window from other ReqNodeNotAvail cases. Also, sacct --json keeps the last pending reason on a finished job (COMPLETED, reason: AssocMaxJobsLimit), which is history, not a current state.
  6. sacct -s FAILED,… -S <time> returned nothing in run 2 while three FAILED jobs existed in the window; the same query without -s, filtered afterwards, found them in runs 3 and 4. Why is [to be validated] against the sacct state-filter semantics. The runner should filter states itself, not rely on -s.
  7. NODE_FAIL exit code. The job record carries return_code 1, status ERROR and the step CANCELLED: classification must use the state, not the exit code (§3.5 already says so; this is the first observation).

Verdict on option (b)

Feasible. Every T3 item that this spike touched runs on a free ubuntu-24.04 runner: the slurm-site shape with slurmdbd and limits, both engines through the unchanged harness, a failure scenario per engine, real OUT_OF_MEMORY, TIMEOUT, NODE_FAIL, the two pending reasons, both heads in sbatch, and fixtures out of the log with an integrity check.

Cost per loop.

  • Money: zero. Standard runners are free and unlimited on public repositories (GitHub-hosted runners). If the repository became private, a 2-CPU / 8 GB runner would need a smaller cluster, and minutes would count against the plan [to be validated].
  • Time: a fixed 1.5 min per loop (job start, bootstrap.sh --ci 40–51 s, cluster 8–10 s), plus the scenarios: about 1–2 min per Nextflow scenario and 1.5–6 min per Snakemake scenario. The whole set here is 14.5–18 min. One focused loop (one engine, one scenario, one fixture) is 3–6 min. Locally the same loop would be seconds for setup and the same scenario time, since the scenarios are bound by Slurm and engine polling, not by the runner.
  • Agent overhead: each loop is a commit, a push and a wait. Logs can be read only after the job ends, so a hung step costs its whole step timeout (run 1). Four iterations were enough for this scope; a full T3 (three profiles, every failure mode, slurmrestd) would be roughly 10–20 loops [estimate].

Compared with (a), a local machine remains faster to iterate on, as the inbox says. But (b) is not blocked: it needs no new permission (logs carry the fixtures), and the profiles and scripts here are reusable as they are. A practical split: write and debug T3 in CI from a cloud session now, and let Thomas re-run the same workflow, or the same cluster.sh locally, when he wants faster loops.

Follow-ups (for T3 and TRIAGE3; nothing was edited)

  • T3 profiles: add gpu with fake GRES and the api node (slurm-rest, Slurm 26.05 images only, sind node definitions) to slurm-site.yaml.tmpl; a slurm-mini without db for the signal-drain test.
  • Harness: read Snakemake task exit codes from the per-job Slurm log (.snakemake/slurm_logs/rule_<rule>/<wildcards>/<jobid>.log) when the executor is slurm; add oom_real and timeout_fatal to the SLURM scenario set.
  • Snakemake test profile: slurm-init-seconds-before-status-checks: 5 for sind.
  • §3.5: findings 1, 2, 4, 5 and 7.
  • Pin the JDK patch level, or record it in the fixture metadata, since setup-java resolved 21.0.12.1 and 21.0.13 on different runner images within the hour.