Skip to content

T3 · Mini SLURM with sind

Done 2026-10-28 · CI-driven (inbox #51 option (b), assumed) · slurm-mini, slurm-site and slurm-rest up in 14–35 s on ubuntu-24.04 (the three green runs), Slurm 26.05.5 · both engines green on slurm-site with every failure mode of the synthetic pipelines · every §3.5 pending reason the probes set, plus OUT_OF_MEMORY, TIMEOUT and NODE_FAIL · slurmrestd answers with a JWT on the first try · a Snakemake head in sbatch works · 24 fixtures, 382 kB · reviews: code and security approve in round 2 (round 3 on the follow-ups), docs 13/14 · 6 CI runs, the last green in 17 min 11 s

T3 figure: a GitHub runner installs the tools into a data directory that every sind node mounts at the same path. slurm-site, Slurm 26.05.5: a controller, slurmdbd with account capped (MaxJobs=1), a submitter where nextflow and snakemake run as user spx, partition htc with worker-0 and worker-1 (2 CPU, 1.5 GiB each), partition bigmem with worker-2 (2 CPU, 3 GiB). slurm-rest adds slurmrestd with JWT, whose key stays inside the node; its jobs, nodes, partitions and slurmdb jobs answers are fixtures. Probes and the record each leaves: OOM gives OUT_OF_MEMORY with exit 0:125, a 1-minute time limit gives TIMEOUT, the capped account gives a pending AssocMaxJobsLimit, the maintenance reservation a pending ReqNodeNotAvail, a power cut of worker-2 gives NODE_FAIL. Engines on SLURM: a Snakemake task job ends FAILED 1:0 whatever the tool returned; the synthetic oom mode ends FAILED 137:0, not OUT_OF_MEMORY; a Snakemake head in sbatch works T3 figure: a GitHub runner installs the tools into a data directory that every sind node mounts at the same path. slurm-site, Slurm 26.05.5: a controller, slurmdbd with account capped (MaxJobs=1), a submitter where nextflow and snakemake run as user spx, partition htc with worker-0 and worker-1 (2 CPU, 1.5 GiB each), partition bigmem with worker-2 (2 CPU, 3 GiB). slurm-rest adds slurmrestd with JWT, whose key stays inside the node; its jobs, nodes, partitions and slurmdb jobs answers are fixtures. Probes and the record each leaves: OOM gives OUT_OF_MEMORY with exit 0:125, a 1-minute time limit gives TIMEOUT, the capped account gives a pending AssocMaxJobsLimit, the maintenance reservation a pending ReqNodeNotAvail, a power cut of worker-2 gives NODE_FAIL. Engines on SLURM: a Snakemake task job ends FAILED 1:0 whatever the tool returned; the synthetic oom mode ends FAILED 137:0, not OUT_OF_MEMORY; a Snakemake head in sbatch works

Every record in the figure is in a fixture of tests/fixtures/slurm/slurm-26.05.5/ (run 40680736055).

What changed

The spike t3-ci proved the approach on a GitHub runner; this step keeps its scripts in durable form and widens the scope to the three profiles, every failure mode and slurmrestd.

  • Cluster profiles, dev-environment §4.1: dev/slurm/cluster.sh up mini|site|rest, down mini|site|rest, with the sind configs dev/slurm/site.yaml and dev/slurm/mini.yaml.
    • slurm-site: controller, db (slurmdbd), submitter, worker-[0-1] in htc, worker-2 in bigmem; accounts lab and capped (MaxJobs=1), AccountingStorageEnforce=associations,limits, ConstrainRAMSpace=yes, SlurmdTimeout=45.
    • slurm-rest: the same plus sind's api node (slurmrestd with JWT).
    • slurm-mini: controller, submitter, one worker in htc, no db.
    • The data directory $SLURM_DATA (default .scenarios/slurm, gitignored) is mounted into every node at its own path. The tools are installed into it by dev/bootstrap.sh (pins of dev/versions.env: sind 0.15.0, node image by digest), so the engines run on the submitter as on a login node.
  • Scenarios, tests/slurm/scenarios.sh. Each prints a RESULT line, writes its fixtures and exits 1 on an unexpected outcome.
    • nextflow[:IDS], snakemake[:IDS]: the unchanged harness run_scenarios.py on the submitter, -profile test,slurm or --workflow-profile profiles/slurm. After each scenario, the squeue and sacct records of its first task job that did not end COMPLETED become fixtures.
    • Each engine scenario must leave at least one Slurm job, and every scenario but happy at least one job that did not end COMPLETED; the fixture job is the first that failed by itself (FAILED, OUT_OF_MEMORY, TIMEOUT, NODE_FAIL), not one the engine cancelled. On slurm-mini, which has no accounting, the jobs are counted with squeue and only happy runs.
    • probes: a real OOM, a time limit, an association limit, two-CPU jobs that must pend with Resources and Priority, PartitionTimeLimit, Dependency, DependencyNeverSatisfied, JobHeldUser, JobHeldAdmin (scontrol hold), a maintenance reservation, NODE_FAIL by a power cut, and pending jobs cancelled by root, which must read CANCELLED by 0. Every one of these must appear.
    • heads: the Snakemake harness inside sbatch --wait on bigmem.
    • rest: a token from scontrol token, then GET of ping (v0.0.43 to v0.0.45), jobs, nodes, partitions and slurmdb/jobs, each of which must answer 200; a call without a token must get 401; spx's token sent with X-SLURM-USER-NAME: root must not cancel a held job of root's.
  • CI, .github/workflows/pipelines-slurm.yml, five parallel jobs: nextflow (five scenarios, then the probes), snakemake-a (happy, flaky_retry), snakemake-b (oom_fatal, timeout_fatal, tool_error_then_fixed_resume, then the head in sbatch), rest, mini (happy). It runs on a push to any branch that touches its paths, nightly and on manual dispatch, with permissions: {} at the top and contents: read for the job, no secrets, a 40-minute timeout, actions pinned by SHA and no ${{ }} in run:. Fixtures go to the slurm-fixtures-* artifacts. ci.yml does not call it, so it is not part of ci-ok.
  • Fixtures, tests/fixtures/slurm/slurm-26.05.5/: 24 JSON files, 382 kB, from the slurm-fixtures-* artifacts of run 40680736055. One change: the runner's checkout path /home/runner/work/spectio/spectio is replaced by $REPO (sed); otherwise byte-identical. Each has a _fixture object with the Slurm version, the source run and the command.
  • Spec rows, dev-environment: the three profile rows of §4.1 that this step builds, and the pipelines-slurm.yml row of §11.
  • Research note, copied from spike/t3-ci as evidence: Spike t3-ci, added to the nav.
  • Security. No credential is written on the host or in the mounted data directory: sind creates the munge and JWT keys inside the containers, and the JWT lives only in the environment of one command on the submitter (tests/slurm/rest_get.py sends it and never prints it). tests/slurm/fixture.py refuses to write a document that looks like a JWT, a token header, a munge credential or a key, and tests/slurm/test_fixture.py (run by the workflow) proves it on a made-up JWT. The harness reports are no longer uploaded. A grep of the committed fixtures for eyJ…, SLURM_JWT, X-SLURM-USER-TOKEN, MUNGE: and PRIVATE KEY finds nothing. cluster.sh accepts only a plain absolute SLURM_DATA. Downloads are the ones of dev/bootstrap.sh, pinned by hash; the sind images are pinned by digest.
  • Compared with the spike (CR-T3-09; the note is unchanged):
    • fixtures go to artifacts (fixture.py writes files), not to fenced, hashed blocks in the job log; extract_fixtures.py is gone;
    • five parallel jobs instead of one sequential job, and a data directory under the checkout (.scenarios/slurm) instead of /data;
    • more scenarios per engine (timeout_fatal, flaky_retry, tool_error_then_fixed_resume), more pending reasons, slurm-rest and slurm-mini;
    • only the Snakemake head runs in sbatch (the spike also ran the Nextflow head), as the prompt asks;
    • fixture names carry the scenario, and the directory carries the Slurm version.
  • Deferred (review round 1, record), each also in the differences below:
    • CR-T3-02, the rest of it: head OUT_OF_MEMORY, head TIMEOUT, PREEMPTED, BOOT_FAIL. Head OOM and walltime are reachable but need a head-sized scenario; PREEMPTED needs a preemption setup (QoS or partition priority); BOOT_FAIL is likely unreachable on sind, whose nodes are containers that are already up.
    • CR-T3-03: partition gpu with fake GRES (no test in W3 asks for a GPU request, and fake GRES on sind is unverified), and the slurm-mini signal-drain scenario (it needs a head job driven by runner code).
    • oom_once_retry is not run on SLURM: oom_fatal covers the same injection, and the retry logic is the engine's.
    • SEC-T3-01: the tools that the nodes put on PATH live in the data directory, which jobs in the cluster can write. T4 has the same convention (the repository mounted into the kind nodes).
    • Neither engine's timeout_fatal yields a Slurm TIMEOUT (differences 8 and 9).
    • Also not built: slurm-versions (25.11), fairshare, pre-flight, and the runner-restart test on slurm-mini (it needs runner code).

Why

Prompt: next-phase.md · T3. It asks for the three sind profiles, the synthetic pipelines on slurm-site with each failure mode, squeue / sacct / sinfo --json and slurmrestd answers as golden fixtures, a check that each pending reason and failure class of schedulers §3.5 appears, NODE_FAIL by a power cycle, and whether a Snakemake head in sbatch works for the 0.0.1 fallback. sind cannot run in a Claude Code cloud session (cgroup v1), so the loop runs in CI: option (b) of inbox #51, assumed — pending Thomas. No decision-log entry: the step decided no behaviour; the differences below wait for TRIAGE3.

How to test

On a machine whose Docker runs on cgroup v2 (not a Claude Code cloud session), from the repository root:

export SLURM_DATA=$PWD/.scenarios/slurm SPECTIO_HOME=$PWD/.scenarios/slurm/tools \
  SPECTIO_BIN=$PWD/.scenarios/slurm/tools/bin UV_PYTHON_INSTALL_DIR=$PWD/.scenarios/slurm/tools/python
mkdir -p "$SLURM_DATA" && bash dev/bootstrap.sh && bash dev/slurm/cluster.sh up site && \
  for s in nextflow:happy,oom_fatal probes snakemake:oom_fatal heads; do CLUSTER=site bash tests/slurm/scenarios.sh "$s" || break; done

Java 17 or later must be on PATH or in JAVA_HOME: up copies it into the data directory. Fixtures land in .scenarios/slurm/fixtures/. bash dev/slurm/cluster.sh down site removes the cluster. For slurmrestd, up rest and CLUSTER=rest bash tests/slurm/scenarios.sh rest. In CI, push a change under tests/slurm/, or dispatch pipelines-slurm.yml once it is on main.

Evidence

CI runs of pipelines-slurm.yml on autopilot/T3, then on autopilot/T3-followups:

Run Commit Result Wall clock What changed after it
40675777338 6861384 green, 5 of 5 jobs 12 min 53 s A failed ALIGN array task pulled its whole array into the fixture (161 kB, 20 jobs): fixtures now keep the task's own job id (sacct JobIDRaw).
40677516348 4f7c60c green, 5 of 5 jobs 11 min 57 s Review round 1 on this head: the fixes are in 53c287d.
40679302148 53c287d rest red; mini green; the others cancelled by the next push — spx's token with X-SLURM-USER-NAME: root got 200 on GET ping: the check now tests the effect (difference 14).
40679709039 09ed12b mini red; rest green; the others cancelled by the next push — The new job-count check called sacct on slurm-mini, which has no accounting: squeue there.
40680311800 05fa754 snakemake-b red; mini and rest green; the others cancelled by the next push — The accounting test sacct -j 0 fails on slurm-site too (job id 0 is invalid), so every engine scenario took the slurm-mini path: the test now queries the last hour.
40680736055 e2a9f6d green, 5 of 5 jobs 17 min 11 s Its fixtures are committed.
40684589839 3d01ea5 rest green; the others cancelled by the next push — Round-2 follow-ups. The positive control passed but read spx's cancelled job as gone: squeue -j without --states=all lists no CANCELLED job.
40684851443 e5011c3 green, 5 of 5 jobs 16 min 36 s The control now reads CANCELLED.

Timings (seconds) per job, run 40680736055:

Job Profile Bootstrap Cluster up Scenarios Job total
nextflow slurm-site 34 20 486: five scenarios 377, probes 109 549
snakemake-a slurm-site 42 27 438 518
snakemake-b slurm-site 44 18 886: three scenarios 523, head in sbatch 363 958
rest slurm-rest 39 22 4 76
mini slurm-mini 61 35 113 226

Per scenario, run 40675777338 (before the review fixes): Nextflow happy + oom_fatal + timeout_fatal + flaky_retry + tool_error_then_fixed_resume 369 s; probes 99 s (NODE_FAIL 69 s after the power cut); Snakemake happy + flaky_retry 529 s; oom_fatal + timeout_fatal + tool_error_then_fixed_resume 471 s; Snakemake head in sbatch 234 s; rest 5 s; slurm-mini Nextflow happy 114 s.

Result lines of run 40680736055:

RESULT nextflow: happy=pass;not-completed:[] oom_fatal=pass;not-completed:[1×CANCELLED 3×FAILED|137:0 ] timeout_fatal=pass;not-completed:[3×FAILED|140:0 ] flaky_retry=pass;not-completed:[2×FAILED|1:0 ] tool_error_then_fixed_resume=pass;not-completed:[1×FAILED|1:0 ]
RESULT probes: pending=AssocMaxJobsLimit PartitionTimeLimit Dependency DependencyNeverSatisfied JobHeldUser JobHeldAdmin ReqNodeNotAvail res:[PENDING/Resources PENDING/Priority PENDING/Priority] oom=OUT_OF_MEMORY timeout=TIMEOUT node_fail=NODE_FAIL after 74 s; root scancel=CANCELLED by 0
RESULT snakemake: happy=pass;not-completed:[] flaky_retry=pass;not-completed:[2×FAILED|1:0 ]
RESULT snakemake: oom_fatal=pass;not-completed:[3×FAILED|1:0 ] timeout_fatal=pass;not-completed:[3×FAILED|1:0 ] tool_error_then_fixed_resume=pass;not-completed:[3×FAILED|1:0 ]
RESULT heads: snakemake head in sbatch: happy pass, head COMPLETED|0:0; plugin says: You are running snakemake in a SLURM job context. This is not recommended, as it may lead to unexpected behavior. If possible, please run Snakemake directly on the login node.
RESULT rest: ping-v0.0.43 200 732 ping-v0.0.44 200 731 ping-v0.0.45 200 712 jobs 200 20183 nodes 200 8323 partitions 200 8957 slurmdb-jobs 200 18110 no-token=401 spx-token-as-root:DELETE-root-job=511,root-job=PENDING
RESULT nextflow: happy=pass;jobs-in-squeue:65

With the round-2 follow-ups, run 40684851443 adds the positive control (spx deletes its own held job) to the rest line:

RESULT rest: ping-v0.0.43 200 731 ping-v0.0.44 200 731 ping-v0.0.45 200 712 jobs 200 20183 nodes 200 8368 partitions 200 8957 slurmdb-jobs 200 18108 no-token=401 spx-token-own:DELETE-own-job=200,own-job=CANCELLED spx-token-as-root:DELETE-root-job=511,root-job=PENDING
  • not-completed counts the task jobs of the scenario that did not end COMPLETED, by State|ExitCode; happy has none.
  • KNOWN GAP lines: for Snakemake oom_fatal, the only failed harness check is "exit code 137 not seen"; it is accepted explicitly, and every other check applies (difference 1 below).
  • Snakemake head in sbatch: works. The head job ends COMPLETED 0:0 and happy passes; the plugin logs "You are running snakemake in a SLURM job context. This is not recommended…" and clears the SLURM_* variables before it submits (spike note, Result). So the 0.0.1 head-job design works for this pipeline and the fallback of dev-environment §15.3 is not needed for it. Not tested: cancelling the head, a head at its walltime.
  • slurmrestd: works on the first try. All three ping versions answer 200, so slurmrestd 26.05.5 serves v0.0.43, v0.0.44 and v0.0.45; every answer names data_parser/v0.0.45, Slurm 26.05.5. A call without a token gets 401. spx's token with X-SLURM-USER-NAME: root: GET ping answers 200 (run 40679302148), DELETE of root's held job answers 511 and the job stays PENDING, while the same DELETE of spx's own held job, with spx's own name, answers 200 and the job ends CANCELLED (run 40684851443). The 511 body reads "error": "Protocol authentication error", "error_number": 1007, "source": "slurm_kill_jobs()": slurmctld refused the kill request as an authentication error, and slurmrestd maps SLURM_PROTOCOL_AUTHENTICATION_ERROR to 511 Network Authentication Required (src/common/http.c:148, branch slurm-26.05, read 2026-10-28).

Review (record): round 1 on 4f7c60c, code-reviewer changes (1 blocker, 2 majors, 7 minors), security-reviewer approve (4 minors); the fixes are in 53c287d, 09ed12b, 05fa754 and e2a9f6d, proved by run 40680736055. Round 2 on 41428f5 (the delta): both approve, with 5 and 3 minors; the docs ones are fixed, the test ones (a positive control for the impersonation check, accounting chosen from the profile, the fixture guard over every pattern and over the committed fixtures, in ci-ok) are fixed on autopilot/T3-followups in 3d01ea5 and e5011c3, proved by run 40684851443, and SEC-T3-06 goes to TRIAGE3. doc-reviewer approve, 13/14. shellcheck -x and bash -n pass on dev/slurm/cluster.sh, dev/slurm/env.sh and tests/slurm/scenarios.sh; tests/slurm/test_fixture.py, 4 tests (one with a subtest per credential pattern, one per committed fixture), passes locally, and runs in the unit job of ci.yml on the follow-ups PR.

CODE  step=T3  review=approve b/M/m=1/2/16  security=approve b/M/m=0/0/11  rounds=3  simplify=notrun -0/+0  tests=pass

Fixtures (tests/fixtures/slurm/slurm-26.05.5/):

File Holds
nextflow-<scenario>-{squeue,sacct}.json, 4 scenarios the first not-COMPLETED task job of oom_fatal (FAILED 137:0), timeout_fatal (FAILED 140:0), flaky_retry and tool_error_then_fixed_resume (FAILED 1:0)
snakemake-<scenario>-{squeue,sacct}.json, 4 scenarios the same four on Snakemake, all FAILED 1:0
squeue-pending-reasons.json eleven pending or running jobs: AssocMaxJobsLimit, Resources, Priority, PartitionTimeLimit, Dependency, DependencyNeverSatisfied, JobHeldUser, JobHeldAdmin, ReqNodeNotAvail with "Reserved for maintenance"
sacct-probes.json OUT_OF_MEMORY (exit 0, signal 125), TIMEOUT, NODE_FAIL (exit 1, ERROR), five jobs cancelled by root (CANCELLED, kill_request_user: root), the finished jobs that were pending
sinfo.json, sinfo-node-down.json the partitions, then worker-2 DOWN, NOT_RESPONDING
rest-{jobs,nodes,partitions,slurmdb-jobs}.json slurmrestd answers, v0.0.45; jobs kept to one COMPLETED and one FAILED (exit 3)

Figures and media

  • t3-sind-slurm sketch, light and dark, at the top of this page: python3 tools/diagrams/t3_sind_slurm.py.
  • No cast or screenshot: cluster.sh and scenarios.sh are development scripts with no user flow, and How to test shows the commands and their RESULT lines.

Findings and open questions

Differences with the specs, for TRIAGE3; the specs are not edited, except the §4.1 and §11 rows this step builds (items 10 to 12). Items 1 to 7 are the spike's (note, Differences); T3 runs confirm them except where said.

  1. Snakemake on SLURM hides the task exit code (§3.5 tool_error): note item 1. Confirmed on oom_fatal, timeout_fatal, flaky_retry and tool_error_then_fixed_resume: every failed Snakemake task job is FAILED 1:0.
  2. The synthetic oom mode is not an OOM for Slurm (§3.5 oom; dev-environment §3 "On a cluster, SLURM reports OUT_OF_MEMORY"): note item 2. Nextflow oom_fatal is FAILED 137:0; only the probe's real allocation gives OUT_OF_MEMORY.
  3. Snakemake status polling dominates small runs: note item 3. The test arguments keep 5 s.
  4. Engine cleanup looks like an external cancel (§3.5 cancelled_external): note item 4. Nextflow oom_fatal left 6 CANCELLED jobs in run 40675777338 and 1 in run 40680736055.
  5. The pending reason in --json is split in two fields, and sacct keeps the last pending reason on a finished job: note item 5. New detail: of the eleven probe jobs, only the maintenance one has a non-empty state_description; for AssocMaxJobsLimit, Resources, Priority, PartitionTimeLimit, Dependency, DependencyNeverSatisfied, JobHeldUser and JobHeldAdmin, state_reason alone carries the reason.
  6. sacct -s returned nothing in a spike run: note item 6. Not retested; scenarios.sh filters states itself.
  7. NODE_FAIL has exit 1, ERROR: note item 7. Confirmed.
  8. New: a Nextflow task over its time on SLURM ends FAILED 140:0, not TIMEOUT (§3.5 timeout: "TIMEOUT"). Nextflow adds --signal B:USR2@30 to every task job unless the user set --signal (SlurmExecutor.groovy:71-75 at v26.10.4), and the task wrapper traps USR2 (command-run.txt:192); 140 is 128 + 12 (USR2). In timeout_fatal the job had a 1-minute limit and ended after 5 s (nextflow-timeout_fatal-sacct.json, job 150). So the task is stopped from inside before Slurm's limit, and the Slurm record never says TIMEOUT. The classifier must map FAILED 140 on a Nextflow SLURM task to timeout, as the synthetic pipeline's errorStrategy already does. Whether this holds for limits well above a minute, where USR2 should come 30 s before the end, is [to be validated], and so is why it came after about 5 s on a 60 s limit (Slurm may send the --signal signal earlier than asked [to be validated]).
  9. New: the Snakemake synthetic timeout is a tool exit, not a scheduler timeout. The Snakefile wraps the task in timeout, so timeout_fatal is FAILED 1:0 on SLURM (item 1). Only the probe produces a Slurm TIMEOUT. For a timeout class test with Snakemake on SLURM, the scenario would need a runtime the job really exceeds.
  10. slurm-mini needs a partition htc, because both engines' slurm profiles of the synthetic pipelines name it. The §4.1 row did not say so; dev/slurm/mini.yaml adds it, and the row now does.
  11. slurm-site has no gpu partition yet (§4.1 row: "gpu (fake GRES)"). Not in this step's runs; the row now marks it as planned.
  12. The §11 row said pipelines-slurm.yml is called by ci.yml and uses GSI-HPC/sind-action@v2. It is standalone (push with a path filter, nightly, manual), outside ci-ok, and gets sind from dev/bootstrap.sh. The row now says so. The Required checks paragraph of §11 still calls the pipeline workflows reusable workflows that ci.yml calls, as for pipelines-k8s.yml (T4).
  13. slurmrestd serves v0.0.43 to v0.0.45 on 26.05.5 (ping 200 on each). Schedulers §3 data_parser negotiation can rely on at least these three on 26.05; which versions 25.11 serves is not measured (its sind images have no slurmrestd).
  14. New: X-SLURM-USER-NAME with a user's own token. spx's token sent with X-SLURM-USER-NAME: root gets 200 on GET ping (run 40679302148), and 511 on DELETE of a held job of root's, which stays pending (run 40680736055). Run 40684851443 adds the control: spx's own name on spx's own held job answers 200 and cancels it. So the header does not let a user act as root, but slurmrestd does not refuse the mismatch outright: the 511 is slurmctld refusing the kill RPC as Protocol authentication error (1007), mapped to 511 in src/common/http.c:148 (branch slurm-26.05, read 2026-10-28). Why ping passes is [to be validated]. For schedulers §3: Spectio must send the token's own user name and treat a 2xx on ping as no proof of identity.
  15. New: who cancelled is a user name in sacct --json. §3.5 maps CANCELLED by <uid>, the text form of sacct (CANCELLED by 0 in run 40680736055). In sacct --json the state is only CANCELLED, and the canceller is kill_request_user: "root", a name, in sacct-probes.json (the five cancelled probe jobs; every other record has ""). The classifier must read that field and compare it with the run's user name.
  16. Deferred from review round 1: head OUT_OF_MEMORY and head TIMEOUT (§3.5 "Head out of memory", "Head walltime"), PREEMPTED (needs a preemption setup) and BOOT_FAIL (likely unreachable on sind) are not produced; partition gpu and the slurm-mini signal drain are not built; oom_once_retry is not run on SLURM; SEC-T3-01, tools on PATH from a cluster-writable mount (same in T4). See What changed, Deferred.

Open questions.

  • Should pipelines-slurm.yml join ci.yml and ci-ok? Its longest job takes about 16 minutes, so probably nightly and on path changes only.
  • slurm-versions (25.11) and partition gpu: the prompt asks for the §4.1 profiles, so they are deferred, not out of scope. Build them in a later step?