Skip to content

T5 · Scenario runner and isolation

Done 2026-10-28 (PR #41, follow-ups PR #43), CI-driven (inbox #51 option (b), assumed) · dev/scenario.py up / down / list · four scenarios up at the same time on one ubuntu-24.04 runner in 54 s (two sind realms that both hold a cluster mini, one kind cluster, one local), no collision, and nothing left after down · green on the first run, 40685163237, 1 min 31 s · review: approve after 4 rounds

T5 figure: three scenarios side by side, each owning one id and one set of names. id 1 and id 2 run happy-slurm-mini: sind realms spx-1 and spx-2, both with a cluster named mini, the same name kept apart by the realm. id 3 runs happy-k8s-mini: kind cluster spx-3 with its API on 127.0.0.1:20301 and its kubeconfig in its data directory. Each id has its Docker network spx-N, its data directory .scenarios/N/ and its ports 20000+100·N to +99; its Compose project spx-N is reserved, not started. A job in each sind cluster writes into its own data directory only, and the kind nodes sit on network spx-3 alone. dev/scenario.py down N removes the realm or cluster, the network and the data directory; docker ps -a, docker network ls and docker volume ls then equal their state before up, and sind get realms and kind get clusters list no spx-* T5 figure: three scenarios side by side, each owning one id and one set of names. id 1 and id 2 run happy-slurm-mini: sind realms spx-1 and spx-2, both with a cluster named mini, the same name kept apart by the realm. id 3 runs happy-k8s-mini: kind cluster spx-3 with its API on 127.0.0.1:20301 and its kubeconfig in its data directory. Each id has its Docker network spx-N, its data directory .scenarios/N/ and its ports 20000+100·N to +99; its Compose project spx-N is reserved, not started. A job in each sind cluster writes into its own data directory only, and the kind nodes sit on network spx-3 alone. dev/scenario.py down N removes the realm or cluster, the network and the data directory; docker ps -a, docker network ls and docker volume ls then equal their state before up, and sind get realms and kind get clusters list no spx-*

What changed

  • dev/scenario.py, one file, Python standard library only (dev-environment §5):
    • up <scenario> [--id N] reads tests/scenarios/<scenario>.yaml, checks free memory, claims the ID by creating .scenarios/N/ (so two parallel up never get the same ID), checks that the ports 20000 + 100·N to +99 are free, creates the Docker network spx-N, then the scheduler's cluster. It prints one JSON object with the names, ports, data directory, URLs and credentials, and keeps a copy in .scenarios/N/scenario.json.
    • slurm-* scenarios: dev/bootstrap.sh into the data directory, then dev/slurm/cluster.sh up <profile> with SIND_REALM=spx-N and SLURM_DATA=.scenarios/N. The cluster keeps its profile name, so two slurm-mini scenarios both hold a cluster mini, in different realms.
    • k8s-* scenarios: dev/k8s/cluster.sh up <profile> with CLUSTER=spx-N, KUBECONFIG=.scenarios/N/kubeconfig, the API server on port 20000 + 100·N + 1, and the nodes on the network spx-N (KIND_EXPERIMENTAL_DOCKER_NETWORK). The JSON gives the API URL and the kubeconfig path.
    • down N needs no state: it deletes the sind realm spx-N and the kind cluster spx-N if they exist, the network, and the data directory. It finds sind and kind on the host PATH only, and keeps the data directory, so the ID stays claimed, when a step fails or when the scenario's sind or kind is not on the host PATH. A failed or interrupted up runs down for its ID.
    • list prints the scenario files and the environments that are up.
  • dev/k8s/cluster.sh: KIND_API_PORT, when set, fixes the API server port (kind networking.apiServerPort) and binds it to 127.0.0.1. Since the review, every node mounts the repository read-only and only the data directory read-write: KIND_DATA_DIR (.scenarios/N from dev/scenario.py), or .scenarios/k8s, the work directory of tests/k8s/scenarios.sh. The Kueue manifest is downloaded to a temporary file in the host's TMPDIR (never the read-write data directory), checked, applied and deleted. dev/slurm/cluster.sh: a header line on SIND_REALM; no code change.
  • Scenario files, tests/scenarios/happy-local.yaml, happy-slurm-mini.yaml, happy-k8s-mini.yaml: the fields of §5 (deployment, scheduler, engine, fixture, params, gate, expect). Only scheduler is used today.
  • Unit tests, tests/scenarios/test_scenario.py (18 tests): the names, ports and paths of an ID, that no two IDs share any of them, the ID range, the YAML subset and what it refuses, the field and value checks, and that every scenario file loads. Since the review: 8 threads claiming at once get 8 distinct IDs, an ID claimed twice and ID 0 are refused, down never passes a PATH with the data directory (subprocess mocked) and keeps the data directory when a step fails or when the cluster tool of a SLURM or Kubernetes scenario is missing from the host PATH (4148643), and a failed up calls down.
  • The proof, tests/scenarios/parallel.sh, and the workflow .github/workflows/scenarios.yml: one job, on a push to any branch that touches its paths, nightly and on manual dispatch, with permissions: {} at the top and contents: read for the job, actions pinned by SHA as in pipelines-k8s.yml, persist-credentials: false, no ${{ }} in run:, a 30-minute timeout. ci.yml does not call it.
  • Spec, dev-environment §5: a "Built in T5" bullet that lists how the build differs; §11: the scenarios.yml row.

Deferred until there is product code, said plainly: up creates environments, it does not run a scenario.

  • The Compose project spx-N is a reserved name and port range in the JSON. Nothing is started, because there is no Compose file and no Spectio server. Port k = 0 waits for Spectio.
  • engine, fixture, params, gate and expect are read and checked, not used: running the engine through the runner, the gate script and the expected events and states all need the runner and the gates.
  • The matrix of §5 and its PR smoke subset, the bundle / compose deployment, and Apptainer.
  • The T3 and T4 harnesses (tests/slurm/scenarios.sh, tests/k8s/scenarios.sh) are not wired to scenario IDs. The SLURM one already follows SIND_REALM, CLUSTER and SLURM_DATA; the Kubernetes one writes to .scenarios/k8s/.

Why

Prompt: next-phase.md · T5. It asks for dev/scenario.py with up, down and list, tests/scenarios/*.yaml, one network, Compose project, port range, data directory and sind realm or kind cluster per scenario ID, JSON output with URLs, and a proof that three scenarios run in parallel without collisions and that down leaves nothing behind. sind and kind cannot run in a Claude Code cloud session (cgroup v1), so the proof runs in CI: option (b) of inbox #51, assumed — pending Thomas. The orchestrator asked for the smallest honest build, reusing dev/slurm/cluster.sh and dev/k8s/cluster.sh. No decision-log entry: the step decided no behaviour.

How to test

Unit tests, anywhere (a cloud session included):

python3 -m unittest discover -s tests/scenarios -p 'test_*.py'

The proof, on a machine whose Docker runs on cgroup v2, with Java 21 on PATH:

bash dev/bootstrap.sh && bash tests/scenarios/parallel.sh

By hand: python3 dev/scenario.py up happy-slurm-mini --id 1, then sind --realm spx-1 exec mini -u spx -- sinfo; python3 dev/scenario.py list; python3 dev/scenario.py down 1. A local scenario (up happy-local) needs only Docker and works in a cloud session: there it was run by hand as --id 7, listed, refused a second up --id 7, and down 7 removed the network spx-7 and .scenarios/7/.

Evidence

CI run 40685163237 on 046e3b1: green, 1 min 31 s for the whole workflow.

Phase Seconds
unit tests, 8 (at 046e3b1) 0.004
dev/bootstrap.sh (runner's tools) 13
four up in parallel, wall clock 54
of which: dev/bootstrap.sh into .scenarios/1/ and .scenarios/2/ 4 and 5
of which: sind mini in realms spx-1 and spx-2 40 and 41
of which: kind spx-3, then the task image pre-pull of dev/k8s/cluster.sh 43, then 10
four down in parallel 8

What tests/scenarios/parallel.sh checked while all four were up, from the job log:

  • the four JSON objects have four names, four data directories and four port ranges;
  • the networks are spx-1, spx-1-mesh, spx-1-mini-net, spx-2, spx-2-mesh, spx-2-mini-net, spx-3 and spx-4;
  • sind get realms lists spx-1 and spx-2, one cluster each, and each realm has a cluster mini; kind get clusters lists spx-3 only;
  • a job in each realm, sbatch --wait as user spx, wrote hello.out into its own data directory: spx-1 worker-0 in .scenarios/1/, spx-2 worker-0 in .scenarios/2/;
  • the kubeconfig in .scenarios/3/ points at https://127.0.0.1:20301, the node spx-3-control-plane is Ready, and it is attached to the network spx-3 alone.

Memory while idle (docker stats): about 116 MiB per slurm-mini realm (controller 30, submitter 22, worker 27, DNS 9 MiB), and 607 MiB for the k8s-mini node.

After down: the docker ps -a, docker network ls and docker volume ls listings equal their state before up; no spx-* realm or kind cluster is left, nor any .scenarios/N/. The last line:

RESULT parallel: 4 scenarios up in parallel (2 sind realms with cluster mini, 1 kind, 1 local), no collision, nothing left after down

The same push ran, all green: dev-env.yml (40685163252), pipelines-k8s.yml (40685163324) and pipelines-slurm.yml (40685163719), so the two cluster.sh changes leave T3 and T4 working. shellcheck -x passes on tests/scenarios/parallel.sh, dev/k8s/cluster.sh and dev/slurm/cluster.sh.

Figures and media

  • t5-scenarios sketch, light and dark, at the top of this page: python3 tools/diagrams/t5_scenarios.py.
  • No cast or screenshot: dev/scenario.py is a development script, and Evidence gives its output.

Findings and open questions

Differences with §5, for triage (the §5 "Built in T5" bullet records them):

  1. Compose project not started. §5 has up create a Compose project; there is nothing to compose yet, so the name and ports are only reserved.
  2. kind on the scenario's network needs KIND_EXPERIMENTAL_DOCKER_NETWORK, which kind 0.33.0 prints as "Here be dragons! This is not supported currently" (provider.go:72-76). Without it, every kind cluster joins the shared network kind, which kind delete cluster never removes, so down would leave it behind. If kind drops the variable, the fallback is to accept the kind network as shared host state.
  3. A port inside the range is used by a cluster: k = 1 is the kind API server.
  4. IDs are 1 to 99, a limit §5 does not state.
  5. The memory check uses rough per-scheduler estimates and is not atomic across parallel up.
  6. Two k8s-kueue scenarios in parallel: not tested. Since the review, each downloads the Kueue manifest to its own temporary file in the host's TMPDIR, so the download no longer races.

  7. sind's address pools: each realm's mesh and each cluster take one Docker network from the daemon's default pools, which a stock daemon fills at about 30 networks (sind realms, v0.15.0). A slurm-* scenario takes three networks (spx-N, mesh, cluster), so about 10 at once on a stock daemon.

  8. Profiles not exercised by CI: slurm-site, slurm-rest and k8s-kueue are accepted by dev/scenario.py (CR-5), but scenarios.yml runs only slurm-mini, k8s-mini and local.

  9. Deferred from the review: ownership of .scenarios/ and of the spx-* names (SEC-T5-04): any local user who can run Docker can act on another user's scenario. Only a symlinked .scenarios/ is refused (SEC-T5-03).
  10. No cast of up / list / down (doc review, 2026-10-28): documentation-system §3.1 asks a build step for a cast of each new CLI flow; none is made yet. The figure draws three of the four CI scenarios (the local one, ID 4, has no cluster). Both left to the next step that touches dev/scenario.py.
  11. The cleanup trap's skip of a released ID (CR-13, CR-16): parallel.sh runs down only for IDs whose .scenarios/<id>/ it still holds. No test runs that path (CI follows the success path); it is checked by reading the code only.

Review (record): round 1 on 5497cfc, code-reviewer changes (0 blockers, 1 major, 6 minors), security-reviewer changes (0 blockers, 2 majors, 5 minors). Round 2 on the fixes (4148643): code-reviewer changes (1 blocker: a failed up could tear down another checkout's spx-<n>, CR-8), security-reviewer approve (5 minors). The round-2 fixes (up owns the network before it may clean up by name, host tools checked before claim, guarded scenario.json, Kueue manifest in TMPDIR, parallel.sh refuses existing IDs) were reviewed in round 3 (on aa05608): changes, 1 major (CR-13: parallel.sh's cleanup could still remove another checkout's scenario) and 1 minor (CR-14: down released the network before the rest was gone), both fixed in PR #43 (aaff4e0, every check green on 5158f4c) and reviewed in round 4: approve, 2 minors (CR-15 fixed, CR-16 recorded as finding 11); SEC-T5-08 is deferred. The unit tests pass (18); shellcheck -x and bash -n pass on dev/k8s/cluster.sh and tests/scenarios/parallel.sh. On the fixes (e347ff4), all green: scenarios.yml 40688967336 (four up in parallel in 72 s, the docker inspect check of the read-write binds included), pipelines-k8s.yml 40688967168 (mini and kueue, the T4 harness writing under the read-only repository through .scenarios/k8s) and dev-env.yml 40688967211. The first push of the fixes, d0d34cb, broke pipelines-k8s.yml (40688649443: DATA was set before dev/lib.sh defined SPECTIO_REPO); e347ff4 fixes it.

CODE  step=T5  review=approve b/M/m=1/2/13  security=approve b/M/m=0/2/10  rounds=4  simplify=notrun -0/+0  tests=pass

Doc review: approve 12/14 (2026-10-28), its changes applied.