Skip to content

P3 · M7 Stages

Running since 2026-10-31 · milestone M7 of step P3 (P2 §7), one task · Task 27 done on autopilot/P3-t27 (stages, declared or detected, above the processes or rules list), its PR into autopilot/integration opening with this page · its review-code: approve after 4 rounds · M7 done proof: the stage tests and the two Playwright checks pass, see Evidence

P3 Task 27: a run's stages, frozen once, above its processes. Top row, the run starts: for Snakemake only, the head runs timeout --foreground 120 snakemake --configfile params.json --list-rules into .spectio/rules.txt before the engine; if it fails or takes over 120 s there is no file, the engine starts and the server reads the rule files instead; a Cancel meanwhile exits 143 and no engine starts. Then the run's first onFlowBegin or workflow_started, before any task (Nextflow's process names came earlier, one per onProcessCreate), and the freeze, once, logged as spectio.stages.frozen, which later launches keep. Middle row, inside freeze, the first source that applies: 2, one lab overlay from ~/.spectio/overlays/, the directory yours and not writable by others, *.spectio.yml, at most 100 of 64 KiB, same engine and workflow name, on Nextflow its version range too; exactly one matches and it applies, two match and none does, the run detail says so. 3, the workflow's own spectio.yml at the root of the directory the launch ran in; for Git the copy fetch wrote under data dir/pipelines/repo/revision, found again for a rebuilt index (CR-P3T27-04); one written for another engine is ignored; read under that directory only. 4, detection: Nextflow one stage per subworkflow scope with its meta.yml description, MULTIQC is Report; Snakemake one per included rule file, the main Snakefile's rules are Final targets; at most 8, under 2 groups no stages. A declared file whose globs overlap on this run's names is refused and detection is used. Bottom: each name gets one stage, to scale: nf-core/rnaseq 3.27.0 star_salmon, 67 processes in 8 detected stages of 9, 11, 12, 6, 13, 9, 6 and 1, the last one Report; mosaicatcher v2.5.1, 101 rules from --list-rules in 8 stages of 15, 18, 18, 14, 13, 7, 15 and 1, the last one Final targets; fnmatchcase on the full name, a name first seen after the freeze has no stage and is listed under Not in a stage. Each stage's state, recounted at the end of each transaction: not_started, running from its first started task, completed when ended with no task left open, skipped when ended with no task, failed when the run failed in it or stopped. Warnings: a source that fails gives no stages, never a failed ingest; YAML has no alias and at most 64 KiB, an error keeps PyYAML's problem, line and column, not the quoted line; rule files are regular files under the main Snakefile's directory, at most 200 of 1 MiB. The stage tests pass P3 Task 27: a run's stages, frozen once, above its processes. Top row, the run starts: for Snakemake only, the head runs timeout --foreground 120 snakemake --configfile params.json --list-rules into .spectio/rules.txt before the engine; if it fails or takes over 120 s there is no file, the engine starts and the server reads the rule files instead; a Cancel meanwhile exits 143 and no engine starts. Then the run's first onFlowBegin or workflow_started, before any task (Nextflow's process names came earlier, one per onProcessCreate), and the freeze, once, logged as spectio.stages.frozen, which later launches keep. Middle row, inside freeze, the first source that applies: 2, one lab overlay from ~/.spectio/overlays/, the directory yours and not writable by others, *.spectio.yml, at most 100 of 64 KiB, same engine and workflow name, on Nextflow its version range too; exactly one matches and it applies, two match and none does, the run detail says so. 3, the workflow's own spectio.yml at the root of the directory the launch ran in; for Git the copy fetch wrote under data dir/pipelines/repo/revision, found again for a rebuilt index (CR-P3T27-04); one written for another engine is ignored; read under that directory only. 4, detection: Nextflow one stage per subworkflow scope with its meta.yml description, MULTIQC is Report; Snakemake one per included rule file, the main Snakefile's rules are Final targets; at most 8, under 2 groups no stages. A declared file whose globs overlap on this run's names is refused and detection is used. Bottom: each name gets one stage, to scale: nf-core/rnaseq 3.27.0 star_salmon, 67 processes in 8 detected stages of 9, 11, 12, 6, 13, 9, 6 and 1, the last one Report; mosaicatcher v2.5.1, 101 rules from --list-rules in 8 stages of 15, 18, 18, 14, 13, 7, 15 and 1, the last one Final targets; fnmatchcase on the full name, a name first seen after the freeze has no stage and is listed under Not in a stage. Each stage's state, recounted at the end of each transaction: not_started, running from its first started task, completed when ended with no task left open, skipped when ended with no task, failed when the run failed in it or stopped. Warnings: a source that fails gives no stages, never a failed ingest; YAML has no alias and at most 64 KiB, an error keeps PyYAML's problem, line and column, not the quoted line; rule files are regular files under the main Snakefile's directory, at most 200 of 1 MiB. The stage tests pass

Tasks

Task Plan Depends on Branch Status
27 · Stages, declared or detected, above the processes or rules list P1 Task 27 4, 5, 18, 20, 25; Task 25 is not built yet, so a run started from a terminal reaches the freeze only through a rebuilt index autopilot/P3-t27 done, PR into autopilot/integration opening with this page
  • Task 27 (its exit check, P1 Task 27 Step 4 and P2 §7 item 3: the stage tests pass, 49 cases, and the smoke passes with the two new checks on the desktop and phone projects, 20 passed and 6 fixme, see Evidence)
  • M7 closes when Task 27 is merged. P2 §7 item 2: Task 27 merges before Task 32, which edits RunDetailPage.tsx and the projector too.

What changed

Task 27 · Stages, declared or detected, above the processes or rules list

Branch autopilot/P3-t27, base 855786c, code head 3d4af9b before the figure and this page; 6 commits, 39 files, +2412 −44. Of these, packages/spectio-server/src/ is +1086 −16, packages/spectio-sdk/ +110, packages/spectio/src/ +42 −5, the Python tests +972 −2, web/ +49 −10, tests/e2e/ +121 −4, tests/pipelines/ +5 −3, ci.yml +2 −1, and the OpenAPI snapshot +25 −3. Where the rules are owned: which source applies and the overlay rules, the checking-scenarios §12.7 note of 2026-10-31; detection, order, state and the listing, the ui-spec §5.1.1 note of 2026-10-31; the API fields, the api §5 note of Task 18 (bullet of 2026-10-31); the tables, api §7.

  • The stages package (packages/spectio-server/src/spectio_server/stages/, new; 8d9b4b4, then the fixes):
    • freeze.py (529 lines): freeze(p, snakefile_main) runs once per run, at its first onFlowBegin or workflow_started. It takes the first source that applies: one lab overlay (find_overlay), else the workflow's own spectio.yml (_own_file), else detection. It writes run_stages, gives every name already known its stage in run_processes, sets tasks.stage_id, and logs spectio.stages.frozen with {mode, source, note}. A source that raises is logged and gives no stages; the freeze is still recorded (38a0946). refresh(p) recounts each stage's task counts and state at the end of each ingest transaction.
    • parse_spectio_yml loads with a SafeLoader that refuses aliases (4fd4bce), caps the file at 64 KiB, and reports errors by problem, line and column only (125cfb7). _run finds the pipeline directory: the launch command's location, else the local location, else a Git copy under the data directory (4fd4bce, 125cfb7).
    • detect_nf.py (155 lines) and detect_smk.py (156 lines): typed ports of the stage-detection spike's detectors, capped at 8 stages. detect_smk.read_confined reads one source file: regular files under the main Snakefile's directory, O_NOFOLLOW | O_NONBLOCK, at most 200 files of 1 MiB. read_rules_txt strips (docstring).
    • assign.py (39 lines): StageDef and assign, the one stage whose match matches a name with fnmatchcase, or Overlap. state.py (36 lines): stage_state, the state words of ui-spec §5.1.1.
  • The spectio.yml model (packages/spectio-sdk/src/spectio_sdk/stages.py, 109 lines): SpectioFile with workflow and stages[] only, strict, unique stage IDs, no detected: IDs; version_in_range and applies, the overlay match. events.py: SnakemakeWorkflowStarted.snakefile_main.
  • Projector and store (projector/, store/schema.py, migration 0005_run_processes.py):
    • Projection.process(name) records each process or rule on first sight in run_processes, with its position, and returns its frozen stage, which tasks.stage_id takes. Nextflow: onProcessCreate records, onProcessTerminate marks ended, onFlowBegin freezes. Snakemake: workflow_started freezes with snakefile_main, run_info sets jobs_planned per rule.
    • Projection.finish() calls refresh before the summary. ingest(…, sources) and StoreSink(writer, runs_root, overlays, data_dir) carry StageSources; serve.py passes <data dir>/overlays and the data directory.
  • Read API (api/runs_read.py, api/models.py): summary.stage_mode on every run (one DISTINCT (run_id, source) query per page), stages_note on GET /runs/{id} from the freeze event, /processes from run_processes first. openapi/0.0.1.json gains the two fields.
  • Head script (packages/spectio/src/spectio/runner/headscript.py): for Snakemake, _LIST_RULES runs timeout --foreground 120 <engine> --snakefile … --directory … --configfile …/params.json <extra args> --list-rules into .spectio/rules.txt.tmp (umask 077), renames it on success, else removes it and prints one line on stderr. It then exits 143 if .spectio/cancel exists (38a0946, CR-P3T27-01). timeout, mv and rm come from the server's PATH.
  • Screens (web/src/runs/RunDetailPage.tsx, api.ts): stages_note, when set, is a role="note" line above the stages or the processes list. The stages themselves were already drawn by UI parity D, fed by /stages.
  • Smoke (tests/e2e/): seed_stages.py (78 lines, new) writes a run directory with the rnaseq preview events (67 onProcessCreate, onFlowBegin, onFlowComplete), which spectio serve indexes at boot; serve-local.sh --seed-stages calls it. skeleton.spec.ts gains two tests: a run of the synthetic happy shows no stage and its processes (needs engines), and the seeded run shows 8 detected stages above its processes (needs SPECTIO_E2E_SEEDED=1). CI job e2e-local passes both.
  • Fixtures (tests/pipelines/real/stages/): in both rnaseq expected/*_detected.json, the description of "Bam qc rnaseq" is null, not "|"; the README says why.
  • Tests: packages/spectio-server/tests/api/test_stages.py (new, 27 tests, 49 cases), packages/spectio/tests/test_headscript.py +3 tests, 4 cases (the listing written or failing, a Cancel during it, none for Nextflow); RunDetailPage.test.tsx +3 tests on the note; small updates for the two new fields in test_read.py, test_projector_nextflow.py, four other web test files and testing.tsx.
  • Specs (P1 Process: the owning spec changes in the same PR), all by the docs commit, marked (assumed — pending Thomas, inbox #93); no section number changed:
    • checking-scenarios §12.7, note 2026-10-31: the overlay key, the pipeline directory, what is read and refused, two overlays, overlap, failures (#93 rules 1 to 3 and 7);
    • ui-spec §5.1.1, note 2026-10-31: include order, the listing, what the server reads, the list's order, state details, the golden change (#93 rules 4 and 5);
    • api §5, bullet of 2026-10-31 in the note of Task 18, and api §7: stage_mode, stages_note, the frozen event, a run_processes row (#93 rule 6).
  • Inbox: #93 new (docs commit).
  • Decision log: 2026-10-31 — the 0.0.1 stages as built.
  • Review record: docs/reviews/code/P3-t27.md.
  • Docs of this task: this page, the p3-t27-stages-freeze figure (tools/diagrams/p3_t27_stages_freeze.py, by diagram-artist, commit 6a00fe5) and its manifest entry.

Differences from the plan's interface (Task 27)

P1 is not edited by this step; these go to the next /apply-findings.

Plan Code Why
test file packages/spectio-server/tests/test_stages.py packages/spectio-server/tests/api/test_stages.py, 27 tests, 49 cases it uses the settings, writer and replay fixtures of tests/api/conftest.py; the plan's 15 named tests are there, plus one per review fix
Files: stages/{detect_nf,detect_smk,assign,state}.py also stages/freeze.py, and table run_processes (migration 0005) the freeze, the sources and the state refresh need one home; a process with no task yet needs a row for its stage (api §7 row added)
store: add run_stages and tasks.stage_id both existed since Task 2; only run_processes is new
read API: no new route no new route; RunDetail.stages_note added the §12.7 note says "the run detail says so" (CR-P3T27-06 r2)
detect_snakemake(main_snakefile, defined: set[str] \| None) defined: Iterable[str] \| None, and cap; detect_nextflow takes a Sequence and cap the rule list is kept in its order for run_processes.position
overlay match on manifest.name and manifest.version Spectio's workflow name and workflow_versions.version nf-spectio sends no manifest; the name is the manifest.name read at launch (#93 rule 2, CR-P3T27-08)
Snakemake order: topological, then include order include order, Final targets last #93 rule 4 (CR-P3T27-07)
the detector reads "the run's pipeline directory" the launch command's location: for Git, the fetched copy; a rebuilt index: the copy under <data dir>/pipelines/ CR-P3T27-03 (r2), CR-P3T27-04 (r3)
--list-rules "a failure there is logged and ignored" as planned, under timeout --foreground 120, stderr dropped; a Cancel during it exits 143 CR-P3T27-01 (r1)
Step 4: a run directory seeded with the rnaseq events tests/e2e/seed_stages.py, serve-local.sh --seed-stages, SPECTIO_E2E_SEEDED=1

Why

  • The prompt: next-phase.md · P3; M7 in P2 §7: Task 27 alone, product code, review-code, "the spectio.yml overlay read is untrusted input".
  • The decision: stages declared or detected in 0.0.1 (Thomas, 2026-10-28, log); detected stage names are inbox #68.
  • The order: after the public demo of PR #111, Thomas wrote "steps & stages pas clickable, pas de détails" (2026-10-31); his "go" set P3-parD, then P1 Tasks 27, 28, 29 and 32, then Task 7 (log, UI parity D). UI parity D drew the stages; Task 27 fills them.
  • What the code had to choose: inbox #93, assumed under the standing rule of autopilot §10.

Evidence

All re-run by doc-writer on the code head 3d4af9b, in the worktree.

The exit check, stage tests (P2 §7 item 3; the path is tests/api/):

$ uv run --locked pytest packages/spectio-server/tests/api/test_stages.py -v
test_detect_nf_golden[rnaseq-3.27.0_star_salmon] PASSED
test_detect_nf_golden[sarek-3.11.0] PASSED
test_detect_smk_golden PASSED
test_detect_smk_unsafe_path PASSED
test_overlay_matching PASSED
test_declared_beats_detected PASSED
test_stages_frozen_at_flow_begin PASSED
test_stage_states PASSED
test_stages_route PASSED
test_freeze_refuses_yaml_aliases[snakemake] PASSED
test_snakemake_freeze_git_workflow[False-True] PASSED
test_old_database_opens PASSED
…
============================== 49 passed in 9.41s ==============================

The exit check, Playwright (P1 Task 27 Step 4). This host looks shared (another uid runs a process here), so spectio serve does not mount the API; the first try ended with "serve-local: no answer on http://127.0.0.1:8779/ after 20 s". The server was run in a private PID namespace instead, with Nextflow 26.10.4 (its SHA-256 equals NEXTFLOW_SHA256 of dev/versions.env):

$ unshare --pid --fork --mount-proc bash tests/e2e/serve-local.sh --port 8779 --seed-stages
== Seeding run run_01K8T27SEEDED0000000000000
INFO spectio_server.serve: index reconciled: 1 runs added
$ CI=true SPECTIO_E2E_BASE_URL=http://127.0.0.1:8779 SPECTIO_E2E_ENGINES=1 SPECTIO_E2E_SEEDED=1 \
    npx --no-install playwright test skeleton.spec.ts --project desktop --project phone
  ✓ [phone]   a run of the synthetic happy shows no stage, only its processes (P1 Task 27) (21.8s)
  ✓ [phone]   a seeded rnaseq run shows 8 detected stages above its processes (P1 Task 27) (655ms)
  ✓ [desktop] a run of the synthetic happy shows no stage, only its processes (P1 Task 27) (22.7s)
  ✓ [desktop] a seeded rnaseq run shows 8 detected stages above its processes (P1 Task 27) (515ms)
  6 skipped
  20 passed (2.9m)

The 6 skipped are the three test.fixme (two wait for sign-in, one for the Snakemake plugin) on both projects. Without engines: 12 passed, 14 skipped. The seeded run through the API:

$ curl -s …/api/v1/runs/run_01K8T27SEEDED0000000000000      # state, stage_mode, stages_note
succeeded detected None
$ curl -s …/api/v1/runs/run_01K8T27SEEDED0000000000000/stages
1 detected:1 detected skipped Prepare genome references + Prepare genome indices
2 detected:2 detected skipped Fastq qc trim filter setstrandedness
3 detected:3 detected skipped Align star + Quantify bam salmon
4 detected:4 detected skipped Bam markduplicates picard
5 detected:5 detected skipped Bam qc rnaseq
6 detected:6 detected skipped Bedtools genomecov fw + … + Bedgraph bedclip bedgraphtobigwig combined
7 detected:7 detected skipped Quantify pseudo alignment
8 detected:8 detected skipped Report
$ curl -s …/processes
67 processes, 0 with no stage

Every stage is skipped because the preview events hold no task.

Test-first, round 3. With freeze.py, sink.py and serve.py of 4fd4bce, then restored:

$ uv run --locked pytest …/test_stages.py -q -k "yaml_error_logged_without_content or extra_key_reported_on_one_line or git_workflow"
7 failed, 42 deselected in 0.75s
E       TypeError: StageSources.__init__() takes 3 positional arguments but 4 were given   # the 3 Git cases
$ git checkout -- <the three files> && uv run --locked pytest …/test_stages.py -q
49 passed in 9.37s

The four YAML cases fail on the file content in the message. The three Git cases fail on the new data_dir argument, not on the rebuilt case alone.

The gate:

$ uv run --locked pytest -q                → 564 passed, 15 deselected in 81.39s
$ uv run --locked ruff check .             → All checks passed!
$ uv run --locked ruff format --check .    → 91 files already formatted
$ uv run --locked mypy packages            → Success: no issues found in 90 source files
$ uv run --locked lint-imports             → Contracts: 4 kept, 0 broken.
$ uv lock --check                          → Resolved 55 packages in 2ms
$ cd web && npx vitest run                 → Test Files 18 passed (18), Tests 178 passed (178)
$ cd web && npx tsc --noEmit               → exit 0
$ cd web && npm run build                  → ✓ built in 3.27s

Review:

CODE  step=P3-t27  review=approve b/M/m=1/2/16  security=approve b/M/m=0/2/16  rounds=4  simplify=changed -12/+7  tests=pass

Two round-4 findings re-checked on Python 3.13.14: Path('/tmp/'+'a%2F'*100).is_dir() raises OSError 36, "File name too long" (SEC-P3T27-05 r4); parse_spectio_yml(b'stages: !SECRET-TOKEN-abc123 []') gives "not YAML: could not determine a constructor for the tag '!SECRET-TOKEN-abc123' (line 1, column 9)" (SEC-P3T27-04 r4), and bytes that are not UTF-8 give "not YAML: ReaderError" (CR-P3T27-06 r4). The review record has the probes.

After the workflow (the orchestrator, test first): both are fixed. _git_copy returns no copy on OSError, so the start event is kept (test_git_copy_name_too_long lost it before the fix). _yaml_problem masks every quoted part of PyYAML's problem; a tag and a %TAG prefix join test_yaml_error_logged_without_content. Gate after the fix: pytest 567 passed, 15 deselected; ruff, ruff format, mypy, lint-imports, uv lock --check clean. The follow-up of this step is FU-P3-30: FU-P3-29 is the public demo's (#111).

Docs: python3 tools/docs/check.py and uvx --with mkdocs-material mkdocs build --strict, results in the PR body.

Figures and media

  • Sketch: p3-t27-stages-freeze (top of this page), tools/diagrams/p3_t27_stages_freeze.py, by diagram-artist (6a00fe5). Its bars come from tests/pipelines/real/stages/expected/.
  • Requested from media-capture, not made in this step: app-run-stages, light and dark, 1440 × 1180, the run detail of the seeded rnaseq run (serve-local.sh --seed-stages), Stages section open with its 8 detected stages; and its phone twin, 390 × 844. Purpose: the first live shot of stages for ui-spec §4.4 and the Features page. No existing shot changes: the only new element on a screen is the stages_note line, shown only when declared stages were refused.

Findings and open questions

Review: docs/reviews/code/P3-t27.md, four rounds; every blocker and major is fixed (CR-P3T27-01 and -02 r1, SEC-P3T27-01 r1, CR-P3T27-03 r2, SEC-P3T27-02 r2).

Inbox: #93, assumed. Its one open option is rule 7: when two overlays match, should the workflow's own spectio.yml still apply (recommended, (b))?

Follow-up FU-P3-30, stages hardening, at the next change of stages/, at the latest before the 0.0.1 release (P1 Task 21):

Findings What is left Owner
CR-P3T27-03 (r1) two matching overlays also skip the author's file FU-P3-30, inbox #93 (b)
CR-P3T27-04 (r1) a forged process name with *, ? or [ makes assign raise outside any try (freeze.py:314) FU-P3-30
CR-P3T27-05 (r1), CR-P3T27-04 (r2) an empty or noisy rules.txt is taken at face value FU-P3-30
CR-P3T27-06 (r1) the listing's stderr is dropped, so the reason it failed is lost FU-P3-30
CR-P3T27-05 (r2) a Ctrl-C or SIGTERM in the head's tmux during the listing, with no cancel marker, starts the engine FU-P3-30
SEC-P3T27-06 (r1), SEC-P3T27-07 (r2) timeout with no --kill-after FU-P3-30
SEC-P3T27-02 (r1), -04 (r2), -04 (r3), CR-P3T27-07 (r2) no cap on the names at the freeze; quadratic inserts and glob checks; rule files read twice FU-P3-30
SEC-P3T27-04 (r1), -05 (r2) read_confined checks then opens by path FU-P3-30
SEC-P3T27-05 (r1, device case), -06 (r2) no test for a device node or for the overlay owner and mode checks FU-P3-30
SEC-P3T27-07 (r1) _failed_stage not scoped to the run FU-P3-30
CR-P3T27-11 (r1) stage times are the batch's last event time FU-P3-30
CR-P3T27-06 (r4) a ReaderError says nothing useful FU-P3-30
CR-P3T27-05 (r3) the _rules_txt comment names the wrong gap for a rebuilt run FU-P3-30
CR-P3T27-09, -10 (r1), SEC-P3T27-08 (r1) the seed_stages.py PID comment; test_old_database_opens checks less than it says FU-P3-30
CR-P3T27-12c (r1) the frozen event's mode and source are never read kept: stored data, a behaviour change to decide
SEC-P3T27-03 (r1, r2), part of -04 (r3) stage sources are read on the single writer thread; a stalled NFS mount blocks every write closed by P3-t28 (2026-10-31): the freeze reads its sources in a file pool of its own (2 threads, apart from the routes' pool, SEC-P3T28-06), 10 s at most, then gives no stages with a note. The bound is on the writer, not the stall's removal: a freeze that stalls still holds the single writer thread for up to 10 s, and once both freeze threads are stuck a freeze is refused at once (test_boot_freeze_pool_apart, test_req13_freeze_reads_off_the_writer; both check less than they say, the second not the "no stages with a note" result: CR-P3T28-05 r2 and CR-P3T28-03 r4, open, FU-P3-31; inbox #94)

Open questions:

  1. Runs started from a terminal. Task 25 is not built, so the freeze for such a run is only exercised through a rebuilt index and the snakefile_main test. Task 25 should re-run test_stages.py with its own events.
  2. The live shot app-run-stages is requested above; until it exists, the stages are shown on the ui-spec §4.4 boards only.
  3. e2e-local on CI now runs serve-local.sh --seed-stages. Its first CI run with this change comes with the PR; the job is still outside ci-ok.