Skip to content

R4 · Implementation constraints and risks

Done 2026-10-26 · 40 findings: 1 blocker, 29 majors, 10 minors · 4 cited from R1 · 0 dropped · 6 questions for Thomas · 25 spikes ranked

What changed

  • New review file: docs/reviews/R4-constraints.md. It holds:
  • No spec was edited. This is a review round (see the review rules in docs/reviews/README.md). The file proposes new section numbers (api §8.1 and §15, architecture §13) for APPLY1 to use.
  • The file was committed on autopilot/integration in ad6bc9b, together with the other W1 review files, the nav entry and map.md.

Why

The specs describe what Spectio should do. They say little about what will break: the server under a run of 100 000 tasks, the runner and the engines under real HPC site policies, and third-party interfaces that are still marked unstable. R4 looks for those limits before any code depends on them. Each risk must name the cheapest spike that would confirm or kill it.

Prompt: next-phase.md · R4. It runs as an autopilot step: decision log, 2026-10-26, autopilot. Several findings test the event path chosen in decision 0001 (Nextflow through nf-spectio).

Evidence

How the round ran (from the round's header in the review file):

Item Value
Commit read autopilot/integration at 3b209d3; every line number in the file refers to it
Reviewers constraints-hunter, two instances in parallel; both returned output, none failed
Scope (a) server, data model, events, SQLite and Postgres, SSE, outbox, background tasks, backups, with numbers for one run of 100 000 tasks and 50 concurrent runs
Scope (b) runner, engines, schedulers, Kubernetes, HPC site policies, stability of third-party interfaces
External code read Nextflow 114f1fe (26.10.3) and four older tags, snakemake 1c67a66, snakemake-interface-logger-plugins f27aaf1, snakemake-executor-plugin-slurm 8c2af56, Slurm 26.05 man pages, and SQLite, Postgres, PgBouncer, Alembic, MDN, Kubernetes, Kueue and systemd docs. The full list is in the file's header.
Raw output 50 findings: (a) 25 as CH-101 to CH-125 (0 blockers, 17 majors, 8 minors); (b) 25 as CH-201 to CH-225 (1 blocker, 17 majors, 7 minors)
Merge 13 original IDs in six overlapping groups were merged into 7 findings (CH-209 was split between CH-003 and CH-007), a net drop of 6. Four findings that restate R1 (SR-005, SR-014, SR-035, SR-037) are cited, not counted. Result: 50 − 6 − 4 = 40 findings.
Merger's check every quoted spec passage of the blocker and of every major was opened at 3b209d3. All reproduced, so 0 findings were dropped.
Benchmark re-run reviewer (a)'s SQLite benchmark was re-run for the merge (Python 3, SQLite 3.45.1, 500 000 events): 733 MB, 27.3 k events/s, 6.6 s to move one run's events to another project. This matches the reviewer's figures.
Earlier rounds findings open in R1 (SR-), R12 (UX-) and R8b (DEP-T-) are cited, not repeated. Where an R4 finding adds numbers or failure mechanics to an R1 finding, it stays in R4 with a Related line.

Counts, checked against the file's headings:

$ for s in blocker major minor; do printf "%s " $s; grep -c "^### CH-[0-9]* · $s ·" docs/reviews/R4-constraints.md; done
blocker 1
major 29
minor 10
$ grep '^- \*\*Found by' docs/reviews/R4-constraints.md | grep -o 'CH-[12][0-9][0-9]' | sort -u | wc -l
46

The 46 original IDs credited in "Found by", plus the 4 cited ones, account for all 50 raw findings. The counts match the summary table in the file (total 40, dropped 0) and the merge summary returned by the step's workers.

Exit criterion. R4 is done when its review file exists, so that TRIAGE1 can read it:

$ ls docs/reviews/R4-constraints.md
docs/reviews/R4-constraints.md
$ git diff --stat 3b209d3 ad6bc9b -- docs/reviews/R4-constraints.md
 docs/reviews/R4-constraints.md | 911 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 911 insertions(+)

The wave W1 exit check in pipeline.yaml also lists docs/reviews/R11-docs.md, which does not exist yet. That file belongs to step R11, not to R4.

Figures and media

None required (review step).

Findings and open questions

The five findings that matter most (the review file explains the choice):

  1. CH-001 · blocker: the recommended SLURM head job runs the engine and its event bridge on a compute node, where outbound network is often filtered.
    • synthesis §10 (l.396) says the network risk is "not the compute nodes", but a head job runs on one.
    • The only fallback, a server-side forwarder, contradicts "the server never mounts storage" (checking §11.1).
    • Proposal: the bridge writes a spool in the launch directory and the runner forwards it; the runner does all network-dependent preparation on the login node, so the head runs offline.
  2. CH-006 · major: nf-spectio depends on TraceObserverV2, which is @Beta and first appears in Nextflow 25.04. Nextflow swallows observer errors at debug level, so an interface break silently stops all events. No spec states a minimum Nextflow version, and synthesis still carries the nf-tower "aborts the run" sentence.
  3. CH-007 and CH-003 · major: duplicate head jobs.
    • There is no commands table, no lease and no idempotent launch. A runner that crashes after sbatch and before its acknowledgement starts a second head job in the same launch directory.
    • Continue or relaunch can race the previous JVM, which still holds the cache lock.
    • Proposal: launches and runner_commands tables, a CONTINUE_REQUESTED state, and adopting an existing job by its name.
  4. CH-004 · major: no scale model and no performance budgets. The Runs list needs per-run counters the schema lacks, and the graph endpoint has no granularity for 100 000 tasks. Proposal: api §15 budgets and a tier-5 benchmark.
  5. CH-011 · major: the append-only event log keeps raw Nextflow trace records, including script and env. That costs about 0.7–0.9 GB per 100 000-task run and may leak secrets. The size drives backup time (CH-022) and migration time (CH-023). Proposal: archive raw payloads after a run ends, and send script and env at most once.

What the numbers say (from the file's summary): raw SQLite write speed is not the limit. About 27 k events/s per stream was measured, against a peak need in the low thousands per second. The risks are long write locks, storage growth, the single event loop, ordering and delivery rules, and what HPC sites allow.

Questions for Thomas. These go to TRIAGE1 first. TRIAGE1 then copies the ones it cannot settle to inbox.md. The full wording is in questions for Thomas.

# Ref Question
1 CH-001 Should the bridge's default path be a spool in the launch directory that the runner forwards, with direct posting only as an optimisation? Should the runner do all network-dependent preparation on the login node, so the head runs with NXF_OFFLINE?
2 CH-006, CH-009 May the docs state a supported floor of Nextflow ≥ 25.04 (or ≥ 25.10, the oldest tested) and Snakemake ≥ 9.0?
3 CH-004 Are "100 000 tasks in one run" and "50 concurrent runs" targets for the 0.2 alpha or for 1.0? This decides whether three changes go into the walking skeleton (the 0.1.x stage, dev-env §14): the projector split, which moves projection out of the ingest transaction (CH-012); SSE coalescing, one delta per run at most once per second (CH-008); and native backups (CH-022).
4 CH-001, CH-015, CH-026, CH-028 Can you run, or ask EMBL IT to run, the one-hour cluster probe? It covers egress from an sbatch job, login-node file systems, purge policy, lingering and cron, cgroup caps, file locks in scratch, and MaxSubmitJobs. It needs an EMBL account.
5 CH-011 Is it acceptable to archive raw event payloads to the file store once a run is terminal, and to stop nf-spectio sending script and env on every event? This changes synthesis §2ter item 4.
6 CH-022 Should the default same-engine backup be native (the SQLite backup API, pg_dump), with JSON Lines kept only for SQLite to Postgres migration and project export? That departs from depictio's model.

Proposed spikes. The top three of the ranked list are the EMBL cluster probe (question 4), nf-spectio failure modes and volume as an extension of T2, and an ingest and SSE load harness on SQLite and Postgres. Many of the others extend planned steps T1 to T4.

Overlap with R1. Several R4 findings add constraints to R1 findings (for example CH-001 and CH-006 to SR-017, CH-007 to SR-002). Each carries a Related line, so TRIAGE1 can triage the pair together.

Next. TRIAGE1 sorts these findings with the triage policy. APPLY1 then changes the specs and adds the decision-log entries.