Skip to content

R2 · Fact verification of eight specs

Done 2026-10-26 · 206 claims checked (153 true, 40 partly, 8 wrong, 5 unverifiable) · 37 findings: 0 blockers, 9 majors, 28 minors, 0 dropped · 34 open-validation rows: 30 items from the specs, 4 new from the findings

What changed

  • Claim table: verification-2026-10-26.md (new). It lists every external claim in eight specs, with a verdict, a source and a pointer to the finding that corrects it:
    • run 1: schedulers-and-monitoring, synthesis (69 claims);
    • run 2: architecture-plugins-and-packaging, dev-environment-and-testing, cli-and-shared-libraries (74 claims);
    • run 3: api-and-data-model, checking-scenarios, admin-and-operations (63 claims).
  • Findings and open validations: R2-open-validations.md (new). It holds:
    • part A: 37 merged findings, FC-001 to FC-037, each with replacement text;
    • part B: 34 rows, each with the cheapest way to settle it. 30 are the distinct [to be validated] items still in docs/specs (B1 to B30). 4 are new items raised by the findings (B31 to B34);
    • proposed spikes: additions to the prompts of D2, D5 and T1 to T4. No new pipeline step is needed;
    • questions for Thomas.
  • Specs: none changed. This is a review round, and specs are edited only in APPLY1.
  • Sources read: pinned commits of Nextflow, Snakemake and its plugins, Slurm, Kueue, sind, depictio and others, plus a few standards and web pages. The table is in sources and commits read.

Why

The specs make many claims about other tools. The repository rule is that every such claim has a source or is marked [to be validated]. R2 checks those claims against primary sources before any code depends on them.

Prompt: next-phase.md · R2. Decision: the round runs as a pipeline step under the autopilot (decision log, 2026-10-26, autopilot).

Evidence

How the round ran:

  1. Three fact-checker runs, dispatched in parallel. Each ran one /verify-claims line from the prompt. All three returned output and none failed.
  2. Merge. The three runs reported 38 findings: 10 majors and 28 minors. The merge did three things:
    • combined duplicates: run 1 FC-103 and run 3 FC-302 became FC-001; run 1 FC-112 and run 3 FC-307 became FC-010;
    • added FC-037, a batch of the table corrections that had no finding of their own;
    • renumbered all findings in one sequence. The findings table keeps each run's original IDs.
  3. Majors checked again. The merger re-checked every major on 2026-10-26. It opened the quoted spec passages on autopilot/integration and read the external evidence again. All ten pre-merge majors reproduced, so none was dropped. Two details were added during this check (FC-005, FC-007).
  4. Counts recomputed. Run 1 reported 50 true and 14 partly. A recount of its rows gives 51 and 13, and the table uses the recount. Run 3 gave no totals, so the merge counted its rows.
  5. Open validations. A grep on autopilot/integration found 34 places. Two describe the marker convention itself, so they are not open items. Three places ask the same question, so two of them are folded into B1. That leaves 30 distinct items (B1 to B30), and the findings add four new ones (B31 to B34), for 34 rows.

Counts from the review files:

Source Value
Claims checked 206: 153 true, 40 partly, 8 wrong, 5 unverifiable
Findings 37: 0 blockers, 9 majors, 28 minors
Dropped 0
Open validations in part B 34 rows (B1 to B34)
Settled by R2 14 of the 29 open items (B19 is deferred by design)

Exit criterion. The review files must exist so that TRIAGE1 can read them. Checked on autopilot/integration after commit ad6bc9b:

$ ls docs/reviews/verification-*.md docs/reviews/R2-open-validations.md
docs/reviews/R2-open-validations.md
docs/reviews/verification-2026-10-26.md
$ grep -c '^### FC-' docs/reviews/R2-open-validations.md
37
$ grep -o '^### FC-[0-9]* · [a-z]*' docs/reviews/R2-open-validations.md | awk '{print $4}' | sort | uniq -c
      9 major
     28 minor
$ grep -c '^| B[0-9]' docs/reviews/R2-open-validations.md
34
$ git diff --stat 3b209d3 ad6bc9b -- docs/reviews/verification-2026-10-26.md docs/reviews/R2-open-validations.md
 docs/reviews/R2-open-validations.md     | 818 ++++++++++++++++++++++++++++++++
 docs/reviews/verification-2026-10-26.md | 335 +++++++++++++
 2 files changed, 1153 insertions(+)

The claim rows per run, counted the same way, are 69, 74 and 63, which matches the 206 in the totals.

Not in scope. Five specs were not checked: telemetry-and-analytics, ui-spec, authentication-and-access, depictio-ecosystem-reuse and documentation-system. Their [to be validated] markers are still listed in part B.

Figures and media

None required (review step).

Findings and open questions

Top findings (all major, all in R2-open-validations.md):

  • FC-001: the wrong stop signal.
    • Snakemake already stops gracefully on SIGTERM.
    • SIGINT cancels running cluster jobs.
    • A plain scancel kills the head job with SIGKILL after KillWait (30 s by default), so a Nextflow head can die before it cancels its task jobs.
    • The pause, stage-stop and cancel designs in synthesis, schedulers-and-monitoring and checking-scenarios use the wrong signal.
  • FC-006 and FC-007: --name lookups find nothing.
    • squeue --name and sacct --name match exact names only (src/squeue/print.c:3263-3275).
    • Reconciliation after a runner restart uses --name 'spectio-*', so it finds nothing, and "no run is lost" fails.
    • Snakemake names its task jobs <prefix>_<uuid>, so the lookup misses them too.
    • R1 SR-037 reports the same problem. R2 adds the source evidence.
  • FC-002: the generated ext.when replaces the pipeline's own. nf-core/rnaseq 3.27.0 sets ext.when = { !params.use_rustqc } on MARKDUPLICATES. A live check would change what the pipeline does, so the generated value must wrap the original.
  • FC-003: depictio ingest needs the provisioning key. depictio-cli ingest --user requires depictio's instance-wide provisioning key (run.py:1637-1643). As written, the runner on the cluster would need that long-lived secret, which breaks the scoped-token rule.
  • FC-005: Snakemake on Kubernetes needs S3. The executor sets implies_no_shared_fs=True, so remote storage (for example S3), with --default-storage-provider and --default-storage-prefix, is mandatory. The chart's "RWX or S3" choice applies only to Nextflow.
  • The other majors:
    • FC-004: uv tool install may use a system Python, so the SQLite floor is not guaranteed. The fix is --managed-python.
    • FC-008: a Helm pre-install migrate hook runs before its ConfigMap and Secrets exist.
    • FC-009: snakemake-executor-plugin-kueue is a 0.1.0 prototype from 2024.

Overlap with R1. The overlap table has six rows. They cover seven R2 findings (FC-001, FC-002, FC-006, FC-007, FC-014, FC-021, FC-034) and their R1 counterparts in R1-consistency.md. Not every row is a merge: some are the same problem, some are related, and FC-002 and SR-011 are complementary, so both apply. Each row suggests a triage, so that every problem is applied once.

Questions for Thomas. These go to TRIAGE1 first. TRIAGE1 then copies them to inbox.md. The full wording is in questions for Thomas.

# Ref Question Recommended
1 FC-003 Keep the depictio provisioning key on the Spectio server only? The sink plugin mints a per-user token, and the runner runs ingest without --user. Should depictio also get run-scoped, short-lived tokens? Yes to both. The depictio part can come after 0.1.
2 FC-025 Mantine 7, to match depictio, or start on 9.x? The import map makes the Mantine major part of the viewer plugin API. 9, unless components shared with depictio are planned before 1.0
3 FC-022 Which SAML library: pysaml2 (needs the xmlsec1 binary) or python3-saml (needs lxml built from source)? pysaml2
4 FC-014 Make the OS keyring an optional extra, spectio[keyring], so the base install stays pure Python? Yes (R1 SR-027 proposes the same)
5 FC-009 Mark the matrix cell "snakemake × k8s-kueue" as pending until D2 or T4 shows that the 0.1.0 plugin runs on Snakemake 9? Yes
6 FC-013 Pin Node 24 LTS now, and move to 26 once it is Active LTS (2026-10-28)? Yes
7 B9 Should depictio accept the depictio:set-filter and depictio:selection embed messages? Thomas's call, as depictio maintainer
8 B16 Is the adoption target "3 groups, 2 outside your team, 10 active users for 8 weeks, one site other than EMBL" the one to commit to? —
9 B24 Can the EMBL HPC admins say whether a long-running Spectio server may run on a service node, or needs a VM? —
10 B30 Who at EMBL confirms which data-protection policy applies to telemetry from an EMBL-hosted instance? —

Follow-ups proposed for later steps. T3 and T4 settle five of the nine majors (FC-001, FC-005, FC-006, FC-007, FC-008), and FC-009 goes with them. The D2 additions are single commands. See proposed spikes.