P3 · M2 Heads¶
Running since 2026-10-30 · milestone M2 of step P3, Tasks 8–10 of P1 and gap G6 (P2 §2) · Task 8 done on autopilot/P3-t08 (spectio init and spectio doctor, with G6's first three tests), its PR into autopilot/integration opening with this page · its review-code: approve after 3 rounds · Tasks 9 and 10 to do

One Progress page per milestone, opened with its first PR and closed with its done proof (P2 §0). This page covers M2, "Heads" (P2 §2). M1 is on P3-M1.
Tasks¶
| Task | P1 | Depends on | Branch | Status |
|---|---|---|---|---|
8 · spectio init and spectio doctor, with G6's init and doctor tests |
Task 8, G6 | 2 | autopilot/P3-t08 |
done; its PR into autopilot/integration opens with this page |
| 9 · Head script, local launcher and the runner loop | Task 9 | 4, 6, 7, 8 | to do | |
| 10 · tmux launcher on the login node | Task 10 | 9 | to do |
- Task 8 (and G6's first three tests:
test_init_local_environment,test_doctor_local_skips_sbatch,test_init_local_refused_off_linux) - Task 9
- Task 10
- M2 done proof:
uv run pytest packages/spectio/tests -vanduv run pytest -m tmux packages/spectio/tests/test_tmux_real.py -v(P2 §2 item 3)
What changed¶
Task 8 · spectio init and spectio doctor¶
Branch autopilot/P3-t08, base 8bae292, head 179b80a before the figure and this page; 6 commits, 15 files, +1431 −7. The configuration file and spectio doctor are owned by architecture §7.3; where the data lives, by dev-env §15.3. This list names the files.
- The first command line (
packages/spectio, entry pointspectio = "spectio.cli.main:app"; new dependenciespydantic,pydantic-settingsandtyper):config.py(181 lines):Settings, in four layers, lowest first: built-in defaults,spectio.toml, the flatSPECTIO_*variables, flags.spectio.tomlis$SPECTIO_CONFIG, else~/.spectio/spectio.toml.portmust be 1024–65535.engines.extra_argsis one list per engine.write_configrenders the file, parses it back withtomllib, and replaces the old one atomically at 0600.cli/main.py(53 lines): the Typer app withinitanddoctor.cli/init.py(160 lines):spectio init, in the order of the figure. Every option has a default, sospectio opencan run it unattended.cli/doctor.py(139 lines):spectio doctor, oneok|warn|fail <check> <detail>line per check, exit 1 on anyfail.loginshell.py(43 lines):run_loginrunsbash -lcwith the environment cleared exceptHOME,USER,LOGNAME,SHELL,LANGandTMPDIR(P1 Assumption 13), and gives up after 30 s with code 124.whichfinds commands on the login shell'sPATH.
- The server side (
packages/spectio-server):bootstrap.py(130 lines), the only server module the CLI imports, inside the command:init_storecreates the rows of P1 Assumption 18 (oneuserprincipal, projectdefault, one environment of kindslurmorlocalwith the runs root, one runner); run again, it updates the environment and adds nothing.db_checksgives doctor'ssqliteanddb_dirlines.store/db.py:preferred_db_dir()splits out ofdefault_db_dir(), so doctor names the default path without creating it.
- Import contracts (
pyproject.toml): the two edgesspectio.cli.initandspectio.cli.doctor→spectio_server.bootstrapare ignored by the two contracts of architecture §6. A third contract keepsspectio.runner,spectio.configandspectio.loginshellfrom importingspectio.cli. 4 contracts in all. - Tests:
test_init.py(18 tests),test_doctor.py(7 tests, 15 cases), and their fixtures inconftest.py(a fake home, a fakebinon the login shell'sPATH).test_store.pychecks thatdb_checksnames the overflow uid asopen_dbdoes. - Specs (P1 Process: the owning spec changes in the same PR):
- architecture §7.3: a dated note on where
spectio.tomllives and on the 0.0.1 settings; a dated note on what the 0.0.1spectio doctorchecks. - dev-env §15.3, Where data lives: dated notes on the data directory's 0700 rule and on the default database directory, which is never recorded.
- Each new rule is marked (assumed — pending Thomas), inbox #83. No section number changed.
- architecture §7.3: a dated note on where
- Decision log: 2026-10-30 — spectio init and doctor.
- Inbox: #83 new.
- Review record:
docs/reviews/code/P3-t08.md. - Docs of this task: this page, the
p3-t08-init-doctorfigure (tools/diagrams/p3_t08_init_doctor.py, bydiagram-artist, commit09d6ced), and its entry indocs/assets/media.yaml.
Differences from the plan's interface (Task 8)¶
P1 Task 8 gives the interface. The code differs as follows (CR-P3T08-04 (r2) asked for this record). Architecture §7.3 and dev-env §15.3 now own the rules (inbox #83); P1's text changes in the next /apply-findings pass.
| Plan | Code | Why |
|---|---|---|
test_init_defaults_unattended: "spectio.toml records" the data directory, the database directory and the runs root |
it records the data directory and the runs root; db_dir only when it is given |
SEC-P3T08-01 (r1): a recorded default counts as an explicit --db-dir, which loses the mkdtemp fallback once the temporary directory is purged |
engines.extra_args (type not given; Task 22 writes engines.extra_args.snakemake = [...]) |
extra_args = {nextflow = [...], snakemake = [...]}; SPECTIO_ENGINES_EXTRA_ARGS_NEXTFLOW and _SNAKEMAKE |
CR-P3T08-01 (r1): a flat list fails Task 22's file and sends Snakemake flags to Nextflow |
Settings fields as listed |
adds executor (slurm or local) and host (the pin) |
the environment kind of --local (G6) and the host pin that doctor checks |
port=8080 |
1024–65535 | SEC-P3T08-06 (r2): §15.3 req. 8 refuses other ports from server.json |
spectio.toml location not given; architecture §7.3 lists ~/.config/spectio/, /etc/spectio/ |
$SPECTIO_CONFIG, else ~/.spectio/spectio.toml |
Assumption 21 and §15.3 put it in the data directory (CR-P3T08-05 (r1); inbox #83 rule 1) |
| "creates the rows of Assumption 18 through the server store" | through spectio_server.bootstrap, one module, imported inside the command; the user principal is its own created_by |
the import contracts name one module; no system principal exists yet (CR-P3T08-02 (r1), FU-P3-03) |
| tests: 7 | 25 tests, 33 cases | G6's first three, and one test per review fix |
| §15.3: the fallback path goes in the state file | not recorded | the state file is P1 Task 11's (SEC-P3T08-07 (r2)) |
P1 Task 9 Step 3: spectio init calls the same idempotent unpack of nf-spectio |
no unpack yet | the unpack is built in Task 9; see open point 2 |
Why¶
Prompt: next-phase.md · P3. Step: P3 in docs/playbooks/pipeline.yaml (wave W5, kind: build). P2 §2 starts Task 8 as soon as Task 2 (the store) is merged, beside Tasks 3 to 7; Task 9 needs its Settings.
Task 8:
- The first command a user runs.
spectio openrunsspectio initon first use, unattended (dev-env §15.3), so every option has a default and nothing prompts. The defaults are the split of "Where data lives": the data directory in the shared home, the database in a node-local temporary directory, the runs on the shared file system. - The engines run as the user runs them. On a login node the engines come from modules or a conda environment. So
initfinds them through the login shell, with Spectio's own environment cleared (P1 Assumptions 13 and 14), and records the command. - A shared temporary directory is hostile ground. Another user can create
$TMPDIR/spectio-<uid>first. The store of Task 2 answers that with amkdtempfallback; round 1 found that recording the default inspectio.tomlturned that fallback off (SEC-P3T08-01 (r1), record). spectio.tomlholds commands that run.doctorruns the engine commands throughbash -lc, and the head script will. So its directory must be the user's at 0700 (req. 5), and a render that would not load never replaces it (SEC-P3T08-02 (r2), SEC-P3T08-05 (r2)). Its own owner and mode are checked in P1 Task 15 (SEC-P3T08-03).- Local mode is Linux only in 0.0.1 (inbox #73, confirmed by Thomas), whatever layer asks for it (CR-P3T08-02 (r2)).
Evidence¶
Task 8 · spectio init and spectio doctor¶
Files changed by Task 8, against its base:
$ git log --oneline 8bae292..179b80a
179b80a refactor(P3-t08): simplify db_checks' two not-private branches into one
bd794b1 refactor(P3-t08): simplify _free_port, drop its unreachable raise
83332d1 refactor(P3-t08): simplify doctor's db_checks call to one branch
0db5b1a fix(cli): post-approval findings of P3-t08 on spectio init
62fcf09 fix(cli): per-engine extra_args; default db_dir resolved at each start
e41b52a feat(cli): spectio init and spectio doctor
$ git diff --stat 8bae292 179b80a | tail -1
15 files changed, 1431 insertions(+), 7 deletions(-)
09d6ced adds the figure.
The task's tests, re-run by doc-writer on 179b80a (uv's UV_NATIVE_TLS warning cut):
$ uv run pytest packages/spectio/tests/test_init.py packages/spectio/tests/test_doctor.py -v
collecting ... collected 33 items
packages/spectio/tests/test_init.py::test_init_creates_one_environment PASSED [ 3%]
packages/spectio/tests/test_init.py::test_runs_root_must_exist_and_be_absolute PASSED [ 6%]
packages/spectio/tests/test_init.py::test_init_records_engine_commands PASSED [ 9%]
packages/spectio/tests/test_init.py::test_init_engine_missing_still_records PASSED [ 12%]
packages/spectio/tests/test_init.py::test_init_defaults_unattended PASSED [ 15%]
packages/spectio/tests/test_init.py::test_init_port_taken_picks_one_and_keeps_it PASSED [ 18%]
packages/spectio/tests/test_init.py::test_init_local_environment PASSED [ 21%]
packages/spectio/tests/test_init.py::test_init_local_refused_off_linux PASSED [ 24%]
packages/spectio/tests/test_init.py::test_init_needs_the_server_package PASSED [ 27%]
packages/spectio/tests/test_init.py::test_settings_layers PASSED [ 30%]
packages/spectio/tests/test_init.py::test_bad_config_refused PASSED [ 33%]
packages/spectio/tests/test_init.py::test_extra_args_per_engine_round_trip PASSED [ 36%]
packages/spectio/tests/test_init.py::test_init_squatted_default_db_dir PASSED [ 39%]
packages/spectio/tests/test_init.py::test_init_local_from_env_refused_off_linux PASSED [ 42%]
packages/spectio/tests/test_init.py::test_init_refuses_open_or_symlinked_data_dir PASSED [ 45%]
packages/spectio/tests/test_init.py::test_principal_is_the_uid_not_the_environment PASSED [ 48%]
packages/spectio/tests/test_init.py::test_init_value_outside_the_bmp_round_trips PASSED [ 51%]
packages/spectio/tests/test_init.py::test_init_privileged_port_refused PASSED [ 54%]
packages/spectio/tests/test_doctor.py::test_doctor_reports_engines PASSED [ 57%]
packages/spectio/tests/test_doctor.py::test_doctor_engine_command_fails PASSED [ 60%]
packages/spectio/tests/test_doctor.py::test_doctor_reports_fs_and_sqlite PASSED [ 63%]
packages/spectio/tests/test_doctor.py::test_doctor_nxf_plugins_default[None-None-False-ok] PASSED [ 66%]
packages/spectio/tests/test_doctor.py::test_doctor_nxf_plugins_default[false-None-True-warn] PASSED [ 69%]
packages/spectio/tests/test_doctor.py::test_doctor_nxf_plugins_default[nf-schema@2.7.2-None-True-warn] PASSED [ 72%]
packages/spectio/tests/test_doctor.py::test_doctor_nxf_plugins_default[nf-spectio@0.0.2-None-True-warn] PASSED [ 75%]
packages/spectio/tests/test_doctor.py::test_doctor_nxf_plugins_default[nf-spectio-None-True-warn] PASSED [ 78%]
packages/spectio/tests/test_doctor.py::test_doctor_nxf_plugins_default[None-plugins {\n id 'nf-spectio@0.0.1'\n}\n-True-warn] PASSED [ 81%]
packages/spectio/tests/test_doctor.py::test_doctor_nxf_plugins_default[nf-schema@2.7.2,nf-spectio@0.0.1-None-False-fail] PASSED [ 84%]
packages/spectio/tests/test_doctor.py::test_doctor_nxf_plugins_default[nf-schema@2.7.2,nf-spectio@0.0.1-None-True-ok] PASSED [ 87%]
packages/spectio/tests/test_doctor.py::test_doctor_nxf_plugins_default[None-plugins { id 'nf-schema@2.7.2' }\n-False-ok] PASSED [ 90%]
packages/spectio/tests/test_doctor.py::test_doctor_local_skips_sbatch PASSED [ 93%]
packages/spectio/tests/test_doctor.py::test_doctor_host_pin_and_config PASSED [ 96%]
packages/spectio/tests/test_doctor.py::test_doctor_default_db_dir PASSED [100%]
============================== 33 passed in 2.72s ==============================
A real run, by doc-writer on 179b80a, in a cloud session running as root (uid 0), with HOME and TMPDIR set to two fresh 0700 directories in the session's scratch directory (shown as $HOME and $TMPDIR). No engine and no sbatch are installed there, so their lines warn or fail, as they should:
$ uv run spectio init
warning: nextflow not found on the login shell's PATH: give it with --nextflow
warning: snakemake not found on the login shell's PATH: give it with --snakemake
Wrote $HOME/.spectio/spectio.toml; database in $TMPDIR/spectio-0; runs under $HOME/spectio-runs; port 8080
$ echo $?
0
$ cat $HOME/.spectio/spectio.toml
data_dir = "$HOME/.spectio"
runs_root = "$HOME/spectio-runs"
port = 8080
executor = "slurm"
host = "vm"
[slurm]
[engines]
nxf_opts = "-Xmx2g"
extra_args = {}
[runner]
head_watch_s = 5.0
spool_s = 2.0
$ stat -c '%a %n' $HOME/.spectio $HOME/.spectio/spectio.toml $HOME/spectio-runs $TMPDIR/spectio-*
700 $HOME/.spectio
600 $HOME/.spectio/spectio.toml
700 $HOME/spectio-runs
700 $TMPDIR/spectio-0
$ uv run spectio doctor
ok config $HOME/.spectio/spectio.toml
ok sqlite 3.53.1
ok db_dir $TMPDIR/spectio-0 (ext4, 0700)
ok host vm
fail nextflow not found on PATH
fail snakemake not found on PATH
ok java 21
ok tmux /usr/bin/tmux
fail sbatch not found on PATH
ok nxf_plugins_default unset; for a run from a terminal: NXF_PLUGINS_DEFAULT=nf-spectio@0.0.1 nextflow run …
$ echo $?
1
spectio.toml has no db_dir: the default is resolved again at each start (SEC-P3T08-01 (r1)). The refusals, in the same home:
$ uv run spectio init --port 80
spectio init: 1 validation error for Settings
port
Input should be greater than or equal to 1024 [type=greater_than_equal, input_value=80, input_type=int]
For further information visit https://errors.pydantic.dev/2.14/v/greater_than_equal
$ echo $?
2
$ chmod 755 $HOME/.spectio && uv run spectio init
spectio init: $HOME/.spectio must be a directory of yours with mode 0700 (owner uid 0, mode 755): chmod 700 $HOME/.spectio
$ echo $?
2
Local mode, after chmod 700 $HOME/.spectio:
$ uv run spectio init --local
…
Wrote $HOME/.spectio/spectio.toml; database in $TMPDIR/spectio-0; runs under $HOME/spectio-runs; port 8080
$ head -4 $HOME/.spectio/spectio.toml | tail -1
executor = "local"
$ uv run spectio doctor | grep -E 'tmux|sbatch'
ok tmux /usr/bin/tmux
ok sbatch not needed in local mode
The lone-surrogate case of CR-P3T08-05 (r3) / SEC-P3T08-07 (r3), reproduced: a non-UTF-8 byte in a SPECTIO_* value passes the tomllib check and fails at the write. spectio.toml is not replaced (same checksum before and after, no temporary file left), but the message is a raw codec error:
$ md5sum $HOME/.spectio/spectio.toml
572cf43df819c00f3175607d6dc46025 …/spectio.toml
$ SPECTIO_ENGINES_SNAKEMAKE_CMD=$'snake\xff' uv run spectio init
warning: nextflow not found on the login shell's PATH: give it with --nextflow
spectio init: 'utf-8' codec can't encode character '\udcff' in position 307: surrogates not allowed
$ echo $?
2
$ md5sum $HOME/.spectio/spectio.toml
572cf43df819c00f3175607d6dc46025 …/spectio.toml
$ ls -a $HOME/.spectio
. .. spectio.toml
Default gate, re-run by doc-writer on 179b80a (Python 3.13.14):
$ uv run pytest -q | tail -1
247 passed, 14 deselected in 29.35s
$ uv run ruff check
All checks passed!
$ uv run ruff format --check
40 files already formatted
$ uv run mypy packages
Success: no issues found in 39 source files
$ uv run lint-imports | tail -1
Contracts: 4 kept, 0 broken.
$ uv lock --check
Resolved 36 packages in 2ms
The 14 deselected are Task 6's engine tests.
Test-first, from the worker's record: the six tests of 0db5b1a were written first, and all six failed before the code changed; the simplifier's three changes kept the gate green.
Review (record). Finding IDs repeat across rounds, so each carries its round:
- Round 1, on
e41b52a: both changes.code-reviewer: 1 major, 6 minors;security-reviewer: 1 major, 5 minors. The majors, fixed in62fcf09: CR-P3T08-01 (r1),extra_argsas one flat list; SEC-P3T08-01 (r1), the recorded default database directory. - Round 2, on
62fcf09: both approve (5 and 6 minors). The worker then fixed six of them in0db5b1a: local mode from any layer (CR-P3T08-02),_owner()indb_checks(CR-P3T08-03), the strict 0700 data directory (SEC-P3T08-02), the principal from the uid (SEC-P3T08-04), TOML strings outside the BMP (SEC-P3T08-05), the port range (SEC-P3T08-06). - Round 3, on
0db5b1a: both approve (2 minors each). code-simplifier, after round 3: three changes,83332d1,bd794b1and179b80a(−23 +15).- Open: see Findings and open questions.
CODE step=P3-t08 review=approve b/M/m=0/1/13 security=approve b/M/m=0/1/13 rounds=3 simplify=changed -23/+15 tests=pass
Exit check. P1 Task 8's Step 4: uv run pytest packages/spectio/tests/test_init.py packages/spectio/tests/test_doctor.py -v passes (33 passed, above).
Docs gate, run after writing this page and adding the cast:
$ python3 tools/docs/check.py
build: progress hook output checked on 3 pages
mermaid: 5 blocks checked
diagrams: 40 scripts re-run, 80 SVGs compared
media: 187 outputs in the manifest
progress: 50 pages checked
docs gate: 0 failure(s)
$ uvx --with mkdocs-material mkdocs build --strict -d /tmp/spectio-site
INFO - Documentation built in 8.12 seconds
$ echo $?
0
Figures and media¶
p3-t08-init-doctorsketch, light and dark, at the top of this page. It is drawn bydiagram-artistwithpython3 tools/diagrams/p3_t08_init_doctor.py, frominit.py,doctor.py,config.py,loginshell.py,bootstrap.pyand the two test files at179b80a. It shows:- the command on the login node and its four settings layers;
- what is refused before anything is written, with exit 2;
- what
initmakes, in five numbered steps: the directories, the database directory (with the squatted case and itsmkdtempway out), the engines throughbash -lwith the port and the host, the first rows, andspectio.tomllast; doctor's ten checks, each with the statuses it can give.
-
The
spectio-init-doctorcast, the first CLI flow, as documentation-system §3.1 asks of abuildstep. It runs the real CLI of179b80afrom the checkout's.venv:python3 tools/media/cast.py tools/media/casts/spectio-init-doctor.yaml. The setup:- the cast's temporary directory is a demo
HOME, shown as~, with a demoTMPDIRinside it, shown as$TMPDIR; nextflowandsnakemakeare two fake scripts in~/binthat only print a version, put on the login shell'sPATHby~/.bash_profile, becauseinitanddoctorfind engines throughbash -l;- the session has no
sbatch.
It shows two steps: 1.
spectio initwith no option, then thespectio.tomlit wrote, with the two engines recorded and nodb_dir; 2.spectio doctor: every lineokexceptfail sbatch, so exit 1; then the modes, 0700 and 0600. - the cast's temporary directory is a demo
- No screenshot: Task 8 adds no screen.
Findings and open questions¶
Deferred from Task 8's review (record):
| Id | What | Goes to |
|---|---|---|
| CR-P3T08-05 (r1) | spectio.toml is in ~/.spectio/ whatever --data-dir says; architecture §7.3 lists other places |
assumed, inbox #83 rule 1; architecture §7.3 says so |
| CR-P3T08-02 (r1) | The first principal is a user that is its own creator; no system principal; the schema docstring says otherwise |
assumed, inbox #83 rule 2; follow-up FU-P3-03 |
| CR-P3T08-04 (r3), SEC-P3T08-08 (r3) | The strict 0700 rule also refuses a 0755 config directory outside the data directory; the refusal always says chmod 700, even when the owner is wrong, and bypasses _owner(); the check duplicates the store's _private_dir |
assumed, inbox #83 rule 4; the wording and the duplicate, follow-up FU-P3-12 |
| SEC-P3T08-07 (r2) | Which mkdtemp directory a start used is not recorded |
P1 Task 11 (server.json) |
| SEC-P3T08-03 (r1, r2) | spectio.toml is loaded with no owner or mode check, and it holds commands that run |
P1 Task 15 (file modes, §15.3 req. 5) |
| SEC-P3T08-04 (r1) | run_login kills only bash on a timeout; what it started keeps running |
follow-up FU-P3-13, before P1 Task 9 reuses the helper |
| CR-P3T08-03 (r1), CR-P3T08-04 (r1), CR-P3T08-05 (r2), CR-P3T08-05 (r3) / SEC-P3T08-07 (r3), SEC-P3T08-05 (r1), SEC-P3T08-06 (r1) | No server extra behind "install spectio[server]"; a bool setting would be written as True; an unsafe TMPDIR never becomes a doctor line; a lone surrogate gives a raw codec error, untested; doctor's db_dir skips the parent and realpath checks of open_db; doctor reads $NXF_HOME/config with no size cap, following symlinks and FIFOs |
follow-up FU-P3-12 |
Open points from Task 8:
- The plan's interface is out of date. See Differences from the plan's interface (Task 8); P1 Task 8's text changes in the next
/apply-findingspass. - Doctor says "run spectio init" for a missing
nf-spectiocopy, butinitdoes not unpack it yet. WithNXF_PLUGINS_DEFAULT=nf-spectio@0.0.1and no copy under$NXF_HOME/plugins/,doctorfails with "… is missing: run spectio init". P1 Task 9 builds the unpack and makesinitcall it (architecture §13.8); until then the advice does not help. Found bydoc-writerreadingdoctor.pyagainstinit.py, not by the reviewers. - Java's version is printed, not checked.
doctorsaysok java 8as readily asok java 21; P1 asks for the major version only. A minimum belongs with Nextflow's own requirement [to be validated]. - The squash message must keep
Release-As: 0.0.1(dev-env §15.2);e41b52acarries it.
Process:
- Follow-up FU-P3-12: at the next change of
spectio initorspectio doctor(P1 Task 9 or 15 at the latest), the small fixes of Task 8's review: thespectio[server]extra (or another message), TOMLtrue/falsefor bools, a clear message and a test for a value that is not UTF-8, the strict refusal through the store's_private_dirand_owner()with nochmodhint when the owner is wrong, doctor'sdb_dirwith the parent and realpath checks, theTMPDIRwarning as awarnline, and a bounded, non-following read of$NXF_HOME/config. - Follow-up FU-P3-13:
run_loginstartsbashin its own process group and kills the group on a timeout (SEC-P3T08-04 (r1)). - Still pending Thomas: inbox #64 (which directory is node-local at EMBL) and #83.
- Doc review: the verdict goes to the journal when the PR opens. The front matter's
reviewstaysnulluntil the milestone page closes with Task 10.