Skip to content

P3 · M8 Job files and run tools

Running since 2026-10-31 · milestone M8 of step P3 (P2 §8), five tasks · Task 28 done on autopilot/P3-t28 (each job's files and the work directory), its PR into autopilot/integration opening with this page · its review-code: approve after 4 rounds · Task 28 proof: the job-file tests pass, 43 cases, see Evidence; its slurm-mini check waits for P1 Task 22

P3 Task 28: each job's files, read inside the run and hidden where needed. Top row: the engine's events (Nextflow workDir and native_id; Snakemake log paths, the SLURM job ID and log) go to the projector on the writer, which keeps a Nextflow task folder only two levels below the run's work folder (2 hex, then 30 hex), a Snakemake path only inside the launch folder (in .spectio/ only head.out, head.err and nextflow.log), a job ID of up to 128 plain characters; anything else is counted as path_outside_run or bad_native_id, not kept. Kept values go to the attempts row, migration 0006: work_dir, native_id, logs. The roots come from the runs root and the run ID, never from a stored path. The stage freeze has its own pool, 2 threads and none waiting, 10 s at most for one run; route reads never take its slots; a freeze that stalls holds the writer 10 s, then the run has no stages. Middle row, one request for a job's files, in order: 1, the listing names every file the attempt may have with a state, present, not_yet, absent, deleted_after_success or cleaned; a read is …/files/NAME as text/plain; n is the latest launch that has it. 2, the file pool, 4 threads and 16 waiting, its own threads, not the writer's nor the runner's, 5 s per listing or read, refused at once when full: 503 runner-offline. 3, the name: Nextflow's eight names only, never .command.env, .command.in, .command.stage or a staged input; Snakemake a path its events recorded, byte for byte: else 422 file-not-allowed. 4, the walk: from the run's own folder by run ID, one folder at a time with O_NOFOLLOW, each one yours, then a regular file of yours, no FIFO, no device, no link followed, not even inside the run; a part over 255 bytes or a name over 4096 refused: 422, 404 log-not-found or 410 log-cleaned. 5, what is shown: a NUL in the first 8 KiB is 415 binary-file; escape sequences removed; .command.run with every value hidden; ?tail= or one Range, 1 MiB at most, 200 or 206. Bottom: .command.run read with a shell lexer, before and after: export TOKEN, FOO=bar before a command, docker -e and --env=, an export inside a here-document and an array all become «hidden»; it fails closed, from the line it cannot read everything is hidden, for example 70 nested $( past the limit of 64. A file system error the walk does not map: EIO or ESTALE gives 503, any other 404, with a fixed text. The tests pass P3 Task 28: each job's files, read inside the run and hidden where needed. Top row: the engine's events (Nextflow workDir and native_id; Snakemake log paths, the SLURM job ID and log) go to the projector on the writer, which keeps a Nextflow task folder only two levels below the run's work folder (2 hex, then 30 hex), a Snakemake path only inside the launch folder (in .spectio/ only head.out, head.err and nextflow.log), a job ID of up to 128 plain characters; anything else is counted as path_outside_run or bad_native_id, not kept. Kept values go to the attempts row, migration 0006: work_dir, native_id, logs. The roots come from the runs root and the run ID, never from a stored path. The stage freeze has its own pool, 2 threads and none waiting, 10 s at most for one run; route reads never take its slots; a freeze that stalls holds the writer 10 s, then the run has no stages. Middle row, one request for a job's files, in order: 1, the listing names every file the attempt may have with a state, present, not_yet, absent, deleted_after_success or cleaned; a read is …/files/NAME as text/plain; n is the latest launch that has it. 2, the file pool, 4 threads and 16 waiting, its own threads, not the writer's nor the runner's, 5 s per listing or read, refused at once when full: 503 runner-offline. 3, the name: Nextflow's eight names only, never .command.env, .command.in, .command.stage or a staged input; Snakemake a path its events recorded, byte for byte: else 422 file-not-allowed. 4, the walk: from the run's own folder by run ID, one folder at a time with O_NOFOLLOW, each one yours, then a regular file of yours, no FIFO, no device, no link followed, not even inside the run; a part over 255 bytes or a name over 4096 refused: 422, 404 log-not-found or 410 log-cleaned. 5, what is shown: a NUL in the first 8 KiB is 415 binary-file; escape sequences removed; .command.run with every value hidden; ?tail= or one Range, 1 MiB at most, 200 or 206. Bottom: .command.run read with a shell lexer, before and after: export TOKEN, FOO=bar before a command, docker -e and --env=, an export inside a here-document and an array all become «hidden»; it fails closed, from the line it cannot read everything is hidden, for example 70 nested $( past the limit of 64. A file system error the walk does not map: EIO or ESTALE gives 503, any other 404, with a fixed text. The tests pass

Tasks

Task Plan Depends on Branch Status
28 · Each job's files and the work directory P1 Task 28 4–7, 9, 18; Task 7 (Snakemake logger plugin) and Task 22 (slurm-mini) are not built, so the Snakemake side is fed by replayed events autopilot/P3-t28 done, PR into autopilot/integration opening with this page
29 · Failure summary and resources per job P1 Task 29 28 not started
30 · Resume, copy to a new launch and the copy actions P1 Task 30 28 not started
31 · Results and reports P1 Task 31 28 not started
32 · The screens P1 Task 32 20, 27, 28–31, G3b not started
  • Task 28 (P1 Task 28 Step 4 and P2 §8 item 3: the Step 2 command passes, 43 cases; the two Nextflow engine tests pass, see Evidence)
  • Task 28 on slurm-mini: a finished Nextflow task shows a non-empty .command.log and a Snakemake job its SLURM log. Waits for Task 22's e2e job, batched with Tasks 29 and 30 (P2 §8 item 2).
  • M8 closes when Tasks 28 to 32 are merged and the cluster checks of 28 to 30 pass.

What changed

Task 28 · Each job's files and the work directory

Branch autopilot/P3-t28, base 3a103a9, code head d4f41fc before this page; 5 commits, 33 files, +2893 −134. Of these, packages/spectio-server/src/ is +1540 −124, the Python tests +1036 −1, the OpenAPI snapshot +249, and the spec notes and inbox +29 −2. Where the rules are owned: what is served and how it is checked, the dev-env §15.3 req. 13 note of 2026-10-31; the files per engine and the head script, the architecture §13.7 note; the routes, the api §5 note of 2026-10-31; the columns, the api §7 note; the events and the drop counters, api §14.1.

  • The files package (packages/spectio-server/src/spectio_server/files/, new):
    • confine.py (228 lines): the walk of checking-scenarios §11.1. open_confined and list_confined open each folder with O_DIRECTORY|O_NOFOLLOW from the root's descriptor and check its owner, then the file with O_NOFOLLOW|O_NONBLOCK: a regular file of the user. safe_rel refuses an absolute name, . or .., a control character, a component over 255 bytes or a name over 4096 (0fee201). Also stat_confined, tail, read_at, size_of.
    • pool.py (72 lines): FilePool, its own daemon threads with a bounded backlog; a caller waits at most its timeout, then gets Stalled; a full pool refuses at once.
    • redact.py (319 lines): redact_command_run, a shell lexer that hides the value of every assignment word (at a command's start, after export and the other declaring words, after -e or --env, inside here-documents, arrays), failing closed from the first line it cannot read or past 64 nested constructs (297ba97, 0fee201). strip_escapes removes CSI, OSC and C0/C1 controls.
    • taskfiles.py (371 lines): the eight Nextflow names served, the five listing states, check_name (a Snakemake name equals a recorded path byte for byte), read_task_file (no link followed for a job file, f362fc5; a NUL in the first 8 KiB is binary), and task_work_rel, launch_rel and native_id, the checks the projector applies to a path or ID from an event.
  • Routes (api/taskfiles.py, 187 lines): GET /tasks/{id}/attempts/{n}/files and …/files/{name} (?tail=, ?download=, one Range), run in the routes' pool with 5 s; refusals map to 422 file-not-allowed, 404 log-not-found, 410 log-cleaned, 415 binary-file and 503 runner-offline; other OSErrors to 503 (EIO, ESTALE) or 404, with a fixed text (0fee201). api/logs.py reads the run logs through the same walk and pool, .spectio/nextflow.log first, with escape sequences removed.
  • Store and projector: migration 0006_attempt_files.py adds attempts.work_dir, native_id and logs. Nextflow's _where keeps the work directory only when it is <2 hex>/<30 hex> below the run's; Snakemake's _logs keeps a path only inside the launch directory, from job_info, job_error and the new snakemake.slurm_submitted. Others are counted path_outside_run or bad_native_id in the spool cursor's drops.
  • SDK (spectio_sdk/events.py): SnakemakeSlurmSubmitted (jobid, external_jobid, log) and SnakemakeJobError.aux.external_jobid. The logger plugin does not send them yet (P1 Task 7).
  • The stage freeze (stages/freeze.py, serve.py, sink.py): the file reads of freeze moved into _sources, which runs in a pool of its own (FREEZE_WORKERS = 2, no backlog, 0fee201) with FREEZE_TIMEOUT_S = 10; a Stalled freeze gives no stages and the note "The stage sources were not read in time". Booted.files is the routes' pool, Booted.stage_files the freeze's.
  • Head script (packages/spectio/src/spectio/runner/headscript.py): Nextflow gets NXF_LOG_FILE=<launch dir>/.spectio/nextflow.log and unset NXF_DEBUG; Snakemake on SLURM gets --slurm-keep-successful-logs.
  • nf-spectio (engine-bridges/nf-spectio/src/spectio/SpectioPlugin.groovy): every task event carries trace.workdir and trace.native_id (the full trace still only with onTaskComplete); enableMetrics() returns true.
  • Tests: packages/spectio-server/tests/test_taskfiles.py (new, 23 tests), tests/api/test_taskfiles_api.py (new, 18 tests, 20 cases), tests/test_serve.py::test_boot_freeze_pool_apart, and in packages/spectio/tests/test_bridge_nextflow.py test_metrics_enabled and test_task_events_carry_workdir_and_native_id (engine tests); test_read.py and test_openapi.py updated.
  • Specs, all marked (assumed — pending Thomas, inbox #94); no section number changed:
    • by the step's commits: architecture §13.7 note; api §5 note; api §14.1 two event rows and two drop counters; dev-env §15.3 req. 13 note;
    • by the docs commit: an api §7 note for the three columns, and the api §5 note corrected on how a screen finds n (CR-P3T28-05 r1); in the req. 13 note, "no link for a job file" (CR-P3T28-03 r3) and the run-log route's different error mapping (CR-P3T28-01 r4).
  • P3-M7: the SEC-P3T27-03 row now says closed by this step, and states the 10 s writer bound.
  • Inbox: #94 new; the docs commit corrected rule 3 (two pools, CR-P3T28-04 r4) and added rules 7 and 8.
  • Decision log: 2026-10-31 — each job's files as built.
  • Review record: docs/reviews/code/P3-t28.md.
  • Docs of this task: this page, the p3-t28-job-files figure (tools/diagrams/p3_t28_job_files.py, by diagram-artist) and its manifest entry.

Differences from the plan's interface (Task 28)

P1 is not edited by this step; these go to the next /apply-findings (CR-P3T28-04 r1, CR-P3T28-06 r2).

Plan Code Why
open_confined(root_fd: int, rel, *, work_dir_fd: int) -> BinaryIO open_confined(root: Root, rel, *, work: Root \| None = None); Root is a descriptor and its path the link-target test is on the path's text (no realpath), so the root's path travels with its descriptor; job files pass no work (f362fc5)
list_confined(root_fd, rel, cap) -> Listing list_confined(root: Root, rel, cap, *, work=None) as above
tail(f, max_bytes=65536) as planned, plus read_at, size_of, stat_confined one Range; a listing's states without opening each file
list_task_files(attempt) list_task_files(attempt: AttemptFiles) and read_task_file(attempt, name, *, max_bytes, byte_range) -> Served the attempt row is read on the request thread; the file part runs in the pool and touches no database
"reads go through the bounded thread pool of the logs route" files/pool.py, new: the routes' FilePool() (4 threads, 16 waiting) and the freeze's FilePool(2, 0) the logs route had no pool; SEC-P3T27-03 and SEC-P3T28-06 (r3)
the logger plugin reads the SLURM ID and log path from the submission text the server side only: spool type snakemake.slurm_submitted and job_error.aux.external_jobid P1 Task 7 is not built (brief); follow-up FU-P3-31
the listing {work_dir, files} also task_id, attempt, native_id #94 rule 6
api §7: native_id on scheduler_jobs attempts.work_dir, native_id, logs (as the plan's Files list says) scheduler_jobs is not built; api §7 note
the freeze is not in Task 28 the freeze reads in the file pool the brief gave Task 28 SEC-P3T27-03
test_metrics_enabled(nf_run) in the server tests test_metrics_enabled(nextflow, tmp_path) in packages/spectio/tests/test_bridge_nextflow.py, plus test_task_events_carry_workdir_and_native_id they run the real Nextflow and the plugin (-m engines)
test_not_written_yet, test_cleaned_workdir_410 both, plus test_not_written_yet_listed, test_cleaned_workdir_listed; also one test per review fix
Step 4 on slurm-mini not run P1 Task 22 is not built

Why

  • The prompt: next-phase.md · P3; M8 in P2 §8: "28 (confinement, req. 13) … are security boundaries", track review-code.
  • The decision: each job's files in 0.0.1 (Thomas, 2026-10-28: "Je veux également que ça soit possible de voir les logs globaux et les lots et stout err log sh et autres de chaque job si possible via workdir", log).
  • The order: after the public demo of PR #111, Thomas's "go" set P3-parD, then P1 Tasks 27, 28, 29 and 32 (log, UI parity D). Task 27 is merged (P3-M7); its open SEC-P3T27-03 came to this task.
  • What the code had to choose: inbox #94, assumed under the standing rule of autopilot §10; recorded in the decision log.

Evidence

All re-run by doc-writer on the code head d4f41fc, in the worktree, as root (so the foreign-uid test runs here; on CI it skips, SEC-P3T28-07 r3).

The exit check (P1 Task 28 Steps 2 and 4):

$ uv run --locked pytest packages/spectio-server/tests/test_taskfiles.py packages/spectio-server/tests/api/test_taskfiles_api.py -v
test_taskfiles.py::test_nf_allow_list PASSED
test_taskfiles.py::test_req13_job_file_symlink_in_work_refused PASSED
test_taskfiles.py::test_req13_command_run_redacted PASSED
test_taskfiles.py::test_req13_redact_lexer PASSED
test_taskfiles.py::test_req13_redact_container_env PASSED
test_taskfiles.py::test_req13_redact_fails_closed PASSED
test_taskfiles.py::test_req13_redact_deep_nesting_hidden PASSED
test_taskfiles.py::test_req13_forged_paths_dropped PASSED
test_taskfiles.py::test_req13_long_log_names_dropped PASSED
test_taskfiles.py::test_req13_symlink_fifo_refused PASSED
test_taskfiles.py::test_req13_dir_swapped_mid_request PASSED
test_taskfiles.py::test_req13_published_symlink_served PASSED
test_taskfiles.py::test_req13_symlink_to_home_refused PASSED
test_taskfiles.py::test_req13_foreign_uid PASSED
test_taskfiles.py::test_engine_log_per_run PASSED
api/test_taskfiles_api.py::test_nf_file_tail PASSED
api/test_taskfiles_api.py::test_cleaned_workdir_410 PASSED
api/test_taskfiles_api.py::test_binary_415 PASSED
api/test_taskfiles_api.py::test_req13_os_errors_mapped[36-404-log-not-found] PASSED
api/test_taskfiles_api.py::test_req13_os_errors_mapped[5-503-runner-offline] PASSED
api/test_taskfiles_api.py::test_req13_os_errors_mapped[116-503-runner-offline] PASSED
api/test_taskfiles_api.py::test_req13_job_file_symlink_422 PASSED
api/test_taskfiles_api.py::test_req13_range_capped PASSED
api/test_taskfiles_api.py::test_req13_smk_name_exact PASSED
api/test_taskfiles_api.py::test_req13_file_pool_separate PASSED
api/test_taskfiles_api.py::test_req13_freeze_reads_off_the_writer PASSED
…
============================== 43 passed in 5.84s ==============================

The Nextflow side, with the real engine (Nextflow 26.10.4; its SHA-256 2cbda6e7…1fdec4 equals NEXTFLOW_SHA256 of dev/versions.env):

$ NEXTFLOW=…/nextflow uv run --locked pytest packages/spectio/tests/test_bridge_nextflow.py -m engines -v
test_bridge_nextflow.py::test_metrics_enabled PASSED
test_bridge_nextflow.py::test_task_events_carry_workdir_and_native_id PASSED
…
================== 15 passed, 1 skipped in 161.88s (0:02:41) ===================

The skipped case is test_unwritable_spool_never_fails_run[mode_0500], which cannot hold as root.

Test-first, round 4. With the five source files of f362fc5 (redact.py, api/taskfiles.py, confine.py, serve.py, stages/freeze.py), then restored:

$ git checkout f362fc53 -- <the five files>
$ uv run --locked pytest -q … -k "deep_nesting or deep_wrapper or os_errors_mapped or long_log_names or freeze_pool_apart"
E               RecursionError: maximum recursion depth exceeded
E       OSError: [Errno 5] secret detail: '/secret/path'
E           AttributeError: 'Booted' object has no attribute 'stage_files'
7 failed, 67 deselected in 2.46s
$ git checkout HEAD -- <the five files> && uv run --locked pytest -q … (same -k)
7 passed, 67 deselected in 1.27s

The gate:

$ uv run --locked pytest -q                → 611 passed, 17 deselected in 86.86s
$ uv run --locked ruff check .             → All checks passed!
$ uv run --locked ruff format --check .    → 100 files already formatted
$ uv run --locked mypy packages            → Success: no issues found in 99 source files
$ uv run --locked lint-imports             → Contracts: 4 kept, 0 broken.
$ uv lock --check                          → Resolved 55 packages in 3ms

No web/ file changed: the screens are Task 32's.

Review:

CODE  step=P3-t28  review=approve b/M/m=0/2/14  security=approve b/M/m=1/1/20  rounds=4  simplify=changed -13/+1  tests=pass

The blocker (SEC-P3T28-01 r1) and the three majors are fixed. The review record has the probes of the open redaction findings, re-run on d4f41fc: for example export "TOKEN"=abc1, exec env T=s3 cmd, docker run -eTOKEN=s3 img and #SBATCH --export=ALL,TOKEN=s3cret are still served unchanged, and strip_escapes keeps U+202E.

Docs: python3 tools/docs/check.py and uvx --with mkdocs-material mkdocs build --strict, results in the PR body.

Figures and media

  • Sketch: p3-t28-job-files (top of this page), tools/diagrams/p3_t28_job_files.py, by diagram-artist. Its .command.run panels are the real output of redact_command_run.
  • Requested from media-capture, not made in this step: none. Task 28 adds no screen: the files are drawn by Task 32, which takes app-run-job-files (the task sheet with its files, light and dark, 1440 × 1180, and the phone twin). A cast of curl on the two routes would add little to the Evidence above.

Findings and open questions

Review: docs/reviews/code/P3-t28.md, four rounds; the blocker and every major are fixed (SEC-P3T28-01 r1, CR-P3T28-01 r1, CR-P3T28-02 and SEC-P3T28-02 r2).

Inbox: #94, assumed. Two open options: rule 4, whether a path from an event is also compared against the runs root's realpath (recommended: not until slurm-mini shows what Nextflow reports); rule 8, whether the values in #SBATCH --export= lines are hidden (recommended: yes, with FU-P3-31).

Follow-up FU-P3-31, job files hardening and the plugin side, before the 0.0.1 release (P1 Task 21) at the latest; the redaction items before any shared deployment:

Findings What is left Owner
— the logger plugin sends snakemake.slurm_submitted and job_error.aux P1 Task 7, FU-P3-31
SEC-P3T28-03 (r1, r2, r3), CR-P3T28-03 (r2) the .command.run lexer still serves values: a quoted name part (export "TOKEN"=v), command prefixes (exec, command, builtin, nohup, sudo, env after them), an attached flag (-eT=v, --set-env=T=v), a leading redirection, function f { T=v; }, scheduler directive lines FU-P3-31, inbox #94 rule 8
SEC-P3T28-07 (r1), SEC-P3T28-08 (r3) strip_escapes keeps bidi and format controls and the rest of an 8-bit CSI FU-P3-31
SEC-P3T28-05 (r1, r2) no cap on attempts.logs; quadratic duplicate check on the writer FU-P3-31
SEC-P3T28-07 (r2) .spectio/ compared case-sensitively FU-P3-31
SEC-P3T28-05 (r3) open_confined(…, work=…) would serve a never-served name at the link target P1 Task 31, FU-P3-31
SEC-P3T28-07 (r3) the owner-check tests skip on CI (non-root) FU-P3-31
CR-P3T28-01 (r4), SEC-P3T28-08 (r4) the run-log route maps every OSError to 404 with the system's message; two copies of the mapping FU-P3-31
CR-P3T28-02 (r4) an OSError mapped to 404 is not logged FU-P3-31
SEC-P3T28-07 (r4) a lone surrogate in a recorded name would make the listing a 500 (not reachable through the spool today) FU-P3-31
CR-P3T28-03 (r1), CR-P3T28-05 (r2), CR-P3T28-03 (r4) three tests check less than they say (test_req13_spectio_dir, test_req13_freeze_reads_off_the_writer, test_boot_freeze_pool_apart) FU-P3-31
CR-P3T28-08 (r1) api.taskfiles re-exports FilePool for a test FU-P3-31
SEC-P3T28-06 (r1, r2, r3) a freeze that really stalls holds the writer 10 s and its run keeps no stages stated in req. 13 and on P3-M7; a retry is a behaviour change to decide
CR-P3T28-06 (r1) FU-P3-30's option (b) for two matching overlays, though freeze.py changed here FU-P3-30

Open questions:

  1. The cluster check. No test has read a real SLURM job's .command.log or a Snakemake SLURM log. Task 22's e2e job runs Step 4 for Tasks 28 to 30 together.
  2. The redaction against real wrappers. Req. 13 keeps [to be validated] on the container forms: no wrapper written by Nextflow for Docker, Singularity, Apptainer or Charliecloud has been run through redact_command_run. A slurm-mini run with a container would give one.
  3. Attempt numbers for the screens. No route gives a task's attempt numbers yet (api §5 note); Task 32 needs one, or Task 29's failure summary (attempt_n).