Skip to content

T4 · kind and Kueue

Done 2026-10-28 on autopilot/T4, CI-driven (inbox #51 option (b), assumed) · k8s-mini up in 58 s and k8s-kueue in 66 s on ubuntu-24.04 · synthetic Nextflow pipeline with -profile k8s green on 26.10.4 and 25.10.9, each head under its own ServiceAccount token · 10 fixtures, 106 kB · the §4 Roles' insufficiency, measured in the spike, is confirmed by replacement Roles that see 0 refusals (403) · task pods get no token · 4 CI runs, the last green in 3 min 37 s · review-code approve

T4 figure: a GitHub runner builds kind clusters and mounts the repository into every node at its own path. k8s-mini, one node, holds the failure objects (an OOMKilled pod with exit 137, a Job and a Pod over their deadline, a Job whose pod exits 3) and the bare Pods of Nextflow 25.10; 26.10 makes one Job per task. k8s-kueue, two workers and Kueue: cq-tight has 1 CPU, one Workload is admitted and two wait, at positions 0 and 1 of the visibility API. ServiceAccounts: under the §4 Roles, nf-head-26 and nf-head-25 exit 1 on refused calls and the runner cannot read Workloads; tests/k8s/rbac.yaml adds pods/status get, pods delete and jobs/status get for 26.10, pods/status get for 25.10, and workloads get, list, watch for the runner; T4 runs see 0 refusals, and task pods run as default with no token T4 figure: a GitHub runner builds kind clusters and mounts the repository into every node at its own path. k8s-mini, one node, holds the failure objects (an OOMKilled pod with exit 137, a Job and a Pod over their deadline, a Job whose pod exits 3) and the bare Pods of Nextflow 25.10; 26.10 makes one Job per task. k8s-kueue, two workers and Kueue: cq-tight has 1 CPU, one Workload is admitted and two wait, at positions 0 and 1 of the visibility API. ServiceAccounts: under the §4 Roles, nf-head-26 and nf-head-25 exit 1 on refused calls and the runner cannot read Workloads; tests/k8s/rbac.yaml adds pods/status get, pods delete and jobs/status get for 26.10, pods/status get for 25.10, and workloads get, list, watch for the runner; T4 runs see 0 refusals, and task pods run as default with no token

The §4 refusals in the figure come from the spike's runs under the spec Roles (note, ServiceAccounts); the rest comes from the T4 runs below.

What changed

The spike t4-ci covered T4's scope; this step keeps what it proved and drops the rest.

  • Cluster profiles, dev-environment §4.2:
    • dev/k8s/cluster.sh up mini|kueue, down mini|kueue. k8s-mini is one kind node; k8s-kueue is a control plane and 2 workers, with Kueue KUEUE_VERSION. Both get the namespace spectio-work with a ResourceQuota, and its default ServiceAccount does not mount a token in pods (dev/k8s/base.yaml).
    • Kueue comes from the release's manifests.yaml, checked against KUEUE_MANIFEST_SHA256. It is set up with two ClusterQueues and two LocalQueues (dev/k8s/kueue-queues.yaml); the script fails if they are missing after the retries, and waits until the visibility API answers.
    • The tools come from dev/bootstrap.sh, at the pins of dev/versions.env. The task image is read from the k8s profile of the synthetic pipeline and pre-pulled on each node.
    • The repository is mounted into every node at its own path, so that task pods can see the pipeline and its work directory under .scenarios/ (newly gitignored).
  • dev/versions.env: two new pins:
    • KUEUE_MANIFEST_SHA256, computed, because no checksum is published;
    • NF_K8S_25_10_VERSION=1.2.3.
  • Scenarios, tests/k8s/scenarios.sh. Each scenario fails on an unexpected outcome.
    • failures: one OOMKilled pod, a Job and a Pod over their deadline, and a Job whose pod exits 3.
    • nf25 and nf26: the synthetic pipeline with -profile test,k8s plus tests/k8s/kind.config. Each runs twice under the head's own token: a happy run that must exit 0, and a tool-error run that must exit 1. No API call may be refused (403). Each grant that neither run used is listed as unused-grants (tests/k8s/api_calls.py). The task pod must have no kube-api-access-* volume, and the head's ServiceAccount must be denied the calls of the deny list.
    • kueue-wait: one Workload admitted and two waiting, with their visibility positions.
    • runner: the runner's work done with its own token, then the deny list plus create pods, which must all be denied.
    • The ServiceAccount kubeconfigs are written to a private temporary directory outside the repository (task pods mount the repository), and never hold the admin key.
  • RBAC, tests/k8s/rbac.yaml: the replacement Roles, which differ from §4 (see Findings).

  • CI, .github/workflows/pipelines-k8s.yml, two parallel jobs:

    • mini: failures nf25;
    • kueue: kueue-wait nf26 runner.

    It runs on a push to any branch that touches its paths, nightly, and on manual dispatch. It has permissions: contents: read, no secrets, a 30-minute timeout, actions pinned by SHA, and no ${{ }} in run:. Its fixtures go to the artifacts k8s-fixtures-mini and k8s-fixtures-kueue. Like dev-env.yml, ci.yml does not call it, so it is not part of ci-ok. - Fixtures, tests/fixtures/k8s/: 10 JSON files, 106 kB, byte-identical to the artifacts of run 40672184666. managedFields and the last-applied annotation are removed. A _fixture object records the Kubernetes and Kueue or Nextflow versions and the source run.

    File Holds
    nf-26.10.4-task-job.json, nf-26.10.4-task-pod.json, nf-26.10.4-task-events.json QC_STATS S01 on 26.10.4: its Job, its pod and their events
    nf-25.10.9-task-pod.json, nf-25.10.9-task-events.json the same task on 25.10.9, which creates a bare Pod and no Job
    failures-pods.json, failures-jobs.json, failures-events.json OOMKilled exit 137; pod DeadlineExceeded; Job Failed with BackoffLimitExceeded or DeadlineExceeded; exit 3
    kueue-workloads.json, kueue-pending-workloads.json three Workloads (one admitted, two waiting) and the visibility answer, positions 0 and 1

  • Spec rows, dev-environment: the k8s-mini row of §4.2 and the pipelines-k8s.yml row of §11, which this step builds, now say what was built and what is still planned.

  • Research note, copied from spike/t4-ci as evidence: Spike t4-ci, added to the nav.
  • Not done here:
    • nf-core/demo -profile test,docker, which pipelines-real.yml already runs in its PR tier;
    • SeaweedFS and the Snakemake Kubernetes executor, k8s-rwx, Kueue preemption, and the previous Kueue minor: none of them is in T4's prompt.

Why

Prompt: next-phase.md · T4. It asks for the k8s-mini and k8s-kueue profiles, the synthetic pipeline with -profile k8s, the pod, Job, event and Workload objects as fixtures, and a check of the §4 ServiceAccounts. kind cannot run in a Claude Code cloud session, so the loop runs in CI: option (b) of inbox #51, assumed — pending Thomas. No decision-log entry: the step decided no behaviour; the §4 differences wait for TRIAGE3.

How to test

On a machine whose Docker runs on cgroup v2 (not a Claude Code cloud session), from the repository root:

bash dev/bootstrap.sh && bash dev/k8s/cluster.sh up kueue && \
  for s in kueue-wait nf26 runner failures nf25; do bash tests/k8s/scenarios.sh "$s" || break; done

Each scenario prints a RESULT line and exits 1 on an unexpected outcome. Fixtures land in .scenarios/k8s/fixtures/. bash dev/k8s/cluster.sh down kueue removes the cluster. In CI, push a change under tests/k8s/, or dispatch pipelines-k8s.yml once it is on main.

Evidence

CI runs of pipelines-k8s.yml on autopilot/T4:

Run Commit Result Wall clock
40668895003 42ac189 mini green. kueue red: kueue-wait waited 2 minutes for a second QuotaReserved=False condition that never came 4 min 51 s
40669626241 4b95db9 kueue red: one jobs/status read answered 404 (the Job was already deleted), and the check counted it as a refusal 3 min 35 s
40670116723 9f516a4 green; only 403 counts as a refusal 3 min 08 s
40672184666 7fe53af green, after the round-1 review fixes: no token in task pods, the wider deny list. The fixtures come from this run 3 min 37 s
40673184674 1e3aa52 red: after merging autopilot/integration, cluster.sh found no task image digest in the k8s profile (#32 moved it to params.python_image) 25 s
40673827899 4c9c64c mini red: the heads' deny check, now run with the head's own token (CR-T4-05), reported create pods/exec allowed to nf-head-25; can-i had read exec as a pod name (CR-T4-08) 3 min 19 s
40674240654 007683c green, the final code: every deny check with --subresource, each ServiceAccount with its own token 3 min 38 s

Timings in run 40672184666:

Job Bootstrap Cluster up Scenarios
mini 14 s 58 s (kind 48 s) 2 min 11 s: failures 43 s, nf25 87 s (happy run 70 s, error run 16 s)
kueue 14 s 66 s (kind 40 s, Kueue 24 s) 80 s: kueue-wait 2 s, nf26 73 s (54 s and 18 s), runner 5 s

Result lines of run 40672184666:

RESULT failures: [{"pod":"fx-exit3-dh8nb","reason":null,"terminated":{"reason":"Error","exitCode":3}},{"pod":"fx-exit3-qswld","reason":null,"terminated":{"reason":"Error","exitCode":3}},{"pod":"fx-oom-hc7wn","reason":null,"terminated":{"reason":"OOMKilled","exitCode":137}},{"pod":"fx-pod-deadline","reason":"DeadlineExceeded","terminated":{"reason":"Error","exitCode":137}}]
RESULT nf25: happy exit=0 error exit=1 {"refused":[],"used":["pods create","pods delete","pods/status get"]} unused-grants=[]
RESULT nf26: happy exit=0 error exit=1 {"refused":[],"used":["batch:jobs create","batch:jobs delete","batch:jobs/status get","pods delete","pods list","pods/status get"]} unused-grants=[]
RESULT kueue-wait: [{"name":"job-fx-kq-1-04f50","positionInLocalQueue":0,"positionInClusterQueue":0},{"name":"job-fx-kq-3-7e5a5","positionInLocalQueue":1,"positionInClusterQueue":1}]
RESULT runner: sufficient; 9 out-of-Role calls denied
  • refused lists every refused call with its HTTP code; only 403 fails the check. In run 40670116723 the nf26 list held {"call":"batch:jobs/status get","code":404,"count":1}, a read racing the Job's deletion, which the check ignored; in run 40672184666 it is empty.
  • Runner, allowed with its own token: create the head Job (as nf-head-26), wait for it, list its pods and read its log; read events, quotas and limit ranges, Workloads, and the visibility answer; delete the Job.
  • Denied, to the runner and to both heads (kubectl auth can-i, each with its own token, in spectio-work): get secrets, list nodes, create pods --subresource=exec, create serviceaccounts --subresource=token, create rolebindings, impersonate serviceaccounts, create jobs -n default, create pods -n default; and create pods to the runner and nf-head-26 (nf-head-25 needs it).
  • No token in task pods: no fixture has a kube-api-access-* volume; in run 40670116723 every pod had one.

Review (the review record):

CODE  step=T4  review=approve b/M/m=0/1/6  security=approve b/M/m=0/0/6  rounds=3  simplify=notrun -0/+0  tests=pass
  • code-reviewer and security-reviewer: approve in round 1, with minors only. Six minors are fixed in 7fe53af, one is recorded below (CR-T4-03), and two go to TRIAGE3 (SEC-T4-03, SEC-T4-05). Round 2 reviewed 7fe53af and found that the heads' deny check ran in namespace default and proved nothing (CR-T4-05, major) plus two minors; fixed in 955a038, approved in round 3 (review record).
  • shellcheck -x and bash -n pass on dev/k8s/cluster.sh and tests/k8s/scenarios.sh; the fixes were then proved by run 40672184666.

Docs gate: python3 tools/docs/check.py and mkdocs build --strict, 0 failures (see the step report).

Figures and media

  • t4-kind-kueue sketch, light and dark, at the top of this page: python3 tools/diagrams/t4_kind_kueue.py.
  • No cast or screenshot: cluster.sh and scenarios.sh are development scripts with no user flow, and How to test shows the commands and their RESULT lines.

Findings and open questions

Differences with the specs, for TRIAGE3; the specs are not edited. The insufficiency of the §4 Roles was measured in spike runs 40662141233 and 40663207755. scenarios.sh tests the replacement Roles of tests/k8s/rbac.yaml: T4 runs show them sufficient (0 refusals, 403) and, for the two heads, minimal (unused-grants=[]). The runner Role's minimality is not measured, because there is no runner code yet: persistentvolumeclaims: get and watch on pods, events and Workloads are granted and never used, and the visibility rule is narrowed to get, against get, list, watch in §4.

  1. Nextflow 26.10 engine-head Role (schedulers §4): not sufficient and over-broad; note, Findings item 1.
  2. Nextflow 25.10 engine-head Role (§4): lacks pods/status: get; closes "25.10 needing pods: create"; note item 1.
  3. Runner Role (§4): lacks workloads.kueue.x-k8s.io: get, list, watch; note item 2.
  4. Nextflow 25.10 needs nf-k8s named (§4 and the k8s profile); note item 3.
  5. "Queued" comes from the visibility API, not from Workload conditions (§4). The visibility API listed a pending Workload whose QuotaReserved=False condition came late or never:
    • run 40672184666: job-fx-kq-1 had only PodsReady=False/WaitForStart, at position 0;
    • run 40669626241: job-fx-kq-3 had only PodsReady=False/WaitForStart, at position 1;
    • run 40668895003: the second pending Workload had no QuotaReserved=False after 2 minutes.
  6. An OOM kill produces no event, only state.terminated in the pod; note item 4.
  7. The head can get a 404 on jobs/status when it reads a Job that it is deleting. That is not a permission refusal, so a check must not count it as one.
  8. The "Required checks" paragraph (dev-environment §11) says the pipeline workflows are reusable workflows that ci.yml calls. pipelines-k8s.yml is standalone (push with a path filter, nightly, manual), so it is outside ci-ok. The pipelines-k8s.yml row of §11 now says so; it no longer lists helm/kind-action (the build uses dev/bootstrap.sh and dev/k8s/cluster.sh), and nf-core/demo stays in pipelines-real.yml. The helm install of the chart waits for the chart.
  9. The k8s-mini row (§4.2) listed SeaweedFS, the Snakemake Kubernetes executor and a runner Role. SeaweedFS and the Snakemake executor are not built: T4's prompt does not ask for them, and the row now marks them as planned. The runner Role is not installed by up mini: the scenarios apply tests/k8s/rbac.yaml themselves (CR-T4-03).
  10. Other spike findings: FailedScheduling: Insufficient cpu is back-pressure (note item 6); task pods mounted the default token (note item 7), fixed here (SEC-T4-02); inbox #51's "cannot download artifacts" does not hold (note item 8), and this step's fixtures came from artifacts.

Inputs for TRIAGE3 from the security review (not fixed here):

  • SEC-T4-03: whoever can create Jobs or Pods in the shared spectio-work can run as any ServiceAccount there, mount a hostPath, and delete another run's pods. Proposal: one namespace per run or per user, with PodSecurity restricted and a policy that pins serviceAccountName.
  • SEC-T4-05: the nf-k8s 1.2.3 plugin is fetched by version, the Kueue controller image is referenced by tag, and the whole bootstrap runs in the job. Proposal: pin the plugin by checksum and the Kueue image by digest, and bootstrap only the tools the job uses.

Open question. Should pipelines-k8s.yml join ci.yml and ci-ok? It takes about 3 to 4 minutes, but kind adds a flake risk.