Skip to content

Spike t4-ci — kind and Kueue on GitHub-hosted runners (2026-10-28)

Question. Can pipeline step T4 (next-phase.md, "T4 · kind and Kueue") be done CI-driven on GitHub-hosted runners, which is option (b) of inbox #51? A Claude Code cloud session cannot run kind, because its cgroup layout is hybrid v1 (spike D2). In detail:

  • create k8s-mini and k8s-kueue (dev-environment-and-testing.md §4.2) in a GitHub Actions job;
  • run the synthetic Nextflow pipeline with -profile k8s;
  • bring the pod, Job, event, OOMKilled and Kueue Workload fixtures back through the job log;
  • check the runner and engine-head ServiceAccounts of schedulers-and-monitoring.md §4.

Answer in one line. Yes, feasible. In 2 pushes, every item of T4's prompt ran on ubuntu-24.04 runners, except nf-core/demo, which pipelines-real.yml already covers. A loop costs 4–5 minutes of wall-clock time and no money: the repository is public, and standard runners are free there. The §4 engine-head Roles are not sufficient: both Nextflow versions need pods/status: get. The runner Role cannot read Kueue Workloads.

This is a spike, not T4. Nothing in pipeline.yaml, the specs or the inbox was changed. Thomas has not answered #51.

Setup

  • Branch spike/t4-ci, from autopilot/integration at 504bc62. The files are in spikes/t4-ci/, plus one workflow, .github/workflows/spike-t4-ci.yml. The workflow runs on push to that branch (paths spikes/t4-ci/** and itself) and on workflow_dispatch. It has permissions: contents: read and no secrets. Each job has timeout-minutes: 40. Actions are pinned by SHA, with the pins copied from dev-env.yml, and run: blocks contain no ${{ }}. The fixture names and spikes/t4-ci/ paths this note cites are on branch spike/t4-ci; the fixtures committed with T4 are in tests/fixtures/k8s/.
  • Runner. ubuntu-24.04, image version 20261025.353.1 (from the job log). Measured: 4 CPUs, 15 GB RAM, 86 GB free on /, Docker 28.0.4 on cgroup v2. GitHub documents 4 CPU and 16 GB for standard Linux runners in public repositories, against 2 CPU and 8 GB in private ones, and says that "Use of the standard GitHub-hosted runners is free and unlimited on public repositories" (GitHub-hosted runners reference). The run timing API reported billable: 0 ms for both runs.

  • Pins, read from dev/versions.env and checked with sha256sum the same way dev/bootstrap.sh checks them (spikes/t4-ci/cluster.sh install):

    Tool Version
    kind 0.33.0, node image kindest/node:v1.37.0@sha256:a1ed56cf…
    kubectl 1.37.1
    Nextflow 26.10.4 and 25.10.9

    Kueue is v0.20.3 (KUEUE_VERSION), installed with kubectl apply --server-side of the release's manifests.yaml (release v0.20.3). No checksum is published for that file, so the spike pins the sha256 of the download made on 2026-10-28 (646840e4…, "computed", as in versions.env). The manifest serves kueue.x-k8s.io/v1beta2 and registers the APIService v1beta2.visibility.kueue.x-k8s.io. Its default integrations.frameworks contains batch/job and not pod (manifests.yaml L27147–L27154 at v0.20.3).

  • Clusters (cluster.sh up): kind with the runner's /home/runner/work mounted into every node at the same path (extraMounts).

    • mini: one node (k8s-mini).
    • kueue: a control plane and 2 workers, then Kueue, a ResourceFlavor, two ClusterQueues and two LocalQueues (manifests/kueue-queues.yaml): cq-main has 2 CPUs of quota and cq-tight has 1.
    • Both have the namespace spectio-work with a ResourceQuota (manifests/base.yaml).

What ran

The workflow has three parallel jobs, each with its own cluster.

Job Shape Scenarios (spikes/t4-ci/scenarios.sh)
k8s-mini nextflow mini failures (plain OOM, deadline and exit-3 objects), nf-happy (26.10.4), nf25-happy (25.10.9, run 2), nf-oom (oom_real)
k8s-kueue kueue kueue-wait (3 one-CPU Jobs on the 1-CPU cq-tight), nf-kueue (synthetic pipeline through lq-main)
rbac mini in run 1, kueue in run 2 can-i matrix; the runner's own work done with its token; each Nextflow head run under its own ServiceAccount token with three Role sets
Run Commit Result Wall clock (push to last job end) Jobs (cluster up / scenarios)
40662141233 fdc55d3 success 3 min 49 s (229 s) nextflow 3:45 (57 s / 2:31), kueue 2:00 (74 s / 37 s), rbac 2:29 (50 s / 86 s)
40663207755 d43ef90 success 5 min 09 s (309 s) nextflow 5:07 (49 s / 4:07), kueue 1:57 (62 s / 43 s), rbac 4:32 (70 s / 3:05)
  • Fixed cost of a loop. About 1 minute: installing the tools takes 5–10 s, the kind cluster 37–45 s, and Kueue another 21–28 s until its webhook and visibility APIService answer. The python:3.12 task image (1.6 GB) is pre-pulled on the worker nodes in parallel, in about 20 s.
  • Time to start. The first job started 2–3 s after each push.

  • Scenario times:

    Scenario Time
    failures 43 s
    nf-happy 67–80 s (65 tasks)
    nf25-happy 80 s
    nf-oom 38–39 s
    kueue-wait 1–3 s
    nf-kueue 36–39 s
    rbac 86 s in run 1, 185 s in run 2 (2 Nextflow versions × 3 Role sets × 2 runs)

Results

The k8s profile runs on kind, and its bin/ open item is solved

Open item 6 of the T1 note is closed for kind. The head runs on the runner, outside the cluster, with -profile test,k8s plus -c spikes/t4-ci/nextflow/kind.config. That config does not touch the profile; it adds:

  • k8s.namespace;
  • k8s.cleanup = false, so pods and Jobs stay for the dumps;
  • a pod hostPath volume of /home/runner/work mounted at the same path (PodOptions.groovy L126–L127 at v26.10.4);
  • executor.pollInterval = '2 sec';
  • optionally, the Kueue queue label.

The launch directory, the work directory and the pipeline, including the bin/synth.py link target, all sit under that mount. Every path the head writes into .command.run therefore resolves inside the task pod (task pod spec in fixture nf26-happy-task-pods). This only works where every node can see the runner's disk, which is the case for kind. A real cluster needs the RWX volume of k8s-rwx or Fusion (§4.2).

Run Result
nf26-happy, Nextflow 26.10.4 exit 0, 65 / 65 tasks COMPLETED, one Kubernetes Job per task (65 Jobs)
nf25-happy, Nextflow 25.10.9 exit 0, 65 / 65, bare Pods and no Job
nf26-oom, oom_real on QC_STATS S05 exit 1. The 3 attempts ran with memory limits of 512Mi, 1Gi and 1536Mi (the config's 512.MB * task.attempt). Every pod state.terminated is reason: OOMKilled, exitCode: 137, and the trace shows FAILED 137 at attempts 1–3. This is the first proof on Kubernetes of oom_real, open item 5 of the T1 note.
  • Why 25.10.9 needs the plugin named. Without it, 25.10.9 stops with "Unknown executor name: k8s" (run 1): its log says Plugins default=[] and Extension executors providers=[]. With -plugins nf-k8s@1.2.3 it runs; 1.2.3 is the version that 25.10.9's own log lists among its core plugins. 26.10.4 needs nothing: its log says Extension executors providers=[K8sExecutor] with no plugin download. Where 26.10 bundles the executor is [to be validated: source location].
  • CPU limits. Nextflow sets requests.cpu, requests.memory and limits.memory, and no CPU limit (the cpuLimits option is off by default, K8sConfig.groovy L226 at v26.10.4).
  • Scheduling. On the one-node k8s-mini, 58 of the 65 pods first got FailedScheduling: 0/1 nodes are available: 1 Insufficient cpu (each task requests 1 CPU), and then ran. That event is normal back-pressure, not an error. Spectio must not show it as a failure.

Plain failure objects (k8s-mini row of §4.2)

Fixtures k8s-failures-pods, k8s-failures-jobs and k8s-failures-events. Identical in both runs:

Object Pod Job
fx-oom (busybox tail /dev/zero, limit 32Mi, backoffLimit: 0) phase: Failed, state.terminated.reason: OOMKilled, exitCode: 137 Failed, reason BackoffLimitExceeded
fx-exit3 (exit 3, backoffLimit: 1) 2 pods, reason: Error, exitCode: 3 Failed, reason BackoffLimitExceeded
fx-job-deadline (Job activeDeadlineSeconds: 10) deleted by the Job controller (event SuccessfulDelete) Failed, reason DeadlineExceeded ("Job was active longer than specified deadline")
fx-pod-deadline (Pod activeDeadlineSeconds: 10) status.reason: DeadlineExceeded, but the container's terminated.reason is Error, exit 137 —
  • No OOM event. The OOM kill produced no event in the namespace: it appears only in the pod's container status. A runner that keeps only events would miss it, so §4's rule to keep "the last terminal state of each pod" is necessary.
  • Failed Jobs carry two conditions. Both FailureTarget and Failed are present, with the same reason.

Kueue: Workloads waiting for quota, with positions

kueue-wait (fixtures kueue-workloads, kueue-pending-lq-tight, kueue-pending-cq-tight, kueue-clusterqueue-cq-tight, kueue-localqueue-lq-tight, kueue-jobs and kueue-events).

The Workloads.

  • The admitted one: QuotaReserved=True, Admitted=True, PodsReady.
  • The two waiting ones: QuotaReserved=False, reason WaitingForQuota, message "couldn't assign flavors to pod set main: insufficient unused quota for cpu in flavor default-flavor, 1 more needed"; Admitted=False, reason NoReservation.

The visibility API. GET /apis/visibility.kueue.x-k8s.io/v1beta2/namespaces/spectio-work/localqueues/lq-tight/pendingworkloads returns a PendingWorkloadsSummary with positionInLocalQueue and positionInClusterQueue 0 and 1, plus priority, localQueueName and the owner Job. The same holds for the ClusterQueue path. This confirms §4's version rule for v0.20 (v1beta2).

Two timing facts for the runner:

  • The visibility API knows first. Run 1 dumped 1 s after the Jobs were created. One waiting Workload already had a visibility position but no QuotaReserved condition yet, and the ClusterQueue status still said pendingWorkloads: 0. Run 2 waits for a settled state before dumping. During nf-kueue, 3 of the 4 Workloads in the visibility answer also had only PodsReady=False/WaitForStart (fixture nf26-kueue-pending-workloads). So a runner that reads only Workload conditions shows "queued" later than one that reads the visibility API.

  • Kueue manages Nextflow 26.10 tasks with no change to Nextflow.

    • The queue label set through k8s.pod ends up on the Job (kueue.x-k8s.io/queue-name: lq-main in metadata.labels). This confirms §4's claim that Nextflow builds the Job from the pod metadata.
    • Each of the 14 Jobs got Suspended → CreatedWorkload → Resumed → FinishedWorkload events.
    • Up to 5 Nextflow Workloads waited at the same time on the 2-CPU cq-main, and the run still ended with exit 0 in 36–39 s.

    Nextflow 25.10's bare Pods were not tried through Kueue (its pod integration is off by default, above).

ServiceAccounts (§4)

manifests/rbac-spec.yaml copies the §4 Roles verbatim. Each check below ran with the ServiceAccount's own bound token (kubectl create token), not only kubectl auth can-i --as.

The head runs on the runner with a kubeconfig that holds the head ServiceAccount's token. For the API server this is the same identity as a head pod with that ServiceAccount. Each Role set got two runs of the 2-sample sheet (nextflow/mini-samplesheet.csv, 14 tasks):

  • a success run, which on 26.10 goes through lq-main, so its Jobs wait suspended with no pod;
  • a tool-error run (ALIGN S07 exits 1), so the head kills the tasks still in flight.

spikes/t4-ci/api_calls.py turns the trace into the list of calls made. It reads the -trace nextflow.k8s.client lines ("[K8s] API response METHOD path", K8sClient.groovy L672–L674) and the refusals ("returned an error code=403", L664–L668).

Head §4 Role (spec) Calls actually made (run 2) Minimal Role, verified
Nextflow 26.10.4 not sufficient: success run exit 1 in 7 s, pods/status get refused (403), pods delete refused (403); Jobs left behind ("Unable to delete job") jobs create, delete; jobs/status get (28, only under Kueue); pods list, delete; pods/status get jobs: create, delete; jobs/status: get; pods: list, delete; pods/status: get: success exit 0, error run exit 1, 0 refusals
Nextflow 25.10.9 not sufficient: exit 1 in 8 s, pods/status get refused (403) pods create, delete; pods/status get pods: create, delete; pods/status: get: exit 0 / 1, 0 refusals

  • Where the extra calls come from:

    • pods/status: every task poll reads /api/v1/namespaces/<ns>/pods/<name>/status (K8sClient.groovy L327–L333 at v26.10.4).
    • jobs/status: read while a Job has no pod yet, which is the Kueue case (jobStateFallback0, L411–L445).
    • pods: delete on 26.10: jobDelete deletes the Job's pods one by one before the Job (L234–L262).

  • Never used by the head: watch, plain get on pods or Jobs, list on Jobs, and pods/log. Nextflow polls; it does not watch. The §4 grants of these are therefore unnecessary. Each verb in the minimal Role was used in at least one of the runs.

  • What the minimal Role does not cover: code paths that these runs did not hit, for example k8s.fetchNodeName (getNodeOfPod) or nextflow kuberun.

  • Runner Role: the §4 Role is sufficient for its listed work and refuses everything outside it. With its token:

    • allowed: create the head Job (with serviceAccountName: nf-head-26), wait for it, list its pods, read its log, read events, ResourceQuotas and LimitRanges, and delete it;
    • refused (403): create pods, get secrets, create pods/exec and list nodes;
    • also refused: workloads.kueue.x-k8s.io (Forbidden) and the visibility pendingworkloads (Forbidden). §4 says that the runner watches Workload objects, but its Role does not grant that. The visibility rule is listed there as optional, so that refusal is expected.

  • Task pods get a token. The task pods run as the namespace's default ServiceAccount, with its token mounted (kube-api-access-* projected volume in nf26-happy-task-pods). Whether Spectio should set automountServiceAccountToken: false for task pods is a question for the security review [to be validated].

How fixtures come back

  • Log blocks. Every fixture is printed once in the job log, between =====SPECTIO-FIXTURE-BEGIN name=… bytes=… items=kept/total===== and =====SPECTIO-FIXTURE-END name=…===== (spikes/t4-ci/fixture.py):

    • managedFields and the last-applied annotation are dropped;
    • each block is capped at 60 kB, by dropping items from the end and recording _spike.truncated.
    • Run 2 printed 41 blocks, from 49 B to 36 kB; one was truncated (8 of 14 finished Workloads). The three job logs total 0.96 MB.

  • Back to files. spikes/t4-ci/extract_fixtures.py LOG OUT_DIR strips the runner timestamps and writes OUT_DIR/<name>.json. On the logs of runs 1 and 2 it wrote 31 + 41 files with 0 broken blocks.

  • Getting the logs. From this cloud session, curl https://api.github.com/repos/weber8thomas/spectio/actions/jobs/<id>/logs (through the agent proxy) returned the full plain-text log once each job had finished (HTTP 302 to a signed blob URL, then 200). The MCP get_job_logs tool works too, but returns a tail.
  • Artifacts work too. GET /repos/weber8thomas/spectio/actions/artifacts/<id>/zip also returned 200 with the zip (tested on an existing scenarios-snakemake-9.27.1 artifact), so they are a second path. This contradicts #51's "cannot download artifacts" for this session. The proxy answers only for repositories attached to the session; for kubernetes-sigs/kueue it said "GitHub access to this repository is not enabled for this session". Whether every cloud session gets this is [to be validated].
  • For T4 proper. Give each fixture a name with the tool version (<scenario>-k8s1.37.0-kueue0.20.3.json), then extract and normalise it (drop UIDs, names with hashes, timestamps and IPs), as T1 does for the trace golden files.

What did not work, and why

  • Nextflow 25.10.9 with -profile k8s as it stands: "Unknown executor name: k8s" (above). The profile, or Spectio's generated config, must name nf-k8s for 25.10.
  • The §4 engine-head Roles: refused on pods/status on both versions, and on pods delete on 26.10 (above). The 26.10 run under the spec Role also leaked Jobs, because jobDelete could not delete their pods.
  • The first Kueue dump was taken too early (above). Fixed in run 2 by waiting for a settled state.
  • Not run:
    • nf-core/demo -profile test,docker: pipelines-real.yml already runs it in its PR tier (corpus.yaml: demo, tier: pr) and nightly, with -profile test,docker (corpus CI note: pass in 4–6 min).
    • k8s-rwx and SeaweedFS, and the Snakemake Kubernetes executor: these are §4.2 items, but not in T4's prompt.
    • Kueue preemption, and Kueue VisibilityOnDemand limits.
    • The previous Kueue minor (KUEUE_PREVIOUS_VERSION 0.19.10).

Result

Option (b) for T4: feasible.

  • Coverage. In two push iterations, a GitHub-hosted ubuntu-24.04 runner (cgroup v2) did everything T4's prompt asks, except nf-core/demo, which is covered elsewhere:

    • k8s-mini and k8s-kueue, with Kueue v0.20.3, a ClusterQueue and a LocalQueue;
    • the synthetic pipeline on Kubernetes, on Nextflow 26.10.4 and 25.10.9;
    • real OOMKilled, with exit 137;
    • pod, Job and event fixtures;
    • Workloads waiting for quota, with their visibility positions;
    • both §4 ServiceAccounts, tested with real tokens.

  • Cost per loop. 4–5 minutes from push to result, three jobs in parallel. About 1 minute of that is fixed (cluster and Kueue); the rest is the scenarios. $0, because the repository is public. On a private repository each loop would use about 12 runner-minutes from the free allowance, on 2-CPU runners. That is [to be validated]: this spike did not measure 2-CPU runners.

  • What (b) costs compared with (a).

    • A local loop is seconds, not minutes.
    • CI cannot be inspected interactively: no kubectl into a live cluster, so every question becomes a scripted dump and a push.
    • Fixtures come back through the log or as an artifact, and both can be fetched from this session.

  • For T3 and T4. T4 is a small number of well-defined captures, so (b) is practical: this spike needed two loops. T3 has more failure modes per loop, so it would likely need more iterations, but the same pattern applies [to be validated].

Findings for TRIAGE (specs not edited)

  1. §4 engine-head Roles are not sufficient (schedulers-and-monitoring §4). Replace them with the minimal sets verified above:

    • 26.10: jobs: create, delete; jobs/status: get; pods: list, delete; pods/status: get;
    • 25.10: pods: create, delete; pods/status: get.

    The "[to be validated on k8s-mini]" note on 25.10 needing pods: create is now verified.

  2. The runner Role lacks workloads.kueue.x-k8s.io: get, list, watch, which §4's "What the runner watches" needs when Kueue is installed.

  3. Nextflow 25.10 needs nf-k8s named for the k8s executor (-plugins nf-k8s@1.2.3, or a plugins {} block in the generated config). The k8s profile of the synthetic pipeline does not name it.
  4. The OOM kill is visible only in the pod status, with no event; a Job deadline deletes its pod. Both support §4's rule that the runner keeps terminal pod states itself (§4.2 golden fixtures).
  5. Kueue's visibility API shows a pending Workload before its QuotaReserved=False condition is set, and before the ClusterQueue's pendingWorkloads counter includes it. The "queued" state should come from the visibility API when the runner can read it.
  6. FailedScheduling: Insufficient cpu is ordinary back-pressure on a small cluster (58 of 65 pods on k8s-mini). Not a failure.
  7. Task pods get the namespace's default ServiceAccount token. For the security review [to be validated].
  8. Inbox #51 says the session "cannot download artifacts". That is not what this session saw: through the proxy, the GitHub REST API returned job logs and artifacts for this repository.