Skip to content

D2 · What a cloud session can run

Done 2026-10-26 · spike commit 217a92c · 8 items tested: 5 work, 1 partly, 2 do not · 8 follow-ups, 6 open questions (2 for Thomas)

What changed

  • New research note: docs/research/spike-cloud-env-2026-10-26.md. It holds:
  • The note was committed on autopilot/integration in ad6bc9b, with its entries in mkdocs.yml (Research) and docs/map.md (Spikes).
  • The spike branch keeps the rest. spike/cloud-env, commit 217a92c ("spike(cloud-env): what a cloud session can run (D2)"), pushed as origin/spike/cloud-env (files at 217a92c). It has 21 files, 1 856 lines:
    • spikes/cloud-env/LOG.md: every command, with its exit code, wall time and output;
    • spikes/cloud-env/fixtures/: 10 raw outputs with versioned names, for example sind-0.13.0-doctor-cgroupv1.txt and domains-proxy-2026-10-26.tsv;
    • the scripts and inputs: probe-domains.sh, redirect-hosts.sh, pw-smoke.js, compose/compose.yaml, nf-hello/main.nf, smk-hello/Snakefile, mmd/.
  • Figure and cast, committed on autopilot/integration in 5d8ef54:
    • tools/diagrams/cloud_env.py and its outputs docs/assets/diagrams/cloud-env_{light,dark}.{svg,png};
    • tools/media/casts/cloud-env.yaml, its output docs/assets/casts/cloud-env.cast, and a cloud-env entry in docs/assets/media.yaml.
  • No spec was edited. The proposed spec changes wait for an apply step.

Why

The dev-environment spec plans for agents to work in Claude Code cloud sessions, and for sind and kind to give real SLURM and Kubernetes clusters for tests. Nobody had checked what such a session can run. D3 (bootstrap and doctor scripts) and the cluster steps T3 and T4 depend on the answer.

Prompt: next-phase.md · D2. It runs as an autopilot step in a git worktree: decision log, 2026-10-26, autopilot. No decision was taken in this step.

Evidence

How the spike ran:

Item Value
Where a Claude Code cloud session (Firecracker VM, Ubuntu 24.04.5, kernel 6.18.44, user root), in the worktree .claude/worktrees/wf_839a7809-2f8-1/ on branch spike/cloud-env
Time box 60 minutes; about 27 used (21:10Z to 21:37Z)
Stop rule an item stops after two failed attempts. sind and kind both stopped there.
Installs user-level, under spikes/cloud-env/.tools/ (git-ignored, 1.9 G at the end), plus 676 MB of puppeteer Chrome in ~/.cache/puppeteer
Resources 4 vCPU, 16 091 MiB, no swap, memory pressure total=0; 22 G of disk left at the end
Checks run by the worker mkdocs build --strict passes and tools/docs/check.py reports 0 failures

Result, from the note's result table:

Item Works Decisive limit
docker run yes dockerd is not running at start and must be started by hand in each session; Docker Hub answered 429 after about 6 anonymous pulls
docker compose yes same as docker run
Nextflow 26.10 yes installed from a pinned release asset; releases/latest returns 403
Playwright + Chromium yes the pre-installed 1.63.0 only
mermaid-cli 11 yes, with two settings as root it needs --no-sandbox; reuse the pre-installed Chromium
Snakemake 9 + SLURM plugin partly no sacct and no cluster
sind no Docker runs on cgroup v1
kind no the control plane never comes up

Decisive lines from spikes/cloud-env/LOG.md (branch spike/cloud-env; long paths shortened to $T):

Docker runs, on cgroup v1 (21:11Z, exit 0):

$ docker info 2>&1 | grep -E "Server Version|Cgroup|CPUs|Total Memory"
 Server Version: 29.9.2
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 CPUs: 4
 Total Memory: 15.71GiB
$ docker run --rm hello-world 2>&1 | grep -E "Hello|working"
Hello from Docker!
This message shows that your installation appears to be working correctly.

sind refuses cgroup v1 (21:14Z, exit 1). The cgroup2 mount has only hugetlb (21:15Z):

$ $T/bin/sind version; $T/bin/sind doctor
sind 0.13.0
✗ cgroupv2: Docker runs containers on cgroup v1 (sind requires cgroupv2)
21:14:53.643 ERRO checks failed: cgroupv2
$ cat /sys/fs/cgroup/unified/cgroup.controllers
hugetlb

Attempt 2 ran dockerd in a private mount namespace with cgroup2 at /sys/fs/cgroup, then with nsdelegate (21:17Z, exit 1):

21:17:30.403 ERRO node controller: creating container controller: exit status 1: Error response from daemon: NanoCPUs can not be set, as your kernel does not support CPU CFS scheduler or the cgroup is not mounted

kind, attempt 2 with kindest/node:v1.34.0 --retain (21:25Z, 4 min 10.8 s). Every static pod failed to get a sandbox:

error: error execution phase wait-control-plane: failed while waiting for the control plane to start: [kube-scheduler check failed at https://127.0.0.1:10259/livez: ... connect: connection refused, ...]
     21 "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-scheduler-spx-d2-control-plane_kube-system(...)\" with CreatePodSandboxError: ...
     19 "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-apiserver-spx-d2-control-plane_kube-system(...)\" with CreatePodSandboxError: ...

Nextflow from a pinned asset, then a local 4-task run (21:26Z and 21:27Z, exit 0):

$ curl -sS -o /dev/null -w "%{http_code} %{redirect_url}\n" https://github.com/nextflow-io/nextflow/releases/latest
403
$ curl -sS -fL -o bin/nextflow https://github.com/nextflow-io/nextflow/releases/download/v26.10.0/nextflow-26.10.0-dist
$ bin/nextflow -version 2>&1 | grep -E "version|Nextflow"
      version 26.10.0 build 14055
$ $T/bin/nextflow run ../../nf-hello/main.nf -with-trace trace.txt
[SUCCESS] completed=4 failed=0 cached=0
real    0m4.600s

Snakemake 9.27.1 with the SLURM plugin 2.8.0: the local run works, a real SLURM run cannot submit (21:30Z; the error is from fixtures/snakemake-9.27.1-slurm-2.8.0-no-cluster.log):

$ $T/bin/snakemake -s $S/Snakefile --cores 2 --quiet rules
5 of 5 steps (100%) done
$ timeout 120 $T/bin/snakemake -s $S/Snakefile --executor slurm --jobs 3
    message: Job submission failed with exception: [Errno 2] No such file or directory: 'sacct'
WorkflowError:
At least one job did not complete successfully.

mermaid-cli as written in CLAUDE.md fails as root, and works with a puppeteer config (21:32Z):

$ npx -y @mermaid-js/mermaid-cli@11 -i $D/mmd/flow.mmd -o $D/.tools/flow.svg 2>&1 | head -8
Error: Failed to launch the browser process:  Code: 1
[1026/213203.481207:ERROR:content/browser/zygote_host/zygote_host_impl_linux.cc:103] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
$ npx -y @mermaid-js/mermaid-cli@11 -p $D/mmd/puppeteer-config.json -i $D/mmd/flow.mmd -o $D/.tools/flow.svg
Generating single mermaid chart
$ npx -y @mermaid-js/mermaid-cli@11 --version
11.17.0

Playwright, pre-installed (21:30Z, exit 0):

playwright 1.63.0
chromium 153.0.8010.12
launch 1499 ms, total 2599 ms
example.com status=200 title="Example Domain"

Network. 43 URLs were probed with probe-domains.sh and none was blocked by the allowlist. This session runs on a broader level than Trusted, so that proves little; a Custom list must be checked again (follow-up 3 in the note).

Exit criterion. D2 is done when its note exists on autopilot/integration, where the next steps can read it:

$ ls docs/research/spike-cloud-env-*.md
docs/research/spike-cloud-env-2026-10-26.md

The wave W1 exit check in pipeline.yaml lists only review files, not this note.

Figures and media

The figure shows the session, what works in it, what does not, the cgroup layout that blocks sind and kind, and the proxy refusals. It also has the measured cold wall times, to scale. Click the figure for the full size.

What a cloud session can run What a cloud session can run

The cast replays the decisive commands: Docker, Nextflow, sind refusing cgroup v1, and mermaid-cli failing then working as root. It does not run the tools again. Stub commands print the output recorded in LOG.md, cut to the decisive lines, with demo paths. Its generator is python3 tools/media/cast.py tools/media/casts/cloud-env.yaml.

Findings and open questions

What the spike found:

  1. No real clusters in cloud sessions. sind and kind do not run, because the VM boots with a hybrid cgroup v1 layout. cpu, memory and pids stay on v1, which the sandbox itself uses, and a session cannot change that. SLURM and Kubernetes scenarios therefore run in CI and on a developer machine (note, result).
  2. Docker must be started in every session. It is installed but not running, and a running process is not kept in the environment snapshot. The note proposes a SessionStart hook that starts dockerd --registry-mirror=https://mirror.gcr.io.
  3. Docker Hub rate limit. It answered 429 after about 6 anonymous pulls; the mirror.gcr.io mirror fixed it.
  4. GitHub repository scope. releases/latest and the API return 403 for repositories not attached to the session. Downloads from a pinned tag pass. Bootstrap must pin every version (note, network).
  5. The mermaid command in CLAUDE.md fails as root. It needs -p with a --no-sandbox puppeteer config. The rule belongs to documentation-system.md (follow-up 7 in the note).
  6. Allowlist and budget. The note gives a 14-line Custom allowlist and a setup-script budget: about 60 s measured, at most 150 s as a target.

Open questions (from the spike report; the note's follow-ups give the details):

# Question Who answers
1 Can Anthropic-hosted cloud VMs boot with cgroup v2? That would make sind and kind work in cloud sessions. Thomas, as a platform request
2 Do sind and kind work on a GitHub Actions ubuntu-24.04 runner and in Codespaces? [to be validated] D3 / CI
3 Is the Custom allowlist enough? Plugin download hosts, the *.pkg.dev regions and the pixi host are not checked. a re-run of probe-domains.sh on Custom
4 Should the project's permission rules let auto mode run nextflow run <repo> and a pinned go install? The classifier refused both. Thomas
5 Does the snapshot keep Docker images, and does a real setup script fit the budget? Not tested. D3
6 The exact kind error was cut off before the node was deleted. optional, only if the answer to 1 is "no"

Next. The two questions for Thomas go to TRIAGE1, then to the inbox. The proposed changes to dev-environment-and-testing.md wait for an apply step. TRIAGE1 and APPLY1 in pipeline.yaml list review rounds only and do not name D2 yet. D3 builds the bootstrap and doctor scripts from this note.