Skip to content

D4 · What runs on every pull request

Built 2026-10-27 on autopilot/D4 · one required aggregate check ci-ok · four security checks on every PR · Claude review on PRs · release-please on an App token, skipped until Thomas sets it up

What runs on a pull request What runs on a pull request

What changed

  • ci.yml, new: the one aggregate check that branch protection requires, besides docs-gate and pr-body, and the place every other check is called from (dev-environment-and-testing.md §11).
    • A changes job reads the PR's merge commit and decides whether the synthetic scenarios or the real-pipeline corpus must run.
    • It calls security.yml, pipelines-synthetic.yml and pipelines-real.yml as reusable workflows.
    • ci-ok needs every gating job, runs with if: always(), and fails only when a job ended failure or cancelled. A skipped pipeline job counts as passing, so no path filter can leave a required check waiting.
    • Until T7, the real-pipeline corpus does not gate ci-ok. It still runs when its paths change and reports its own checks, but it is red on main's nightly run too (run 40463254070: 18 of 23 jobs failed). The first run on this PR showed it: 5 corpus jobs failed, methylseq among them.

  • security.yml and tools/ci/security.sh, new: four checks on every PR and push, with no path filter, and weekly on the whole history. The same script runs on a laptop.

    Check Tool Fails on
    secret scanning gitleaks 8.30.1 any secret in the PR's commits
    dependency audit, Python and JavaScript osv-scanner 2.7.0 any known vulnerability in a lockfile
    SAST, Python and JavaScript semgrep 1.182.1, p/python and p/javascript any finding of severity ERROR
    workflow linters actionlint 1.7.12 and zizmor 1.31.1 any error, and zizmor findings from low up

    Release binaries are checked against their SHA-256 on every run. Exceptions live only in code-owned ignore files; inline gitleaks:allow and nosemgrep comments do not count. - claude-review.yml, new: spec-reviewer reviews PRs that touch docs/**, security-reviewer reviews PRs that change anything else. Findings go to inline comments and one summary comment. - It is advisory and never a required check. - It is skipped for drafts, fork PRs and bots, and while no Claude secret is set. - It checks out the base branch, reads the PR through git only, and cannot edit, push, merge or approve. - The commands Claude runs get a scrubbed environment and PID isolation, which reduce the risk that a prompt injection posts the Claude credential. - release-please.yml: now uses a GitHub App token (actions/create-github-app-token), from the release environment. While RELEASE_APP_ID or RELEASE_APP_PRIVATE_KEY is missing, the job warns and skips its steps. - .github/CODEOWNERS, new: workflows, .claude/, CLAUDE.md, pipeline.yaml, the autopilot, the scanners and their exception files, the release-please config, and auth and secrets code all need Thomas's review, once branch protection asks for it. - Every existing workflow hardened, so that the linters pass: - every action pinned by full commit SHA, with its version in a comment; - persist-credentials: false on every checkout; - no ${{ }} expansion inside a run: script; - a timeout-minutes on every job, and a concurrency group on every top-level workflow. - pipelines-synthetic.yml and pipelines-real.yml became reusable workflows, called by ci.yml. - A real leak, redacted. The first gitleaks run found nf-core's Slack webhook URL in six megatest params files (commit b3c49ea1). The files are redacted, and tools/fixtures/megatests.py now redacts hook_url when it curates. The old commit stays in history, listed in .gitleaksignore, until a history rewrite (inbox #5). - Specs: - dev-environment-and-testing.md: §1 (pins), §8 (Claude review), §11 (workflows as built, the security checks table, Claude review), §12 (App token wiring and its one-time setup); - autopilot.md §8: the protection payload now requires ci-ok, docs-gate, pr-body and code-owner reviews.

Why

Thomas asked for security checks that are "béton" on every PR, and decided on one aggregate required check, ci-ok (inbox #26, R8b DEP-T-001). A path-filtered workflow cannot be required: when a PR skips it, it never reports, and the PR waits for ever. CODEOWNERS covers the paths that steer agents (DEP-T-008). The release-please bot needs an App token, because events created with GITHUB_TOKEN start no workflow (inbox #8).

Prompt: next-phase.md · D4. Decisions: inbox #8 and #26, answered 2026-10-27.

Evidence

All commands ran on 2026-10-27 in the step's worktree, on Linux x86_64 with shellcheck 0.9.0, the version the GitHub runner image ships.

The four checks pass on the branch:

$ bash tools/ci/security.sh secrets
INF 120 commits scanned.
INF no leaks found
$ bash tools/ci/security.sh deps
No issues found
$ bash tools/ci/security.sh sast
Ran 79 rules on 41 files: 0 findings.
$ ZIZMOR_OFFLINE=true bash tools/ci/security.sh workflows      # actionlint, then zizmor
No findings to report. Good job! (3 ignored, 24 suppressed)
$ shellcheck tools/ci/security.sh; echo $?
0

Each check catches what it should. Each finding was planted in scratch only, and nothing was pushed:

Planted Command Result
a fake ghp_ token in a detached scratch commit SECRETS_LOG_OPTS=<range> security.sh secrets exit 1, RuleID: github-pat, leaks found: 1
the same kind of token with # gitleaks:allow gitleaks dir, without and with --ignore-gitleaks-allow exit 0, then exit 1 (leaks found: 1)
lodash 4.17.15 in a package-lock.json security.sh deps exit 1, 6 vulnerabilities
a Flask route that runs subprocess with shell=True security.sh sast exit 1 (subprocess-injection, dangerous-subprocess-use)
the same line with # nosemgrep semgrep, without and with --disable-nosem 0 findings (exit 0), then 3 blocking findings (exit 1)
a tools/ci/.semgrepignore containing *, staged security.sh sast exit 1, "a .semgrepignore outside the repository root hides code from SAST"
a workflow with an untrusted ${{ github.event... }} in run:, unpinned actions and no permissions security.sh workflows exit 1: actionlint [expression]; zizmor template-injection, unpinned-uses, artipacked, excessive-permissions

Before D4, the existing workflows had 64 zizmor findings (unpinned actions, template injection, persisted credentials, cache poisoning) and 2 actionlint findings (SC2086 in docs.yml, SC2034 in previews.yml). All are fixed.

Not run locally: zizmor's online audits, which need a token, and the Claude review itself, because no Claude secret exists. Both run on the PR.

Sources. Each tool's inputs and flags were read from its own docs at the pinned version. The links and read dates are in §11 and §12.

Code review (review-code; findings in docs/reviews/code/D4.md):

CODE-REVIEW  verdict=approve  blockers=0  majors=0  minors=1  simplify=0
SECURITY-REVIEW  verdict=approve  blockers=0  majors=0  minors=1
SIMPLIFY  changed=0 files  lines=-0/+0  tests=fail (shellcheck passed; `security.sh workflows` was blocked by permissions and did not run)
  • Rounds. There were three review rounds, run independently and in parallel:
    • round 1: code changes (1 major), security changes (3 majors);
    • round 2: code approve, security changes (1 major: a nested .semgrepignore);
    • round 3: both approve. The two final minors were fixed in 21fd4b1.
  • code-simplifier changed nothing: every remaining line is a pin, a guard or a control a reviewer asked for. It could not run the workflow linters itself because of its permissions. The worker ran them on the same tree, 21fd4b1, and they passed.

Figures and media

What runs on a pull request, and which three checks can block its merge. Click the figure for the full size.

What runs on a pull request What runs on a pull request

Generator: python3 tools/diagrams/d4_pr_checks.py.

The cast runs three of the four checks for real on the repository, through the same script CI runs: secrets, dependencies, and the workflow linters (zizmor offline). SAST is left out because it fetches semgrep's rulesets, which takes longer. Generator: python3 tools/media/cast.py tools/media/casts/security-checks.yaml.

Findings and open questions

For Thomas (one-time setup, nothing is set by agents):

What Where Needed for
ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN repository secret Claude review; it is skipped without it. It is not free: API billing, or Thomas's subscription limits
RELEASE_APP_ID (variable) and RELEASE_APP_PRIVATE_KEY (secret) release environment, deployment branches limited to main release-please; also a repo-only App and a v* tag ruleset (§12)
RELEASE_PLEASE_ENABLED=true repository variable turns release-please on, after the App exists
protection payload autopilot.md §8 makes ci-ok, docs-gate, pr-body and CODEOWNERS binding

Open questions:

  1. create-github-app-token v3.3.0 marks app-id as deprecated in favour of client-id. Switch to a RELEASE_APP_CLIENT_ID variable?
  2. If the autopilot opens PRs under Thomas's account, code-owner reviews can only be met through an admin bypass. Should it get its own identity, so that enforce_admins can be turned on?
  3. The nf-core webhook is still in history: rewrite it before the repository goes public (inbox #5), and tell nf-core (inbox #27).
  4. On the first real Claude review, check that the workspace stays at the base commit, and that gh still posts with the environment scrub on [to be validated].
  5. Still unpinned: semgrep's registry rulesets, the docs job's pip install, and wrangler@4 in previews.yml (D5). Python has no lockfile yet, so the Python audit has nothing to scan.
  6. The scanner pins live in tools/ci/security.sh. They move to dev/versions.env once D3 lands.