D1 · Validate the Claude setup

Done 2026-10-27 · PR #14 · all 14 agents and 22 skills load · new CI check claude-config · CLAUDE.md corrected in 6 places · 2 open questions for Thomas

Left: the 14 agent files and 22 skill folders of .claude/. A headless run loaded all of them. 18 skills are in Claude's own skill list and 4 are slash-only (yellow). Middle: tools/claude/check_frontmatter.py and its rules, run by the new claude-config workflow. Right: five broken fixtures run through claude plugin validate --strict and through the check. The validator passed three of them with exit 0, so CI uses the check.

What changed

• New check tools/claude/check_frontmatter.py, about 100 lines of Python with PyYAML. It fails, naming the file, when an agent (any .md under .claude/agents/, recursively) or a skill (.claude/skills/<name>/SKILL.md) has:
  • no frontmatter on line 1, or frontmatter that is not closed, does not parse or is not a mapping;
  • an agent name that is missing, starts with -, contains :, is over 256 characters or is used twice;
  • a missing description (for skills a repository rule, see the spec row);
  • a field Claude Code does not document;
  • also when a skill folder has no SKILL.md.
• Its tests, tools/claude/test_check_frontmatter.py: 18 cases. 16 write a broken fixture to a temporary folder and expect exit 1 with the file named; one valid fixture and the repository's own .claude expect exit 0.
• New workflow .github/workflows/claude-config.yml: on every PR and on pushes to main and autopilot/integration, it runs the tests, then the check. It has concurrency, a 5-minute timeout, permissions: {} with contents: read on the job, actions/checkout pinned by SHA (v4.4.0) and persist-credentials: false.
• Owning spec: dev-environment-and-testing.md §11 has a new claude-config.yml row and a sentence under Workflow conventions. The §8 skills row now names all eight workflow skills.
• CLAUDE.md corrected where it no longer matched the repository:
  1. the repository also holds tools/ and tests/;
  2. from wave W3 the work includes build steps for the development layer, not only reviews, spikes and planning;
  3. a spike runs on branch spike/<name> (.claude/skills/spike/SKILL.md:12);
  4. review-code runs security-reviewer beside code-reviewer (.claude/skills/review-code/SKILL.md:15), and the agent table says so;
  5. ux-reviewer has Write and Edit and edits a board when asked to fix it (.claude/agents/ux-reviewer.md), so it is named as an exception to "read-only";
  6. a new Skills section explains why four skills are missing from Claude's own list, and the check is in Useful commands.
• No agent or skill file changed. All of them load. The vendored superpowers skills are untouched.
• Sketch: tools/diagrams/d1_claude_config.py, listed in docs/assets/media.yaml.

Why

Prompt: next-phase.md · D1. An agent whose frontmatter breaks does not fail loudly. Claude Code skips a nameless agent, and drops every field of a skill whose YAML does not parse. It also ignores an unknown field (sub-agents, skills, read 2026-10-27). A reviewer could then vanish from a review round and nobody would notice. The CI check makes that failure loud.

The prompt suggested claude plugin validate. On the fixtures below it missed a nameless agent and an unknown field, even with --strict, so the check is a small script whose rules come from those two docs pages.

Evidence

Do the agents load? A headless session on this branch (Claude Code 2.1.311), asked to list what it could dispatch, with a debug log:

$ claude -p --debug-file debug.log --max-turns 1 "…list every subagent_type… then every skill…"
AGENT code-reviewer
AGENT code-simplifier
AGENT constraints-hunter
AGENT depictio-liaison
AGENT diagram-artist
AGENT doc-reviewer
AGENT doc-writer
AGENT fact-checker
AGENT infra-engineer
AGENT media-capture
AGENT security-reviewer
AGENT spec-reviewer
AGENT test-architect
AGENT ux-reviewer
…                       (plus the built-in agents)
$ grep "Loaded .* unique skills" debug.log
[DEBUG] Loaded 36 unique skills (36 unconditional, 0 conditional, managed: 0, user: 1, project: 22, additional: 0, legacy commands: 0)
$ grep -ciE "does not recognize|failed to parse" debug.log
0

All 14 project agents are listed. All 22 project skill folders load. The session's skill list showed 18 of them. The other four (review-round, apply-findings, verify-claims, spike) set disable-model-invocation: true, which "removes the skill from Claude's context entirely" but keeps /name working (skills).

claude plugin validate on the repository passes, and its JSON report lists no file with a problem:

$ claude plugin validate .claude/agents
√ Validation passed
$ claude plugin validate --json --strict .claude
{ "success": true, "strict": true, … "contents": [], … }

The same five broken agents through both tools. Each fixture is one file, <case>/.claude/agents/a.md, in a scratch folder (not committed; the unit tests cover the same cases for the check). A scratch loop ran claude plugin validate --strict <case>/.claude and python3 tools/claude/check_frontmatter.py <case>/.claude on each; exit 1 means caught.

Case	Frontmatter between the --- lines
unclosed-quote	name: q, description: "unterminated, tools: Read
flow-sequence	name: [broken, description: x: y: z
no-description	name: nodesc, tools: Read
no-name	description: An agent without a name.
unknown-field	name: typo, description: x, allowedTools: Read

unclosed-quote  validate=1 check=1  agents/a.md: YAML does not parse (file line 4): found unexpected end of stream
flow-sequence   validate=0 check=1  agents/a.md: YAML does not parse (file line 3): expected ',' or ']', but got ':'
no-description  validate=1 check=1  agents/a.md: `description` is missing or empty
no-name         validate=0 check=1  agents/a.md: `name` is missing or empty (the agent would not load)
unknown-field   validate=0 check=1  agents/a.md: unknown field `allowedTools`

The flow-sequence fixture (name: [broken) is rejected by PyYAML, while Claude Code's own parser passes it. The check is stricter there, on purpose.

Test first. The tests were written before the check and all failed (Ran 17 tests … FAILED (failures=17)). With the check, and after the review fixes (18 tests):

$ python3 -m unittest tools/claude/test_check_frontmatter.py
..................
Ran 18 tests in 1.0s
OK
$ python3 tools/claude/check_frontmatter.py
14 agent(s), 22 skill(s) checked in .claude: 0 problem(s)

Code review (docs/reviews/code/D1.md):

CODE  step=D1  review=approve b/M/m=0/0/2  security=approve b/M/m=0/0/3  rounds=1  simplify=nochange -0/+0  tests=pass

Three minors were fixed in aee3798: agents scanned recursively, the skill description rule worded as a repository rule, persist-credentials: false. Both reviewers then approved the fix range with no findings. Two minors stay open (below).

Docs gate (with SPECTIO_VIRGIL_TTF set, so the sketch renders in Virgil):

$ python3 tools/docs/check.py
mermaid: 5 blocks checked
diagrams: 10 scripts re-run, 20 SVGs compared
media: 59 outputs in the manifest
progress: 23 pages checked
docs gate: 0 failure(s)

doc-reviewer: revision 1 approve 12/14, with three changes: a cast for the new command, a key in the sketch, and a link back from the spec row. Revision 2 approve 14/14.

Figures and media

• Sketch: docs/assets/diagrams/d1_claude_config_{light,dark}.{svg,png}, from tools/diagrams/d1_claude_config.py (embedded above).
• Cast: the check passing on a copy of the repository's .claude, then failing on a copy where spec-reviewer has lost its name line. Generated by python3 tools/media/cast.py tools/media/casts/claude-config.yaml; the output is real.

Findings and open questions

• SEC-D1-02 (minor, open, for D4): CI installs PyYAML pinned by version, not by hash; docs.yml does not pin it at all. D4 owns the dependency audit and the workflow conventions of §11.
• SEC-D1-03 (minor, open, question for Thomas): should the check also enforce that the reviewer agents stay read-only? It would reject Edit, Write or permissionMode: bypassPermissions on spec-reviewer, security-reviewer, code-reviewer and the others. D1 checks validity, not policy.
• Question for Thomas: claude-config is not a required check. Branch protection requires only ci-ok, docs-gate and pr-body (§11). Once ci.yml exists, should it call this workflow (which then needs on: workflow_call) so that ci-ok covers it?
• Observed, not changed: headless claude -p here prints "Ignoring 23 permissions.allow entries from .claude/settings.json: this workspace has not been trusted". A headless autopilot run (tools/autopilot/run.sh) gets no project allow-list until the workspace is trusted [to be validated] against run.sh on Thomas's machine.