Skip to content

T6 · Playwright end-to-end skeleton

Done 2026-10-28 · narrowed to the smallest skeleton by the MVP decision of 2026-10-28 · tests/e2e/: Playwright 1.63.0 in TypeScript, projects unit, desktop and phone, 11 tests green locally in 1.8 s and in CI (40699486247) · side-by-side report with the boards deferred · review: approve after 2 rounds

T6 figure: where the end-to-end tests point, first match wins. 1, SPECTIO_E2E_BASE_URL, an http or https URL refused when it carries a user, a password or a query. 2, urls.spectio of a scenario JSON named by SPECTIO_SCENARIO, refused until a scenario runs Spectio. 3, otherwise the static stand-in page served on 127.0.0.1:8710. Three Playwright projects run 11 tests: unit (9 tests of the config, no browser), desktop and phone (one smoke test each). Against the stand-in, the smoke test saves RunsTable--desktop.png and RunsTable--phone.png, named after the RunsTable board of design/ui-canvas. Two targets verified: e2e.yml green in CI against the stand-in, and the Cloudflare docs preview passing on desktop and phone, run locally. The side-by-side report of these screenshots and the board renders is drawn as deferred T6 figure: where the end-to-end tests point, first match wins. 1, SPECTIO_E2E_BASE_URL, an http or https URL refused when it carries a user, a password or a query. 2, urls.spectio of a scenario JSON named by SPECTIO_SCENARIO, refused until a scenario runs Spectio. 3, otherwise the static stand-in page served on 127.0.0.1:8710. Three Playwright projects run 11 tests: unit (9 tests of the config, no browser), desktop and phone (one smoke test each). Against the stand-in, the smoke test saves RunsTable--desktop.png and RunsTable--phone.png, named after the RunsTable board of design/ui-canvas. Two targets verified: e2e.yml green in CI against the stand-in, and the Cloudflare docs preview passing on desktop and phone, run locally. The side-by-side report of these screenshots and the board renders is drawn as deferred

What changed

The behaviour is in dev-environment §6.1. This list names the files.

  • tests/e2e/, a Node package of its own:
    • package.json and package-lock.json: @playwright/test 1.63.0, the PLAYWRIGHT_VERSION of dev/versions.env (§1). The lockfile holds three packages (@playwright/test, playwright, playwright-core) with their integrity hashes.
    • playwright.config.ts: the base URL from base-url.ts; three projects, unit (*.unit.ts, no browser), desktop (Desktop Chrome) and phone (Pixel 7, both on *.e2e.ts); the stand-in page served by python3 -m http.server when no other target is set; traces kept on failure for the stand-in only.
    • base-url.ts: picks the target, first match wins. (1) SPECTIO_E2E_BASE_URL, http or https only, refused when it has a user, a password or a query, and the error does not repeat the URL. (2) urls.spectio from the scenario JSON that SPECTIO_SCENARIO names; a JSON without that key is refused with the reason. (3) The stand-in at http://127.0.0.1:8710.
    • boards.ts: screens/<Board>--<project>.png, where <Board> must match [A-Za-z0-9]+ and a file design/ui-canvas/<Board>.dc.html must exist.
    • smoke.e2e.ts: one test. The home page returns a 2xx status, its title contains "Spectio", its main landmark is visible, and the page raises no script error. Against the stand-in only, it saves the RunsTable screenshot.
    • Unit tests: base-url.unit.ts (6), boards.unit.ts (2), pin.unit.ts (1: @playwright/test in package.json equals PLAYWRIGHT_VERSION).
    • static/index.html, the stand-in page ("Runs · Spectio", one main region), and .gitignore (node_modules/, playwright-report/, test-results/, screens/).
  • .github/workflows/e2e.yml, one job on ubuntu-24.04, 15-minute timeout. It runs on a push to any branch that touches tests/e2e/, dev/versions.env, design/ui-canvas/ or the workflow itself, and on manual dispatch. Steps: Node from NODE_VERSION of dev/versions.env through actions/setup-node, then npm ci --ignore-scripts, then npx --no-install playwright install --with-deps chromium and npx --no-install playwright test. The HTML report and screens/ go to the e2e-report artifact, kept 14 days. Hardening as in scenarios.yml: permissions: {} at the top, contents: read for the job, actions pinned by SHA, persist-credentials: false. ci.yml does not call it.
  • dev/versions.env: a comment line saying the Playwright pin is also in tests/e2e/package.json, checked by tests/e2e/pin.unit.ts. No value changed.
  • Spec, dev-environment-and-testing.md:
    • §5: a bullet on urls.spectio, the key a scenario that runs Spectio will write;
    • new §6.1, end-to-end tests;
    • §11: the e2e.yml row, plus additions to "Not pinned" (Playwright's Chromium download, actions/setup-node) and a new "Not scanned" bullet (tests/ and semgrep).
  • Review record, docs/reviews/code/T6.md.

Deferred. These need product code, or wait for the full UI:

  • The side-by-side HTML report of the screenshots next to the design/ui-canvas renders (tools/canvas/render.js). The MVP UI is ultra simple, and the boards were drawn for the full product, so the pairs would not match.
  • A destructive-specs project, the tier 4 journeys, and running against the bundle or Compose.

Why

Prompt: next-phase.md · T6. It asks for:

  • tests/e2e with Playwright in TypeScript;
  • a config that takes the base URL from the scenario JSON;
  • desktop and phone projects;
  • screenshots named after the canvas boards;
  • a side-by-side HTML report against design/ui-canvas renders;
  • runs against the Cloudflare docs preview and a static page until the product exists.

Thomas's MVP decision of 2026-10-28 narrowed the step to the smallest skeleton. The MVP UI is ultra simple: launch and monitor only, with no Kubernetes, no gates and no visualisation. So the step builds every item above except the side-by-side report (see Deferred). The decision log has no entry for that decision yet (finding 5). The closest entry is 2026-10-27 — 0.0.1 revised. The step decided behaviour that §6.1 records.

How to test

In a cloud session, Playwright 1.63.0 uses the Chromium pre-installed under /opt/pw-browsers (chromium-1243). Elsewhere, run npx --no-install playwright install chromium first.

cd tests/e2e && npm ci && npx --no-install playwright test

Against the Cloudflare docs preview (§7.1), browser projects only:

SPECTIO_E2E_BASE_URL=https://spectio.pages.dev/ npx --no-install playwright test --project desktop --project phone

Evidence

Local, cloud session, on autopilot/T6 at 5b6947f:

$ cd tests/e2e && npm ci && npx --no-install playwright test
  ✓   9 [unit] › base-url.unit.ts:37:5 › SPECTIO_SCENARIO without urls.spectio says why (2ms)
  ✓  10 [desktop] › smoke.e2e.ts:5:5 › the home page answers, shows its main region, no script errors (257ms)
  ✓  11 [phone] › smoke.e2e.ts:5:5 › the home page answers, shows its main region, no script errors (305ms)

  11 passed (1.8s)
$ ls screens
RunsTable--desktop.png
RunsTable--phone.png

That is 9 unit tests and the smoke test on desktop and phone. The worker's own run gave 11 passed (1.7s).

Against the docs preview:

$ SPECTIO_E2E_BASE_URL=https://spectio.pages.dev/ npx playwright test --project desktop --project phone
  ✓  1 [phone] › smoke.e2e.ts:5:5 › the home page answers, shows its main region, no script errors (2.3s)
  ✓  2 [desktop] › smoke.e2e.ts:5:5 › the home page answers, shows its main region, no script errors (17.4s)

  2 passed (18.1s)

The worker's run gave 2 passed (7.0s). Before a fix, the test asserted a visible h1, and that failed on the preview: mkdocs-material hides the home page's h1. The smoke test now asserts the main landmark.

On PR #45's own preview (https://pr-45.spectio.pages.dev/), 3 of 6 runs first failed with page.goto: Test timeout of 30000ms exceeded: the load event waited on same-origin assets (stylesheets/extra.css, a .webp screenshot, gate-review.mp4) that stalled through this cloud session's proxy. The smoke test now waits for domcontentloaded (smoke.e2e.ts); after that change, 8 of 8 runs passed on desktop and phone, and the stand-in still gives 11 passed.

Test first:

  • base-url.unit.ts was written before the config. Its first run gave "No tests found".
  • pin.unit.ts failed with @playwright/test set to 1.62.0, then passed at 1.63.0.

CI, e2e.yml:

Run Head Result Tests Node
40698975901 b69fdd2 green 10 v22.23.3, from the runner image
40699486247 af7bcf0 green 11 v24.22.1, from the dev/versions.env pin

The 11th test is the credential test, added in review round 1 (SEC-T6-01).

Security tooling, run on the step's files:

  • osv-scanner (tools/ci/security.sh deps) scanned tests/e2e/package-lock.json: 3 packages, no issues.
  • actionlint and zizmor 1.31.1 (--offline --min-severity low): clean on e2e.yml.
  • semgrep: the repository-wide SAST check does not scan tests/ (finding 1). tests/e2e/base-url.ts, scanned on its own under p/javascript and p/typescript, has 0 findings (record, SEC-T6-05).

Review (record):

  • Round 1, on b69fdd2: code-reviewer changes (0 blockers, 1 major, 6 minors), security-reviewer approve (0 blockers, 0 majors, 5 minors).
    • The major, CR-T6-01: e2e.yml did not run when dev/versions.env or design/ui-canvas/ changed. Both are now in its push paths.
    • The other fixes: Node from the pin, a credential in the URL refused and never echoed, npx --no-install, the board screenshot taken against the stand-in only, test outputs in test.info().outputPath().
  • Round 2, on af7bcf0: both reviewers approve. One minor is deferred: SEC-T6-05.
  • code-simplifier, on a274636: no change; the 11 tests pass.
CODE  step=T6  review=approve b/M/m=0/1/6  security=approve b/M/m=0/0/5  rounds=3  simplify=nochange -0/+0  tests=pass

Figures and media

  • t6-e2e sketch, light and dark, at the top of this page: python3 tools/diagrams/t6_e2e.py. The PNGs were rendered with the fallback font, because the Virgil font was not available in the session, as for T5. documentation-system §3.3 item 3 expects Virgil, and sketch.py falls back without a flag (D6, finding 2). Re-render the PNGs where docs/fonts/Virgil.ttf exists. The SVGs that the gate compares do not change.
  • No screenshot is committed. RunsTable--desktop.png and RunsTable--phone.png are taken of the stand-in page, not of a Spectio screen. Each run of e2e.yml uploads them to its e2e-report artifact, kept 14 days. documentation-system §3.1 asks a build step for screenshots of each new screen; this step adds no product screen.
  • No cast. npx --no-install playwright test is a development command, not a new Spectio CLI flow. Evidence quotes its output.

Findings and open questions

  1. tests/ is not scanned by the SAST check (SEC-T6-05, deferred). With no root .semgrepignore, semgrep's built-in ignore list seems to skip tests/. The cause is [to be validated] (§11, "Not scanned"). The proposed fix is a root .semgrepignore, which is code-owned. Filed as inbox #63.
  2. urls.spectio is a contract that nothing writes yet. base-url.ts reads it, and §5 names it, but dev/scenario.py does not write it, because no scenario runs Spectio. Until then, SPECTIO_SCENARIO always fails with "this scenario does not run Spectio yet".
  3. Two downloads in CI that we do not hash. actions/setup-node downloads Node without checking NODE_SHA256 of dev/versions.env. Whether it checks a checksum of its own is [to be validated]. playwright install downloads Chromium from Playwright's CDN. Both are recorded under "Not pinned" in §11.
  4. e2e.yml is standalone, outside ci-ok, like scenarios.yml. A red run does not block a merge.
  5. The MVP decision of 2026-10-28 is now in the decision log (recorded on the integration branch while T6 ran); §6.1 links it.
  6. Media: no committed screenshot or cast (see Figures and media). The first screenshots of real pages come with the first UI milestone, along with the side-by-side report.