From 1b67d1f15e73504fabe86dadeeb2995b5719e27b Mon Sep 17 00:00:00 2001 From: Ken Yasue Date: Thu, 2 Jul 2026 12:26:15 +0200 Subject: [PATCH] Require Playwright E2E tests in every verification step - New skill e2e-testing: on-demand Playwright setup (config, test:e2e script), acceptance-criteria-driven specs (one per feature + app-shell smoke), blueprint-driven semantic locators, and the shared verification checklist - /add-feature Step 7 and /execute-milestones Step 2.6 now write/update E2E specs and run npm run test:e2e alongside test/lint/typecheck; milestone gates and completion criteria include E2E - /generate-app Step 5 sets up Playwright and creates the app-shell smoke spec for the web target (integration_test / UI automation noted for flutter/winui3) - steering tasklist template, milestone-planning Definition of Done, milestone-execution task slicing/gates, development-guidelines pointers, and implementation-validator checks updated - CLAUDE.md, README, and settings.json updated Co-Authored-By: Claude Fable 5 --- .claude/agents/implementation-validator.md | 6 + .claude/commands/add-feature.md | 12 +- .claude/commands/execute-milestones.md | 12 +- .claude/commands/generate-app.md | 10 +- .claude/settings.json | 1 + .../skills/development-guidelines/SKILL.md | 1 + .claude/skills/e2e-testing/SKILL.md | 118 ++++++++++++++++++ .claude/skills/milestone-execution/SKILL.md | 12 +- .claude/skills/milestone-planning/template.md | 3 +- .claude/skills/steering/templates/tasklist.md | 3 + CLAUDE.md | 4 +- README.md | 5 +- 12 files changed, 166 insertions(+), 21 deletions(-) create mode 100644 .claude/skills/e2e-testing/SKILL.md diff --git a/.claude/agents/implementation-validator.md b/.claude/agents/implementation-validator.md index 2cfec79..c8e4725 100644 --- a/.claude/agents/implementation-validator.md +++ b/.claude/agents/implementation-validator.md @@ -53,6 +53,7 @@ Verify that the implemented code meets the following criteria: - [ ] Is the coverage target met? - [ ] Are edge cases tested? - [ ] Are the tests named appropriately? +- [ ] Are there Playwright E2E tests covering the feature's acceptance criteria (e2e/*.spec.ts)? **Evaluation Criteria**: - ✅ Sufficient: Coverage of 80% or more, covering the main cases @@ -213,6 +214,11 @@ npm test npm run test:coverage ``` +### Run E2E Tests (Playwright) +```bash +npm run test:e2e +``` + ### Build Check ```bash npm run build diff --git a/.claude/commands/add-feature.md b/.claude/commands/add-feature.md index 919fb14..7b3fa88 100644 --- a/.claude/commands/add-feature.md +++ b/.claude/commands/add-feature.md @@ -121,15 +121,19 @@ If any of the following situations occur while the implementation loop is runnin **Once this step completes successfully, never stop; immediately proceed to Step 7.** -## Step 7: Run Automated Tests +## Step 7: Run Automated Tests (including Playwright E2E) -1. Run the following commands in order and confirm that all tests pass. +1. Write or update the feature's end-to-end tests following the **e2e-testing skill** (`Skill('e2e-testing')`): + - Create/update `e2e/[feature-name].spec.ts` covering the feature's acceptance criteria. + - If Playwright is not yet set up in the repository, set it up first as described in the skill. +2. Run the following commands in order and confirm that all of them pass. ```bash Bash('npm test') Bash('npm run lint') Bash('npm run typecheck') + Bash('npm run test:e2e') ``` -2. If any command produces an error, analyze the problem, generate and apply a fix, and then run this step again. +3. If any command produces an error, analyze the problem, generate and apply a fix, and then run this step again. Never skip or delete a failing test to make verification pass. **Once this step completes successfully, never stop; immediately proceed to Step 8.** @@ -164,7 +168,7 @@ If any of the following situations occur while the implementation loop is runnin This workflow completes automatically once all of the following conditions are met. - Step 5: All tasks in `tasklist.md` are complete (`[x]` or skipped for a valid reason). - Step 6: The `implementation-validator` subagent's validation passes. -- Step 7: The `test`, `lint`, and `typecheck` commands all succeed without errors. +- Step 7: The `test`, `lint`, `typecheck`, and `test:e2e` (Playwright) commands all succeed without errors, and the feature has E2E tests covering its acceptance criteria. - Step 8: Handover notes are recorded in `tasklist.md`, all six persistent documents in `docs/` have been reconciled with the feature (updates applied, or a skip reason noted for each), and the feature is checked off in its phase file (if `docs/milestones/` exists). Until these completion criteria are met, continue to think autonomously, solve problems, and carry on the work. diff --git a/.claude/commands/execute-milestones.md b/.claude/commands/execute-milestones.md index 510a041..a59dd57 100644 --- a/.claude/commands/execute-milestones.md +++ b/.claude/commands/execute-milestones.md @@ -103,15 +103,19 @@ Use the `Task` tool to launch the `implementation-validator` subagent: - `description`: "Milestone implementation validation" - `prompt`: "Please validate the quality of all changes implemented for milestone `[NN] [name]`. The target files are `[list of implemented file paths]`. Focus on coding standards, error handling, testability, consistency with existing patterns, and whether every feature's acceptance criteria in `.steering/[dir]/requirements.md` are met." -#### 2.6 Run Automated Tests +#### 2.6 Run Automated Tests (including Playwright E2E) + +1. Write or update the milestone's end-to-end tests following the **e2e-testing skill** (`Skill('e2e-testing')`): one `e2e/[feature-name].spec.ts` per feature in the milestone, covering its acceptance criteria. Set up Playwright first (as described in the skill) if it is missing. +2. Run: ```bash Bash('npm test') Bash('npm run lint') Bash('npm run typecheck') +Bash('npm run test:e2e') ``` -If any command fails, analyze, fix, and re-run until all pass. +If any command fails, analyze, fix, and re-run until all pass. Never skip or delete a failing test to make verification pass. #### 2.7 Retrospective and Document Reconciliation @@ -124,7 +128,7 @@ Apply `/add-feature` Step 8 for the milestone: #### 2.8 Milestone Gate — then Continue -The milestone is done only when: all tasks are `[x]`, validation passed, tests/lint/typecheck are green, the six documents are reconciled, and the milestone status is updated in the phase file and the roadmap. +The milestone is done only when: all tasks are `[x]`, validation passed, tests/lint/typecheck/E2E (Playwright) are green, the six documents are reconciled, and the milestone status is updated in the phase file and the roadmap. **Once the gate passes, never stop; immediately start Step 2 for the next milestone in the queue.** If the gate genuinely cannot be passed (external blocker), record the blocker in the milestone's `tasklist.md`, the phase file, and `roadmap.md`, then stop and report — this is the only permitted early stop. @@ -138,7 +142,7 @@ The milestone is done only when: all tasks are `[x]`, validation passed, tests/l - Every milestone in the specified phase file has status `Completed`. - The phase's status is `Completed` in both the phase file and `docs/milestones/roadmap.md`. - Each executed milestone has its own `.steering/[date]-milestone-[NN]-[name]/` directory with all tasks `[x]`. -- Tests, lint, and typecheck pass on the final state. +- Tests, lint, typecheck, and Playwright E2E tests pass on the final state, with E2E coverage for every implemented feature's acceptance criteria. - The six persistent documents reflect everything implemented. Completion message: diff --git a/.claude/commands/generate-app.md b/.claude/commands/generate-app.md index aa65b09..43e74fc 100644 --- a/.claude/commands/generate-app.md +++ b/.claude/commands/generate-app.md @@ -117,10 +117,13 @@ Apply the requirements for the selected target: If new directories/files were created, update `docs/repository-structure.md` so it reflects the generated layout. -### Step 5: Verify +### Step 5: Verify (including Playwright E2E) -- If the target is `web` and the repo has `npm test`, `npm run lint`, `npm run typecheck`, run them and fix any errors caused by the generated code. -- For `flutter` / `winui3`, run the platform's available static checks if the toolchain is present; otherwise note what should be checked manually. +- If the target is `web`: + - Run `npm test`, `npm run lint`, `npm run typecheck` (where present) and fix any errors caused by the generated code. + - Set up Playwright following the **e2e-testing skill** (`Skill('e2e-testing')`) if it is not set up yet, and create the app-shell smoke spec `e2e/app-shell.spec.ts` (shell renders, every blueprint route navigates, primary navigation works). + - Run `npm run test:e2e` and fix any failures caused by the generated code. +- For `flutter` / `winui3`, run the platform's available static checks and UI/integration tests if the toolchain is present (`integration_test` for Flutter, the platform's UI automation for WinUI 3); otherwise note what should be checked manually. ## Completion Criteria @@ -140,6 +143,7 @@ Generated: ✅ Routes / navigation (from ui-blueprint.json) ✅ Screens (one per blueprint screen) ✅ Reusable components +✅ Playwright E2E smoke spec (e2e/app-shell.spec.ts, web target) Next steps: - Run the app and verify the shell renders diff --git a/.claude/settings.json b/.claude/settings.json index d21716e..c1511eb 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -12,6 +12,7 @@ "Skill(platform-ui-generation)", "Skill(milestone-planning)", "Skill(milestone-execution)", + "Skill(e2e-testing)", "Skill(steering)" ], "deny": [], diff --git a/.claude/skills/development-guidelines/SKILL.md b/.claude/skills/development-guidelines/SKILL.md index fffc94d..4187141 100644 --- a/.claude/skills/development-guidelines/SKILL.md +++ b/.claude/skills/development-guidelines/SKILL.md @@ -90,6 +90,7 @@ When creating development guidelines: ./template.md ### When Designing Tests - "Test Strategy" in ./guides/process.md (pyramid, coverage) - "Test Code" in ./guides/implementation.md (implementation patterns) +- E2E tests: the **e2e-testing skill** (Playwright — E2E is part of every verification step) ### When Preparing a Release - "Git Workflow Rules" in ./guides/process.md (policy for merging into main) diff --git a/.claude/skills/e2e-testing/SKILL.md b/.claude/skills/e2e-testing/SKILL.md new file mode 100644 index 0000000..31ea271 --- /dev/null +++ b/.claude/skills/e2e-testing/SKILL.md @@ -0,0 +1,118 @@ +--- +name: e2e-testing +description: Guide for writing and running Playwright end-to-end tests as part of every verification step. Use when verifying a feature, milestone, or generated app shell, or when setting up Playwright in the repository. +allowed-tools: Read, Write, Edit, Bash +--- + +# E2E Testing Skill (Playwright) + +This skill defines how end-to-end tests are written and run in this project. **Every verification step includes E2E tests**: a feature, milestone, or app shell is not "verified" until its Playwright tests pass alongside unit tests, lint, and typecheck. + +## Core Principles + +1. **E2E is part of verification, not an extra**: the standard verification sequence is `npm test` → `npm run lint` → `npm run typecheck` → `npm run test:e2e`. All four must pass. +2. **Tests are derived from acceptance criteria**: every feature's acceptance criteria (from its phase file or steering `requirements.md`) map to E2E assertions. If a criterion cannot be asserted in an E2E test, note why in the tasklist. +3. **Blueprint-driven selectors**: screens, routes, and actions come from `docs/design/ui-blueprint.json`. Tests navigate by the blueprint's routes and interact via the blueprint's actions. +4. **One spec file per feature** plus one smoke spec for the app shell. +5. **Web-first**: Playwright covers the `web` target. For `flutter` use `integration_test`, for `winui3` use the platform's UI automation — apply the same acceptance-criteria mapping; note in the tasklist when Playwright does not apply. + +## Setup (run once, on demand) + +If Playwright is not yet set up in the repository (no `@playwright/test` in `package.json`), set it up before the first E2E run: + +```bash +npm install -D @playwright/test +npx playwright install --with-deps chromium +``` + +Create `playwright.config.ts`: + +```typescript +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + fullyParallel: true, + retries: process.env.CI ? 2 : 0, + reporter: [['list'], ['html', { open: 'never' }]], + use: { + baseURL: process.env.E2E_BASE_URL ?? 'http://localhost:3000', + trace: 'retain-on-failure', + screenshot: 'only-on-failure', + }, + webServer: { + command: 'npm run dev', // adjust to the project's start command + url: 'http://localhost:3000', // adjust to the project's port + reuseExistingServer: !process.env.CI, + }, +}); +``` + +Add the script to `package.json`: + +```json +"test:e2e": "playwright test" +``` + +Adjust `baseURL`, `webServer.command`, and the port to the project's actual dev server (see `docs/architecture.md`). Add `e2e/` artifacts (`playwright-report/`, `test-results/`) to `.gitignore`. + +## Test Layout + +``` +e2e/ +├── app-shell.spec.ts # smoke: shell renders, every blueprint route navigates +├── [feature-name].spec.ts # one spec per feature +└── helpers/ # shared fixtures/utilities (only when duplicated 3+ times) +``` + +## Writing Feature Specs + +For each feature, create `e2e/[kebab-case-feature-name].spec.ts`: + +- One `test.describe` block per feature; one `test` per acceptance criterion (or a tight group of criteria). +- Navigate using the route of the affected screen from `ui-blueprint.json`. +- Prefer semantic locators: `getByRole`, `getByLabel`, `getByText` — matching the blueprint's component types and labels. Avoid CSS/XPath selectors tied to implementation details. +- No fixed waits (`waitForTimeout`); rely on Playwright's auto-waiting and explicit `expect(...).toBeVisible()` style assertions. +- Keep tests independent: each test sets up its own state and does not depend on execution order. + +Example: + +```typescript +import { test, expect } from '@playwright/test'; + +test.describe('Edit user profile', () => { + test('saves a changed display name', async ({ page }) => { + await page.goto('/profile'); + await page.getByLabel('Display name').fill('New Name'); + await page.getByRole('button', { name: 'Save' }).click(); + await expect(page.getByText('Profile updated')).toBeVisible(); + }); +}); +``` + +## Writing the App-Shell Smoke Spec + +Created by `/generate-app` (web target). `e2e/app-shell.spec.ts` must verify: + +- The app starts and the entry screen renders. +- Every route in `ui-blueprint.json` is reachable and renders its screen title. +- Primary navigation (from the blueprint's nav component) works. + +## Running + +```bash +npm run test:e2e +``` + +- If a test fails, analyze the trace/screenshot in `test-results/`, fix the code (or the test if the spec was wrong against the acceptance criteria), and re-run until green. +- Never delete or skip a failing test to make verification pass; a skipped test requires a written reason in the tasklist, same as a struck task. + +## Verification Checklist + +Used by `/add-feature` Step 7, `/execute-milestones` Step 2.6, and `/generate-app` Step 5: + +- [ ] Playwright is set up (`test:e2e` script exists) — set it up via this skill if missing. +- [ ] Every implemented feature has `e2e/[feature].spec.ts` covering its acceptance criteria. +- [ ] The app-shell smoke spec still passes. +- [ ] `npm test`, `npm run lint`, `npm run typecheck`, and `npm run test:e2e` all pass. +- [ ] No test was skipped or deleted without a written reason. diff --git a/.claude/skills/milestone-execution/SKILL.md b/.claude/skills/milestone-execution/SKILL.md index 12b5b5a..5965238 100644 --- a/.claude/skills/milestone-execution/SKILL.md +++ b/.claude/skills/milestone-execution/SKILL.md @@ -46,14 +46,15 @@ Tasks are **grouped by feature**, in the implementation order chosen in `design. - [ ] [Task 1: data/model layer] - [ ] [Task 2: logic/service layer] - [ ] [Task 3: UI per affected screen] -- [ ] [Task 4: tests for the acceptance criteria] +- [ ] [Task 4: unit tests + Playwright E2E spec for the acceptance criteria] ## Feature: Log in / log out - [ ] ... ## Milestone Verification - [ ] All acceptance criteria in requirements.md confirmed -- [ ] npm test / lint / typecheck pass +- [ ] E2E specs exist for every feature (e2e/[feature].spec.ts, per the e2e-testing skill) +- [ ] npm test / lint / typecheck / test:e2e pass ## Post-Implementation Retrospective _Filled at the end: completion date, plan vs. actual, lessons learned._ @@ -62,8 +63,8 @@ _Filled at the end: completion date, plan vs. actual, lessons learned._ ## How to Divide a Feature into Tasks - 3–8 tasks per feature; each completable in **one implementation-loop iteration**. -- Slice along the natural layers the feature touches: data/model → logic/service → UI (one task per affected screen) → tests. -- Every feature's section ends with a task that verifies its acceptance criteria. +- Slice along the natural layers the feature touches: data/model → logic/service → UI (one task per affected screen) → tests (unit + Playwright E2E). +- Every feature's section ends with a task that verifies its acceptance criteria via the feature's E2E spec (see the e2e-testing skill). - Tasks that unblock several features go into a "Shared Foundations" section at the top — never duplicated per feature. ## Gate Criteria Between Milestones @@ -72,7 +73,8 @@ A milestone passes its gate only when **all** of the following hold. Only then m - [ ] Every task in the milestone's `tasklist.md` is `[x]` (or struck with a technical reason). - [ ] The `implementation-validator` subagent's validation passed. -- [ ] `npm test`, `npm run lint`, `npm run typecheck` all succeed. +- [ ] Every feature of the milestone has a Playwright E2E spec covering its acceptance criteria. +- [ ] `npm test`, `npm run lint`, `npm run typecheck`, `npm run test:e2e` all succeed. - [ ] The six persistent documents in `docs/` are reconciled with the milestone's changes. - [ ] In the phase file: every feature of the milestone is checked off, and the milestone status is `Completed` with Completion Notes. - [ ] `roadmap.md` points to the next milestone (or, after the phase's last milestone, the phase is `Completed` and the roadmap points to the next phase file). diff --git a/.claude/skills/milestone-planning/template.md b/.claude/skills/milestone-planning/template.md index cb1077c..cb37ced 100644 --- a/.claude/skills/milestone-planning/template.md +++ b/.claude/skills/milestone-planning/template.md @@ -59,7 +59,8 @@ Use this structure for each `docs/milestones/phase[N]-milestones.md`. A phase fi ### Definition of Done - [ ] All features above are checked off -- [ ] `npm test`, `npm run lint`, `npm run typecheck` pass +- [ ] Every feature has a Playwright E2E spec covering its acceptance criteria +- [ ] `npm test`, `npm run lint`, `npm run typecheck`, `npm run test:e2e` pass - [ ] Persistent documents in `docs/` are reconciled with all features - [ ] [Milestone-specific criterion, e.g. "demo flow X→Y→Z works end-to-end"] diff --git a/.claude/skills/steering/templates/tasklist.md b/.claude/skills/steering/templates/tasklist.md index c6f705a..e91a8b9 100644 --- a/.claude/skills/steering/templates/tasklist.md +++ b/.claude/skills/steering/templates/tasklist.md @@ -59,6 +59,9 @@ When skipping, always state the reason clearly: - [ ] `npm run lint` - [ ] Confirm that there are no type errors - [ ] `npm run typecheck` +- [ ] Confirm that the E2E tests pass (Playwright — see the e2e-testing skill) + - [ ] E2E specs exist for this work's acceptance criteria + - [ ] `npm run test:e2e` - [ ] Confirm that the build succeeds - [ ] `npm run build` diff --git a/CLAUDE.md b/CLAUDE.md index 823a441..d4e2533 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,7 +14,7 @@ 1. **Document creation**: Define "what to build" in persistent documents (`docs/`) 2. **Work planning**: Plan "what to do this time" in steering files (`.steering/`) 3. **Implementation**: Implement according to tasklist.md and update progress as you go -4. **Verification**: Testing and operation checks +4. **Verification**: Testing and operation checks — unit tests, lint, typecheck, and Playwright E2E tests (see the e2e-testing skill) 5. **Update**: Reconcile all `docs/` documents with the implemented feature (see `/add-feature` Step 8) ### Important Rules @@ -103,7 +103,7 @@ Define "what to do this time" for a specific development task: - `.claude/agents/` - Subagent definitions (doc-reviewer, implementation-validator) - `.claude/commands/` - Slash commands (setup-project, define-design, generate-app, update-design, plan-milestones, execute-milestones, add-feature, review-docs) -- `.claude/skills/` - Task-specific skills (prd-writing, functional-design, architecture-design, repository-structure, development-guidelines, glossary-creation, ui-design, design-tokens, platform-ui-generation, milestone-planning, milestone-execution, steering) +- `.claude/skills/` - Task-specific skills (prd-writing, functional-design, architecture-design, repository-structure, development-guidelines, glossary-creation, ui-design, design-tokens, platform-ui-generation, milestone-planning, milestone-execution, e2e-testing, steering) ## Development Process diff --git a/README.md b/README.md index 507270a..48849b0 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,7 @@ A fully autonomous loop, one feature per run. Each run: 2. Reads the feature's entry in its phase file and the design spec first — and updates the design first if the feature changes the UI 3. Implements every task in `tasklist.md`, checking items off as it goes 4. Validates quality with the `implementation-validator` subagent -5. Runs `npm test`, `npm run lint`, `npm run typecheck` until green +5. Writes Playwright E2E tests for the feature's acceptance criteria, then runs `npm test`, `npm run lint`, `npm run typecheck`, and `npm run test:e2e` until green 6. Records a retrospective, reconciles all six `docs/` documents, and checks the feature off in its phase file ### Step 8 — Repeat and maintain @@ -78,7 +78,8 @@ A fully autonomous loop, one feature per run. Each run: - Document quality checks → `/review-docs ` **A feature is "done" when**: all tasks in its tasklist are `[x]`, validation passes, -test/lint/typecheck succeed, and the persistent documents are back in sync with the code. +test/lint/typecheck and the Playwright E2E suite succeed, and the persistent documents are back +in sync with the code. ## Notes