Create E2e Tests

Phase Overview

Phase 1: Discover  → test-strategy.md    (understand the app & user goals)
Phase 2: Specify   → test-spec.md        (define what to test in Given/When/Then)
Phase 3: Plan      → test-plan.md        (prioritize, structure, per-test guidance)
Phase 4: Implement → *.test.yaml files   (setup project, write tests, run them)
Phase 5: Verify    → updated spec files  (coverage check, reconcile spec ↔ tests)

Fast-Track

Check for existing artifacts before starting. The only way to skip artifact generation is if the user explicitly says so.

Phase 1: Discover

Goal: Understand the application, the user's role, and what matters most to test.

Output: <project>/test-specs/test-strategy.md

Steps

1. Get project path — ask where to create the test project (e.g., ./my-tests). All artifacts and tests will live here. Create the test-specs/ directory.

   If cloud MCP tools are available (SHIPLIGHT_API_TOKEN is set), use the $cloud skill to fetch environments and test accounts — this can pre-fill the target URL and credentials.

2. Silent scan — before asking questions, gather context from what's available:

   • Codebase: routes, components, package.json, framework
   • Git branch diff (what changed recently)
   • Existing tests (what's already covered)
   • PRDs, docs, README files
   • Cloud environments (if cloud MCP tools available)

3. Understand what to test — ask the user what they'd like to test, then ask targeted follow-up questions (one at a time, with recommendations based on your scan) to fill gaps: risk areas, user roles, authentication, data strategy, critical journeys. Skip questions the user has already answered.

4. Write test-strategy.md containing:

   • App profile: name, URL, framework, key pages/features
   • Risk profile: what matters most, what's fragile
   • Testing scope: what's in/out, user roles to cover
   • Data strategy: how test data will be created and cleaned up
   • Environment: target URL, auth method, any special setup

Phase 2: Specify

Goal: Define concrete test scenarios in structured Given/When/Then format, prioritized by risk. Surface ambiguities that would cause flaky or incomplete tests.

Input: reads test-specs/test-strategy.md

Output: <project>/test-specs/test-spec.md

Steps

1. Read test-strategy.md to understand scope and priorities.

2. Generate user journey specs — for each critical journey, write:

   • Title: descriptive name (e.g., "New user signup with email verification")
   • Priority: P0 (must-have), P1 (should-have), P2 (nice-to-have)
   • Preconditions: what must be true before the test starts (Given)
   • Happy path: step-by-step actions and expected outcomes (When/Then)
   • Edge cases: at least 2 per journey (e.g., invalid input, timeout, empty state)
   • Data requirements: what test data is needed

3. Review for testing risks — scan each journey for issues that would cause flaky or incomplete tests: data dependencies, timing/async behavior, dynamic content, auth boundaries, third-party services, state isolation, environment differences. Add a Testing Notes section to each journey with identified risks and mitigations. If anything is ambiguous, ask the user (one at a time, with a recommended answer and impact statement).

4. Write test-spec.md with all journey specs.

5. Checkpoint — present a summary table for user review:

   #	Journey	Priority	Steps	Edge Cases	Risks
   1	User signup	P0	5	3	Timing
   2	...	...	...	...	...

   Ask: "Does this look right? Any journeys to add, remove, or reprioritize?"

   Wait for user confirmation before proceeding.

Phase 3: Plan

Goal: Create an actionable implementation plan with per-test guidance.

Input: reads test-specs/test-spec.md

Output: <project>/test-specs/test-plan.md

Steps

1. Read test-spec.md.

2. Define test file structure — map journeys to test files:

   tests/
   ├── auth.setup.ts          (if auth needed)
   ├── signup.test.yaml        (Journey 1)
   ├── checkout.test.yaml      (Journey 2)
   └── ...
   

3. Set implementation order — ordered by:

   • Dependencies first (auth setup before authenticated tests)
   • Then by priority (P0 before P1)
   • Then by risk (highest risk first)

4. Per-test guidance — for each test file, specify:

   • Data strategy: what data to create/use, cleanup approach
   • Wait strategy: where to use WAIT_UNTIL vs WAIT, expected loading points
   • Flakiness risks: specific things to watch for in this test

5. Write test-plan.md.

6. Checkpoint — present summary:

   Ready to implement N test files. Shall I proceed?

Phase 4: Implement

Goal: Set up the project and write all YAML tests guided by the plan.

Input: reads test-specs/test-plan.md

Setup

Skip any steps already done (project exists, deps installed, auth configured).

1. Configure AI provider — check if the test project already has a .env with an AI API key. If not, ask the user to choose a provider:

   To run YAML tests, I need an AI provider for resolving test steps. Which provider would you like to use?

   A) Google AI — GOOGLE_API_KEY (Get key) — default model: gemini-3.1-flash-lite-preview B) Anthropic — ANTHROPIC_API_KEY (Get key) — default model: claude-haiku-4-5 C) OpenAI — OPENAI_API_KEY (Get key) — default model: gpt-5.4-mini D) Azure OpenAI — requires AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT — set WEB_AGENT_MODEL=azure:<deployment> E) AWS Bedrock — uses AWS credential chain — set WEB_AGENT_MODEL=bedrock:<model_id> F) Google Vertex AI — uses GCP Application Default Credentials — set WEB_AGENT_MODEL=vertex:<model> G) I already have it configured

   After the user chooses, ask for their API key and save it to the test project's .env file. For A/B/C, the model is auto-detected from the key. For D/E/F, also save WEB_AGENT_MODEL with the appropriate provider:model prefix. Optionally, the user can set WEB_AGENT_MODEL to override the default model (e.g., WEB_AGENT_MODEL=claude-sonnet-4-6).

2. Scaffold the project — call scaffold_project with the absolute project path. This creates package.json, playwright.config.ts, .env.example, .gitignore, and tests/. Save the API key to .env.

3. Install dependencies:

   npm install
   npx playwright install chromium
   

4. Set up authentication (if needed) — follow the standard Playwright authentication pattern.

   Add credentials as variables in playwright.config.ts:

   {
     name: 'my-app',
     testDir: './tests/my-app',
     dependencies: ['my-app-setup'],
     use: {
       baseURL: 'https://app.example.com',
       storageState: 'tests/my-app/.auth/storage-state.json',
       variables: {
         username: process.env.MY_APP_EMAIL,
         password: { value: process.env.MY_APP_PASSWORD, sensitive: true },
         // otp_secret_key: { value: process.env.MY_APP_TOTP_SECRET, sensitive: true },
       },
     },
   },
   

   Standard variable names: username, password, otp_secret_key. Use { value, sensitive: true } for secrets. Add values to .env.

   Write auth.setup.ts with standard Playwright login code. For TOTP, implement RFC 6238 using node:crypto (HMAC-SHA1 + base32 decode) — no third-party dependency needed.

   Verify auth before proceeding. Run npx shiplight test --headed to execute the auth setup and confirm it saves storage-state.json. If it fails, escalate to the user — auth is a prerequisite for everything else.

   If the test plan involves special auth requirements (e.g., one account per test, multiple roles), confirm the auth strategy with the user before proceeding.

Write tests

For each test in the plan (or each test the user wants):

1. Open a browser session — call new_session with the app's starting_url.
2. Walk through the flow — use inspect_page to see the page, then act to perform each action. This captures locators from the response.
3. Capture locators — use get_locators for additional element info when needed.
4. Build the YAML — construct the .test.yaml content following the best practices below.
5. Save and validate — write the .test.yaml file, then call validate_yaml_test with the file path to check locator coverage (minimum 50% required).
6. Close the session — call close_session when done.

Important: Do NOT write YAML tests from imagination. Always walk through the app in a browser session first to capture real locators. Tests without locators are rejected by validate_yaml_test.

When guided by test-plan.md:

• Apply the specified wait strategy at loading points
• Cover the edge cases and assertions defined in the spec

Run tests

After writing all tests, run them:

npx shiplight test --headed

When a test fails:

1. Report — tell the user which test failed and why (one sentence).
2. Classify the failure:
   • Implementation fix (wrong locator, missing wait, timing) → fix and retry.
   • Spec mismatch (app behavior differs from spec) → ask the user whether to update the spec or skip the scenario.
3. Escalate if a fix doesn't work — don't keep retrying the same approach.

Phase 5: Verify

Goal: Validate test coverage against the spec and reconcile any drift.

Input: reads test-specs/test-spec.md, test-specs/test-plan.md, and all .test.yaml files

This phase only runs when spec artifacts exist.

Coverage check

For each spec journey, confirm the test covers the happy path and all listed edge cases.

Present a coverage summary:

Flag gaps and extras (test steps not in the spec).

Reconcile

Update spec artifacts to match what was actually implemented:

1. Update test-spec.md — mark skipped scenarios with reason, add scenarios that emerged during implementation, update edge cases to reflect what was tested
2. Update test-plan.md — correct file structure, note deviations from the original plan
3. Show diff summary — tell the user what changed and why

This keeps artifacts accurate for future test maintenance and expansion.

YAML Format Reference

Read the MCP resource shiplight://yaml-test-spec-v1.3.0 for the full language spec (statement types, templates, variables, suites, hooks, parameterized tests).

Read the MCP resource shiplight://schemas/action-entity for the full list of available actions and their parameters.

YAML Authoring Best Practices

These best practices bridge the YAML language spec and the action catalog to help you write fast, reliable tests.

Statement type selection

• ACTION is the default. Capture locators via MCP tools (act, get_locators) during browser sessions, then write ACTION statements. ACTIONs replay deterministically (~1s).
• DRAFT is a last resort. Only use DRAFT when the locator is genuinely unknowable at authoring time. DRAFTs are slow (~5-10s each, AI resolution at runtime). Tests with too many DRAFTs are rejected by validate_yaml_test.
• VERIFY for assertions. Use VERIFY: for all assertions. Do not write assertion DRAFTs like "Check that the button is visible".
• URL for navigation. Use URL: /path for navigation instead of action: go_to_url.
• CODE for scripting. Use CODE: for network mocking, localStorage manipulation, page-level scripting. Not for clicks, assertions, or navigation.

The intent field

intent is the intent of the step — it defines what the step should accomplish. The action/locator or js fields are caches of how to do it. When a cache fails (stale locator, changed DOM), the AI agent uses intent to re-inspect the page and regenerate the action from scratch.

Because intent drives self-healing, it must be specific enough for an agent to act on without any other context. Describe the user goal, not the DOM element — avoid element indices, CSS selectors, or positional references that break when the UI changes:

# BAD: vague, agent can't re-derive the action
- intent: Click button

# BAD: tied to DOM structure that can change
- intent: Click the 3rd button in the form
- intent: Click element at index 42

# GOOD: describes the user goal, stable across UI changes
- intent: Click the Submit button to save the new project
  action: click
  locator: "getByRole('button', { name: 'Submit' })"

ACTION: structured format vs js: shorthand

Use structured format by default for all supported actions. Read the MCP resource shiplight://schemas/action-entity for the full list of available actions and their parameters.

Use js: only when the action doesn't map to a supported action — e.g., complex multi-step interactions, custom Playwright API calls, or chained operations:

- intent: Drag slider to 50% position
  js: "await page.getByRole('slider').first().fill('50')"

- intent: Wait for network idle after form submit
  js: "await page.waitForLoadState('networkidle')"

js: coding rules

• Always resolve locators to a single element (e.g., .first(), .nth(1)) to avoid Playwright strict-mode errors
• Always include { timeout: 5000 } on actions for predictable timing
• The intent is critical — it's the input for self-healing when js fails
• page, agent, and expect are available in scope

VERIFY best practices

• Always set a short timeout (e.g., { timeout: 2000 }) on js: assertions that have an AI fallback, so stale locators fall back to AI quickly instead of waiting the default 5s
• Always use VERIFY: shorthand — do not use action: verify directly
• Be aware of false negatives with js: assertions. The AI fallback only triggers when js throws (element not found, timeout). If js passes against the wrong element (stale selector matching a different element), the assertion silently succeeds — no fallback occurs. Keep js: assertions simple and specific to minimize this risk.

IF/WHILE js: condition best practices

• Use natural language (AI) conditions for DOM-based checks (element visible, text present, page state). AI conditions self-heal against DOM changes; js: conditions are brittle and cannot auto-heal.
• Use js: conditions only for counter/state logic — e.g., js: counter++ < 10, js: retryCount < 3. Never use js: for DOM inspection like js: document.querySelector('.modal') !== null.
• If you need a JavaScript-based DOM check, use CODE: to evaluate it and store the result, or use VERIFY: with js: (which at least has AI fallback on failure).

Waiting syntax

• WAIT_UNTIL: — AI checks the condition repeatedly until met or timeout. Default timeout is 60 seconds. Each AI check takes 5–10s, so set timeout_seconds to at least 15.
• WAIT: — fixed-duration pause. Use seconds: to set duration.

See Smart waiting in E2E Test Design for when to use each.

General conventions

• Put intent first in ACTION statements for readability
• xpath is only needed when an ACTION has neither locator nor js.
• Single-test vs Suite vs Parameters:
  • Single-test file — one isolated test, no shared state
  • Suite — tests that have sequential dependencies (e.g., test A creates a file, test B consumes it). Each test in a suite still covers one journey — the suite just guarantees execution order and shares browser state. Do NOT use suites to bundle unrelated tests.
  • Parameters — same test structure, different data inputs

E2E Test Design Best Practices

These principles govern what to test and how to structure tests — independent of the YAML format. Apply them during Phase 2 (Specify) and Phase 4 (Implement).

Test isolation

Each test must run independently — never depend on another test's side effects, execution order, or leftover state. If a test needs data, it creates that data itself.

# BAD: depends on a previous test having created "My Project"