test-creator

Every step must complete before proceeding. Do not skip steps or merge them together.

Step 1: Deep Project Analysis

This step is mandatory and must be thorough. Shallow analysis is the root cause of incomplete tests. Do not proceed to Step 2 until all 4 sub-agents have returned results.

Run 4 sub-agents in parallel

Use the prompt templates in references/project-analysis-prompts.md to invoke:

Author agent:
  → Sub-agent 1: API discovery       (parallel)
  → Sub-agent 2: Page discovery      (parallel)
  → Sub-agent 3: Data model          (parallel)
  → Sub-agent 4: Logging config      (parallel)
  ↓
Collect all 4 results before proceeding

What to store from analysis results

• Complete endpoint inventory (used to ensure every endpoint has a test)
• Complete page/flow inventory (used to ensure every flow has an E2E test)
• DB connection string / config file path (used for Test Point 4: Data Validation)
• Log query command + format (used for Test Point 5: Log Validation)
• Tech stack + test framework (used for adapter selection in Step 6)

Step 2: Pre-research Q&A Page

Generate an interactive HTML page so users can visually select options instead of answering questions one by one in the terminal.

Use scripts/generate-qa-page.sh to generate the page with project data injected from Step 1.

What the page collects

1. Test types (multi-select, all checked by default):

2. Modules to cover:

• Module list is auto-generated from Step 1 analysis
• All modules checked by default; all use the same selected test types

3. Test environment:

URL supports "Auto-detect" — agent reads from project config.

4. Database configuration (required for Test Point 4: Data Validation):

5. Log configuration (required for Test Point 5: Log Validation):

6. Test data strategy:

7. Additional context:

• Existing test coverage estimate?
• Known bugs or weak areas to focus on?
• CI/CD pipeline — should tests integrate?

Page implementation flow

Step 1 results (endpoints, modules, stack)
  → scripts/generate-qa-page.sh injects data into HTML template
  → Script prints:
      "Open in browser: file:///D:/path/to/tests/qa-page.html"
      "Config will be read from: D:/path/to/tests/qa-config.json"
  → User opens page in browser, fills in selections
  → User clicks "Save Config" → browser downloads qa-config.json
  → User saves qa-config.json to the same directory as the HTML file
  → User tells agent "config done"
  → Agent reads <html-dir>/qa-config.json directly
  → Agent uses JSON to drive Steps 3 → 7

See references/qa-page-spec.md for HTML structure details.

Step 3: Test Development Plan

Before writing any test code, produce a written test development plan document and confirm with the user.

What the plan must contain

File: tests/TEST-PLAN.md (or equivalent path under the test directory)

# Test Development Plan

## Project: <name>
## Generated: <date>
## Config: <summary of Q&A selections>

## Test Directory Structure
tests/
  api/          — API test files
  e2e/          — E2E test files
  unit/         — Unit test files
  integration/  — Integration test files
  fixtures/     — Shared test data and factories
  helpers/      — Shared test utilities
  TEST-PLAN.md  — This file

## API Test Coverage
For each endpoint from Step 1 analysis:
| Endpoint | Test file | 6 points covered |
|----------|-----------|-----------------|
| POST /api/v1/users | tests/api/test_users.py | 1,2,3,4,5,6 |
...

## E2E Test Coverage
For each page/flow from Step 1 analysis:
| Page/Flow | Test file | Scenarios |
|-----------|-----------|-----------|
| /login → /dashboard | tests/e2e/test_auth_flow.py | happy path, wrong password, session expiry |
...

## Data Validation Strategy
- DB connection: <how tests will connect to DB>
- Validation approach: <query pattern used to verify stored data>

## Log Validation Strategy
- Log query: <command to read logs during tests>
- Key operations to verify: <list>

Do not begin implementation until the user has reviewed and approved this plan.

Step 4: Test System Design

Using the Q&A JSON output + the test matrix below, finalize the test plan.

4 Test Categories

6 Mandatory Test Points (per selected type)

Once a test type is selected, ALL 6 points must be covered. No cherry-picking. If a point genuinely does not apply (e.g., no HTTP layer in a unit test), add a comment explaining why — do not silently omit it.

Test Point 4 — critical rule: Data validation means querying the DB (or cache) directly, not inspecting response field types. See references/test-coverage-details.md for the correct pattern.

See references/test-coverage-details.md for per-type coverage specifics.

Step 5: Implementation

Test file organization

All test-related files go under a single tests/ directory. Do not scatter test files in the project root or alongside source files (except unit tests co-located by framework convention).

tests/
  api/              — API tests
  e2e/              — E2E / browser tests
  unit/             — Unit tests
  integration/      — Integration tests
  fixtures/         — Test data factories and static fixtures
  helpers/          — Shared utilities (DB helpers, log capture, auth helpers)
  TEST-PLAN.md      — Test development plan (from Step 3)
  .test-creator-quality.json  — Quality thresholds config

Generate the following

1. Test files — one file per module per test type, organized per structure above
2. Fixtures — data factories and static fixtures in tests/fixtures/
3. Helpers — DB query helpers, log capture utilities, auth token helpers in tests/helpers/
4. Test runner config — framework configuration file
5. CI integration — pipeline config if user requested it

Implementation rules

• Every test file must cover all 6 mandatory test points for its module × type combination
• Test Point 4 must use direct DB/storage queries — never just inspect response fields
• Test Point 5 must capture and assert on actual log output
• Do not fix source code bugs found during implementation — document them in tests/TEST-PLAN.md under a "Known Issues" section

Step 6: Verification

After implementation, always run run-all-checks.sh. Do not run the test framework directly.

run-all-checks.sh provides coverage analysis, flaky test detection, and performance profiling that raw test commands do not. Running pytest or npm test directly bypasses all of this.

How to invoke

<skill-root>/scripts/run-all-checks.sh --project-path <path> --test-cmd <command> [--config <path>] [--output <path>]

For example, if your skill root is at /home/user/.claude/skills/test-creator:

/home/user/.claude/skills/test-creator/scripts/run-all-checks.sh \
  --project-path ./my-project \
  --test-cmd "npm test"

The --output flag is optional — it defaults to <project-path>/tests. To override:

/home/user/.claude/skills/test-creator/scripts/run-all-checks.sh \
  --project-path ./my-project \
  --test-cmd "npm test" \
  --output ./my-project/tests

What it does

1. Detects your test framework (Jest, pytest, Go test, etc.) via adapter
2. Runs tests with coverage and parses the results
3. Re-runs tests N times to detect flaky tests
4. Records per-case timing to flag slow tests
5. Generates quality-report.json and quality-report.md in the output path

Available adapters

If your project uses a framework not listed, create a new adapter following the interface in references/quality-scripts.md, then re-run.

If the script fails

Do NOT bypass it by running test commands directly. Debug the failure:

1. Check if an adapter was detected — if not, create one for your framework
2. Check if the test command works when run directly — if not, fix test setup first
3. Check adapter functions — detect(), get_test_cmd(), get_coverage_cmd() must all work
4. Re-run the script after fixing

Step 7: Quality Evaluation

Testing the tests. Two layers: automated scripts for mechanical checks, sub-agent review for semantic checks.

Layer 1: Automated Quality Scripts (already done in Step 6)

The run-all-checks.sh output from Step 6 IS the Layer 1 result. Review the generated quality-report.json and quality-report.md.

Config file (tests/.test-creator-quality.json):

{
  "coverage": { "line_threshold": 80, "branch_threshold": 70 },
  "flaky": { "runs": 5 },
  "performance": {
    "slow_unit_threshold_ms": 1000,
    "slow_api_threshold_ms": 5000,
    "slow_e2e_threshold_ms": 30000
  }
}

Layer 2: Sub-agent Deep Review

Scripts cannot judge whether tests are actually correct or sufficient. That requires semantic analysis — and must be done by a sub-agent, not the author agent.

An agent reviewing its own work has inherent bias. Sub-agent review provides adversarial perspective.

How to invoke the sub-agent:

1. Pass the sub-agent: quality-report.json, test file paths, source file paths, Q&A config JSON, and Step 1 project analysis results
2. Use the prompt template in references/sub-agent-review-prompt.md
3. Sub-agent returns a prioritized issue list
4. Author agent fixes test issues, re-runs run-all-checks.sh + sub-agent review
5. Loop until all high/medium issues are resolved

When bugs are found: Document them in tests/TEST-PLAN.md under "Issues Found". Do not fix source code.

Complete Quality Flow

Step 6: run-all-checks.sh → quality-report.json
  → Step 7 Layer 1: Review quality-report for mechanical issues
  → Step 7 Layer 2: Sub-agent review (report + tests + source + requirements + project analysis)
  → Merge: scripts = quantitative, sub-agent = qualitative
  → Fix test issues by severity → re-run run-all-checks.sh → re-invoke sub-agent → loop until clean

Delivery Checklist