Sdd Verify

Rule: Layers 1-3 MUST pass before requesting human layers 4-5. If the base is broken, do not waste human time on review.

What You Receive

From the orchestrator:

• Change name
• The proposal.md content
• The delta specs from specs/
• The design.md content
• The tasks.md content (with completion status)
• Project config from openspec/config.yaml

Execution and Persistence Contract

From the orchestrator:

• artifact_store.mode: auto | engram | openspec | none
• detail_level: concise | standard | deep

Rules:

• If mode resolves to none, do not create report files; return verification result only.
• If mode resolves to engram, persist verification report in Engram and return references.
• If mode resolves to openspec, save verify-report.md as defined in this skill.

What to Do

Step 1: Check Completeness

Verify ALL tasks are done:

Read tasks.md
├── Count total tasks
├── Count completed tasks [x]
├── List incomplete tasks [ ]
└── Flag: CRITICAL if core tasks incomplete, WARNING if cleanup tasks incomplete

Step 2: Check Correctness (Specs Match) — Verification Matrix

For EACH spec requirement and scenario, build a verification matrix cross-referencing specs against implementation evidence:

FOR EACH REQUIREMENT in specs/:
├── Search codebase for implementation evidence
├── For each SCENARIO:
│   ├── Is the GIVEN precondition handled?
│   ├── Is the WHEN action implemented?
│   ├── Is the THEN outcome produced?
│   └── Are edge cases covered?
└── Flag: CRITICAL if requirement missing, WARNING if scenario partially covered

Step 3: Check Coherence (Design Match)

Verify design decisions were followed:

FOR EACH DECISION in design.md:
├── Was the chosen approach actually used?
├── Were rejected alternatives accidentally implemented?
├── Do file changes match the "File Changes" table?
└── Flag: WARNING if deviation found (may be valid improvement)

Step 3.5: Pyramid Layer 1 — Type Checking, Linting & Build (Mandatory)

This is the base of the AI Validation Pyramid. If this layer fails, ALL higher layers are invalid.

LAYER 1 CHECK (automated by agent):
├── 1a. Linting (if configured):
│   ├── Node.js: npx eslint . --max-warnings=0 (or biome, oxlint)
│   ├── Python: ruff check . (or flake8, pylint)
│   ├── Go: golangci-lint run
│   └── If no linter → WARNING: recommend adding one
│
├── 1b. Type Checking (if configured):
│   ├── TypeScript: npx tsc --noEmit
│   ├── Python: mypy . (or pyright)
│   ├── Go: go vet ./...
│   └── If no type checker → SUGGESTION: recommend adding one
│
├── 1c. Build:
│   ├── Node.js: npm run build (or npx next build for Next.js)
│   ├── Python: python -m py_compile or pytest --collect-only
│   ├── Go: go build ./...
│   └── Docker: docker build . (if Dockerfile exists)
│
├── If ANY of 1a/1b/1c FAILS → CRITICAL (pyramid base broken, stop here)
├── If all pass with warnings → WARNING (note them)
└── If all pass clean → PASS Layer 1

FALLBACK (when no venv/dev environment exists):

IF no virtual environment or node_modules found:
├── Python: Use `python -c "import ast; ast.parse(open(f).read())"` for syntax validation per file
├── Node.js: Use `node --check {file}` for syntax validation per file
├── Go: `go vet` works without extra setup
├── Any language: Verify imports resolve, basic file structure is valid
├── Mark Layer 1 as PARTIAL (not SKIP) with note: "No dev environment — syntax-only validation performed"
└── NEVER mark Layer 1 as SKIP entirely — always run what you can

The fallback exists because the base of the Validation Pyramid should never be entirely empty. Even syntax-only validation catches import errors, typos, and structural problems that would otherwise reach human reviewers.

This step is NOT optional. The Validation Pyramid principle: automate the base layers to catch problems before they reach human reviewers. If a build/lint/type-check command is not available, use the fallback above. Never skip Layer 1 entirely.

Step 3.6: Pyramid Layer 1e — Sync-in-Async Detection

Scan changed files for common async anti-patterns that cause production performance issues:

SYNC-IN-ASYNC CHECK (automated by agent):
├── Search for async def functions in changed files
├── Inside each async def, flag these anti-patterns:
│   ├── sync HTTP clients: requests.get/post, httpx.Client (not AsyncClient), anthropic.Anthropic (not AsyncAnthropic)
│   ├── sync file I/O: open() without aiofiles in async handlers
│   ├── sync database calls: engine.execute(), session.query() without async equivalents
│   ├── blocking sleep: time.sleep() instead of asyncio.sleep()
│   └── sync subprocess: subprocess.run() instead of asyncio.create_subprocess_exec()
├── If found → WARNING: "Sync operation inside async function blocks the event loop. Use async equivalent."
├── Include specific file:line references for each finding
└── This check is INFORMATIONAL (WARNING, not CRITICAL) — but flag prominently as performance risk

This check exists because sync-in-async bugs are invisible in development (single user) but catastrophic in production (blocks the event loop for all concurrent requests). Manual code review often misses them.

Step 3.7: Pyramid Layer 1d — Code Documentation Check (MANDATORY)

Verify that generated code meets Batuta's documentation standard (DOCUMENTATION > CODE):

LAYER 1d CHECK (automated by agent):
├── For each source file in the change:
│   ├── Has module docstring? (first line with """ or // or /*)
│   ├── Count public functions/methods
│   ├── Count documented public functions (with docstring)
│   └── Calculate documentation ratio
│
├── Aggregate metrics:
│   ├── Total files: {N}
│   ├── Files with module docstring: {M}
│   ├── Total public functions: {P}
│   ├── Documented public functions: {D}
│   ├── Documentation ratio: {D/P * 100}%
│   └── Security comments present: {count of # SECURITY: lines}
│
├── Thresholds:
│   ├── Module docstrings < 100% of files → WARNING
│   ├── Function doc ratio 80-99% → WARNING ("documentation almost complete")
│   ├── Function doc ratio < 80% → CRITICAL ("documentation incomplete — fix before proceeding")
│   ├── Function doc ratio < 50% → CRITICAL + FAIL ("code without documentation is incomplete code — pyramid base broken")
│   └── Zero security comments on auth/crypto code → WARNING
│
└── Report: Include documentation metrics in Layer 1 results

This check ensures the Batuta philosophy (DOCUMENTATION > CODE) is enforced automatically, not left to human discipline.

Step 4: Pyramid Layer 2 — Unit Tests

Verify test coverage for spec scenarios (automated by agent):

LAYER 2 CHECK:
├── Search for test files related to the change
├── Do tests exist for each spec scenario?
├── Do tests cover happy paths?
├── Do tests cover edge cases?
├── Do tests cover error states?
├── If test runner exists (npm test, pytest, go test), RUN tests and report results
├── If tests FAIL → CRITICAL (do not proceed to Layer 3)
└── Flag: WARNING if scenarios lack tests, SUGGESTION if coverage could improve

Step 4.5: Pyramid Layer 3 — Integration / E2E Tests (When Possible)

The top agent-automated layer. If the project has a dev server and the environment supports it:

LAYER 3 CHECK (best-effort, not blocking):
├── If webapp with Playwright configured:
│   ├── Start dev server (npm run dev)
│   ├── Run Playwright tests against spec scenarios
│   └── Report: which scenarios pass in real browser
├── If API project:
│   ├── Start server
│   ├── Run health check endpoint
│   └── Report: server responds correctly
├── If neither is available:
│   └── Note: "Layer 3 skipped — no E2E/integration test framework configured"
└── This step is BEST-EFFORT: if it fails, report as SUGGESTION, not CRITICAL

Playwright integration: If Playwright MCP is available and the project is a webapp, use it to validate spec scenarios SC-xxx in a real browser. This provides true E2E validation beyond code inspection.

Step 4.7: Cross-Layer Security Check

After Pyramid Layers 1-3 pass, run the security-audit skill checklist:

SECURITY CHECK (cross-layer, automated by agent):
├── Run AI-First Security Checklist (10 points) against changed files
├── Run secrets scanning protocol (grep for secret patterns)
├── Run dependency audit (if new deps were added)
├── If CRITICAL finding → FAIL (block deploy, fix immediately)
├── If HIGH finding → WARNING (fix before deploy)
├── If MEDIUM/LOW → NOTE in report
└── Reference: security-audit SKILL.md for full checklist

This check integrates with the Threat Model from sdd-design: verify that all mitigations documented in the threat model are actually implemented.

Step 4.8: Testing Strategy by Solution Type

Identify the solution type from design.md and apply additional checks:

Type A — Pure Automation (no AI): Standard pyramid Layers 1-3 only.

Type B — Automation + LLM: Type A plus golden dataset testing (N inputs, compare outputs), confidence score validation, cost verification vs proposal budget, prompt regression check.

Type C — Full Agent: Type B plus behavior boundary testing (guardrails), escalation path testing (human handoff), drift detection baseline, multi-turn consistency.

Report solution type and additional checks in the verification report under a "Solution Type Testing" section.

Step 4.9: Reality Check Protocol

Principle: Default to NEEDS WORK. The burden of proof is on the implementation, not the verifier. First iterations typically need 2-3 revision cycles — a C+/B- rating is normal and acceptable. A+ requires extraordinary evidence.

Before moving to documentation verification, run these five adversarial questions. Each challenges a different assumption that standard verification takes for granted:

REALITY CHECK (mandatory — never skip):

├── Q1: "Is the spec itself correct?"
│   ├── Re-read the original proposal.md
│   ├── Compare proposal intent vs spec requirements
│   ├── Does the spec capture what the user actually asked for, or was something lost in translation?
│   └── Flag: WARNING if spec diverges from proposal intent
│
├── Q2: "Did we build what the user needs vs what we specified?"
│   ├── Does the implementation solve the original problem (proposal), not just satisfy the spec?
│   ├── Could a user use this and still be unsatisfied because the spec missed the point?
│   └── Flag: WARNING if implementation is spec-correct but proposal-incomplete
│
├── Q3: "Is there scope creep?"
│   ├── Does the implementation include features NOT in the spec or tasks?
│   ├── Were files modified that weren't in the design's File Changes table?
│   ├── Is there "improvement" code beyond what was requested?
│   └── Flag: WARNING if scope grew beyond spec (even if the additions seem useful)
│
├── Q4: "Does this only work on the happy path?"
│   ├── What happens with empty input? Null? Unexpected types?
│   ├── What happens under concurrent access, high load, or network failure?
│   ├── Do error paths return meaningful feedback, or silently fail?
│   ├── Are there scenarios the tests DON'T cover that a real user would trigger?
│   └── Flag: WARNING if golden-path bias detected, CRITICAL if error paths are unhandled
│
└── Q5: "Is the evidence real or self-reported?"
    ├── Were tests actually RUN (not just "tests exist")?
    ├── Were linting results from this session (not cached/stale)?
    ├── Do pass/fail claims match actual tool output?
    ├── Are there "PASS" verdicts on layers that were never executed?
    └── Flag: CRITICAL if evidence is fabricated or assumed, WARNING if evidence is stale

Automatic Fail Triggers — if ANY of these appear in the verification, escalate to CRITICAL:

• "Zero issues found" claim without comprehensive evidence
• Perfect scores (100%, all-PASS) on a first implementation
• "Production ready" verdict when Pyramid Layers 1-3 were not all executed
• Claims that contradict actual tool output (e.g., "tests pass" when test runner reported failures)
• PASS on a layer that was marked SKIP

Integration with report: Include Reality Check results as a new section in the verification report, between "Testing Detail" and "Documentation Verification":

### Reality Check
| Question | Verdict | Notes |
|----------|---------|-------|
| Spec correctness | PASS / WARNING | {details} |
| User need alignment | PASS / WARNING | {details} |
| Scope creep | PASS / WARNING | {details} |
| Happy-path bias | PASS / WARNING / CRITICAL | {details} |
| Evidence authenticity | PASS / WARNING / CRITICAL | {details} |

Step 5: Documentation Verification

Verify that ALL documentation promised or implied during the design phase was actually created or updated. Documentation is a first-class deliverable, not an afterthought.

FOR EACH documentation commitment:
├── README Updates
│   ├── Was the project README updated if new features/APIs/config were added?
│   ├── Are setup/install instructions still accurate after the change?
│   └── Flag: WARNING if README is stale, CRITICAL if new feature has zero README mention
│
├── API Documentation
│   ├── Are new endpoints/functions/interfaces documented?
│   ├── Do docs include request/response examples?
│   ├── Are error codes and edge cases documented?
│   └── Flag: CRITICAL if public API lacks documentation, WARNING if incomplete
│
├── Architecture Documentation
│   ├── Were architecture decision records (ADRs) created for significant decisions?
│   ├── Do diagrams reflect the current state (if diagrams exist in the project)?
│   ├── Are dependency changes documented?
│   └── Flag: WARNING if significant architectural change lacks documentation
│
├── Inline Code Documentation
│   ├── Do complex functions have comments explaining WHY (not WHAT)?
│   ├── Are non-obvious business rules annotated?
│   ├── Are workarounds and technical debt marked with TODO/FIXME and context?
│   ├── Are public interfaces documented with JSDoc/TSDoc/docstrings as appropriate?
│   └── Flag: SUGGESTION if WHY-comments are missing on complex logic
│
├── Changelog / Migration Notes
│   ├── Were breaking changes documented?
│   ├── Are migration steps provided if applicable?
│   └── Flag: CRITICAL if breaking change lacks migration guide
│
└── Stakeholder-Facing Documentation
    ├── Were user-facing docs updated (help pages, tooltips, error messages)?
    ├── Is the change visible to users? If yes, is it communicated?
    └── Flag: WARNING if user-visible change lacks user-facing documentation

Step 6: O.R.T.A. Checklist Verification

Verify operational readiness using the O.R.T.A. framework. Every production-bound change must meet these four pillars:

O.R.T.A. VERIFICATION:

├── [O] Observability — Can we SEE what is happening?
│   ├── Are structured logs emitted at key decision points?
│   ├── Are log levels appropriate (info for flow, warn for recoverable, error for failures)?
│   ├── Is distributed tracing propagated (trace IDs, span context)?
│   ├── Are meaningful metrics exposed (counters, gauges, histograms)?
│   ├── Are health check endpoints updated if applicable?
│   └── Flag: WARNING if logging is absent, CRITICAL if error paths have no observability
│
├── [R] Repeatability — Can we REPRODUCE it reliably?
│   ├── Is the change deterministic (same inputs produce same outputs)?
│   ├── Can the deployment be repeated without manual steps?
│   ├── Are environment-specific configs externalized (not hardcoded)?
│   ├── Is there a seed/fixture strategy for data-dependent behavior?
│   ├── Can a new developer set up and run this change from scratch using only the docs?
│   └── Flag: WARNING if manual steps required, CRITICAL if non-deterministic behavior detected
│
├── [T] Traceability — Can we TRACE what happened and why?
│   ├── Are changes linked to the original proposal/ticket/issue?
│   ├── Do commits reference the change name or ticket ID?
│   ├── Is there an audit trail for data mutations (who changed what, when)?
│   ├── Are database migrations versioned and reversible?
│   ├── Can we reconstruct the sequence of events from logs alone?
│   └── Flag: WARNING if audit trail is incomplete, CRITICAL for data mutations without tracing
│
└── [A] Auto-supervision — Can the system MONITOR itself?
    ├── Are alerts configured for failure conditions?
    ├── Do circuits/breakers exist for external dependencies?
    ├── Is there a graceful degradation strategy?
    ├── Are retry policies defined with backoff?
    ├── Does the system detect and report its own unhealthy state?
    └── Flag: SUGGESTION if self-monitoring could improve, WARNING if no failure detection exists

Batuta Educator Note: When reporting O.R.T.A. results to stakeholders, translate each finding into business terms. Example: "Observability gap in payment flow" becomes "If the payment system fails, the team cannot quickly diagnose the cause, increasing downtime from minutes to potentially hours."

Step 7: Save Verification Report

Create the verification report file:

openspec/changes/{change-name}/
├── proposal.md
├── specs/
├── design.md
├── tasks.md
└── verify-report.md          ← You create this

Step 8: Return Summary

Return to the orchestrator the same content you wrote to verify-report.md:

## Verification Report

**Change**: {change-name}
**Verified by**: sdd-verify (Batuta)
**Date**: {ISO-8601 date}

### Executive Summary (Non-Technical)

{2-3 sentences a product owner can understand. What was built, does it work, what is the risk level?}

### Completeness
| Metric | Value |
|--------|-------|
| Tasks total | {N} |
| Tasks complete | {N} |
| Tasks incomplete | {N} |

{List incomplete tasks if any}

### Correctness (Specs) — Verification Matrix
| Requirement | Status | Notes |
|------------|--------|-------|
| {Req name} | PASS | {brief note} |
| {Req name} | PARTIAL | {what's missing} |
| {Req name} | FAIL | {not implemented} |

**Scenarios Coverage:**
| Scenario | Status |
|----------|--------|
| {Scenario name} | PASS |
| {Scenario name} | PARTIAL |
| {Scenario name} | FAIL |

### Coherence (Design)
| Decision | Followed? | Notes |
|----------|-----------|-------|
| {Decision name} | Yes | |
| {Decision name} | Deviated | {how and why} |

### AI Validation Pyramid Status
| Layer | Owner | Status | Notes |
|-------|-------|--------|-------|
| 1. Type Check / Lint | AGENT | PASS / PARTIAL / FAIL | {lint + type + build results} |
| 2. Unit Tests | AGENT | PASS / PARTIAL / FAIL | {test run results} |
| 3. Integration / E2E | AGENT | PASS / PARTIAL / SKIP | {E2E results or "not configured"} |
| 4. Code Review | HUMAN | PENDING | {key concerns for reviewer} |
| 5. Manual Testing | HUMAN | PENDING | {scenarios requiring manual validation} |

**Agent Layers (1-3) Score**: {count passing}/3
{If any agent layer FAILS: "Base layers incomplete — resolve before requesting human review."}

### Testing Detail
| Area | Tests Exist? | Coverage |
|------|-------------|----------|
| {area} | Yes/No | {Good/Partial/None} |

### Documentation Verification
| Document Type | Status | Notes |
|--------------|--------|-------|
| README | PASS / PARTIAL / FAIL / N/A | {details} |
| API Docs | PASS / PARTIAL / FAIL / N/A | {details} |
| Architecture Docs | PASS / PARTIAL / FAIL / N/A | {details} |
| Inline Comments (WHY) | PASS / PARTIAL / FAIL / N/A | {details} |
| Changelog / Migration | PASS / PARTIAL / FAIL / N/A | {details} |
| Stakeholder-Facing Docs | PASS / PARTIAL / FAIL / N/A | {details} |

**Documentation Debt**: {count of PARTIAL + FAIL items}
{Brief description of what documentation is missing and business impact}

### O.R.T.A. Operational Readiness
| Pillar | Status | Key Findings |
|--------|--------|-------------|
| **[O] Observability** | PASS / PARTIAL / FAIL | {summary} |
| **[R] Repeatability** | PASS / PARTIAL / FAIL | {summary} |
| **[T] Traceability** | PASS / PARTIAL / FAIL | {summary} |
| **[A] Auto-supervision** | PASS / PARTIAL / FAIL | {summary} |

**O.R.T.A. Score**: {count of PASS}/4 pillars passing
{One sentence: business-impact translation of any failures}

### Issues Found

**CRITICAL** (must fix before archive):
{List or "None"}
{For each: one-sentence business impact}

**WARNING** (should fix):
{List or "None"}

**SUGGESTION** (nice to have):
{List or "None"}

### Verdict
{PASS / PASS WITH WARNINGS / FAIL}

{One-line summary of overall status}
{One-line business-impact summary for non-technical stakeholders}

### Archive Readiness
**archive_ready**: {true/false}

{Deterministic field for sdd-archive to check. Rules:
- `true`: No CRITICAL issues that block archive. PASS or PASS WITH WARNINGS verdict.
- `false`: CRITICAL issues exist that must be fixed before archiving, OR verdict is FAIL.
This field removes ambiguity about whether "CRITICAL (fix before deploy)" means "also blocks archive."}

Sub-Agent Output Contract

Every response from this skill MUST return a structured envelope with the following fields: