S3 Audit

Evidence Standard: Every claim in the audit must cite specific file paths and line numbers.

Why tiered? An audit gate that only warns is a notification system, not a control. Missing verification evidence means the feature effectively does not exist in a compliant state — approving it creates "phantom compliance." Failed tests promoted to production create immediate technical debt and potential liability.

Audit Rules (R1-R5) — Advisory Checks

R1: Pinned References Check

WARN IF: Context Pack or any artifact contains:

• Generic version references: "latest", "stable", "current", "newest"
• Missing version/date for external libraries/docs
• Unpinned Git references (use commit hash, not branch name)

VERIFICATION:

1. Read .quint/context.md
2. Search for patterns: latest, stable, current (case insensitive)
3. Verify all citations include: version number OR commit hash OR date

VALID EXAMPLES:

• ✅ "React 18.2.0"
• ✅ "Node.js v20.11.0 (LTS)"
• ✅ "commit a1b2c3d"
• ✅ "Express 4.18.x (accessed 2024-01-15)"

INVALID EXAMPLES:

• ❌ "Latest React"
• ❌ "Node stable"
• ❌ "main branch"
• ❌ "current version"

R2: Assumption Ledger Check + Implicit Assumption Detection

WARN IF:

• Any OPEN items exist without WAIVER justification
• Assumptions lack evidence citations
• Implicit assumptions discovered during audit (not in ledger)

VERIFICATION:

1. Read .quint/context.md Assumption Ledger section
2. Verify all items have status: VERIFIED or WAIVER
3. Search implementation for assumptions not in ledger:
   • Hardcoded paths
   • Environment variable dependencies
   • Service availability assumptions
   • Data format assumptions

IMPLICIT ASSUMPTION DETECTION: Flag patterns like:

// Implicit: process.env.API_KEY exists
const apiKey = process.env.API_KEY;

// Implicit: /tmp directory exists and is writable
fs.writeFileSync('/tmp/data.json', data);

// Implicit: Redis is running on localhost:6379
redis.connect('localhost:6379');

R3: Constraint Drift Check (via Diff)

WARN IF: DRR Constraints Bundle differs from S2 artifacts:

• Constraints dropped or weakened in spec.md
• NFRs missing from original DRR
• Performance/security constraints omitted in tasks.md

VERIFICATION:

1. Read DRR Constraints Bundle from .quint/decisions/<drr-id>.md
2. Read S2 openspec/changes/<change-id>/spec.md
3. Read S2 openspec/changes/<change-id>/tasks.md
4. Line-by-line comparison:
   • Every C-F* (functional) must appear in spec.md requirements
   • Every C-NF* (non-functional) must appear in spec.md NFRs
   • All constraints must have verification tasks

DRIFT CATEGORIES:

R4: Scope Drift Math

CALCULATION:

Total Modified Files = count(git_status modified + added + deleted)
Out-of-Scope Files = count(files outside DRR scope declaration)
Scope Drift % = (Out-of-Scope Files / Total Modified Files) × 100

THRESHOLDS:

WARN / ESCALATE IF:

• 10% of modified files are outside DRR scope declaration

• Unrequested changes in unrelated modules
• Drive-by refactoring not in spec

VERIFICATION:

1. Run git_status to list all modified files
2. Read DRR Scope Boundaries (IN-SCOPE / OUT-OF-OF-SCOPE)
3. Categorize each modified file:
   • ✅ In-scope (matches DRR scope)
   • ⚠️ Gray area (needs judgment)
   • ❌ Out-of-scope (not in DRR, not incidental)
4. Calculate percentage

EXCLUDED from calculation (incidental):

• Test files for modified code
• Configuration updates required by changes
• Documentation updates for changed features

R5: Verification Evidence Presence and PASS Status

BLOCK IF (critical — archive halted):

• verification_result.json missing
• verify.log missing
• verification_result.json contains "status": "FAIL"
• Test failures exist in verification logs
• Archive executed without PASS evidence

This rule is NEVER waivable. Missing or failed verification evidence means the implementation has not been confirmed to work. Archive cannot proceed until re-verification produces a PASS status.

VERIFICATION:

1. Verify file exists: openspec/changes/<id>/verification_result.json
2. Verify file exists: openspec/changes/<id>/verify.log
3. Parse JSON and verify: status === "PASS"
4. Check all test sections: unit, integration, contract all PASS
5. Verify coverage thresholds met

EVIDENCE STRUCTURE CHECK:

{
  "change_name": "<must match $CHANGE_NAME from S2>",
  "timestamp": "<ISO 8601>",
  "status": "PASS",  // MUST be PASS
  "tests": {
    "unit": { "status": "PASS" },
    "integration": { "status": "PASS" }
  },
  "coverage": { "lines": ">=80", "branches": ">=70" },
  "constraints_verified": [...]  // Must match DRR constraints
}

R6: Skill Execution Integrity Check

WARN IF:

• Execution log is missing for the target DRR
• Required phases/states were skipped (gaps in sequence)
• Tool call failures detected in execution log

VERIFICATION:

1. Read .quint/execution/{drr_id}/execution.jsonl
2. Parse all phase_start and phase_end entries
3. Verify S1 sequence: Q0 → Q1 → Q2 → Q3 → Q4 → Q5
4. Verify S2 sequence: S0 → S1 → S2 → S3 → S5 → S6 → S7
5. Check for any tool_call entries with status: "error"

PHASE SKIP DETECTION:

TOOL FAILURE DETECTION:

// Check execution log for errors
logEntries
  .filter(e => e.type === 'tool_call' && e.status === 'error')
  .map(e => ({
    tool: e.tool_name,
    error: e.error_message,
    timestamp: e.timestamp
  }))

R7: Supply Chain Integrity (AI-Specific)

WARN IF:

• Any new dependency added during this change does not exist in the official package registry
• New package has unusually low download counts / recent registration date (slopsquatting risk)
• Import paths reference package names not present in requirements.txt / package.json / pyproject.toml

Why: LLMs frequently hallucinate package names. Attackers pre-register these names with malicious code ("slopsquatting"). A new dependency that wasn't in the requirements before this change must be verified as real and intentional.

VERIFICATION:

1. Run git diff on requirements.txt / package.json / pyproject.toml to find newly added packages
2. For each new package, verify it exists in the registry (PyPI, npm, etc.)
3. Flag packages with unusually generic or convenience names (e.g., fast-json-parser-v2, requests-helper)

R8: License Contamination (AI-Specific)

WARN IF:

• New code blocks match patterns from GPL/AGPL-licensed libraries
• Verbatim code was likely memorized and regurgitated from an open-source copyleft project
• No license header or provenance comment for substantially copied code blocks

Why: AI models may reproduce memorized GPL/AGPL code into proprietary codebases, inadvertently triggering copyleft obligations. This is an IP risk that standard linters miss.

VERIFICATION:

1. Review any new utility functions or algorithm implementations — are these novel or suspiciously "textbook perfect"?
2. Flag large (>20 lines) new code blocks that implement standard algorithms without attribution
3. Suggest running a snippet-matching tool (Black Duck, FOSSA) on the change if risk is detected

R9: Context Boundary Failures (AI-Specific)

WARN IF:

• Security constraints or auth checks declared early in the development conversation are absent from implementation
• Functions that should check permissions/roles have no auth guard
• Input validation specified in the DRR is missing from boundary functions

Why: LLMs have limited context windows. Security invariants stated at conversation start (e.g., "ensure user is admin") are frequently "forgotten" by the time the relevant function is generated 50+ turns later.

VERIFICATION:

1. Read DRR Constraints Bundle — identify any C-* constraints related to auth, permissions, or input validation
2. Find the implementing functions in the diff
3. Verify each security constraint traces to a concrete check in the code

R10: Prompt Residue & Hardcoded Secrets (AI-Specific)

WARN IF:

• Hardcoded placeholder secrets appear in code (API_KEY="sk-...", password="test123")
• LLM response artifacts are present (e.g., # As an AI, I cannot..., # TODO: implement this)
• High-entropy strings appear in source code that look like secrets/tokens

VERIFICATION:

1. Grep changed files for: sk-, password=, secret=, token=, api_key=
2. Grep for LLM response phrases: As an AI, I cannot, placeholder, TODO: implement
3. Grep for common placeholder patterns: your_key_here, xxxx, 1234

Audit Execution Flow

START
  │
  ▼
┌─────────────────┐
│ Load DRR        │ ← Read `.quint/decisions/<drr-id>.md`
│ Load Context    │ ← Read `.quint/context.md`
│ Load Exec Log   │ ← Read `.quint/execution/{drr_id}/execution.jsonl`
└────────┬────────┘
         │
         ▼
┌──────────────────────────────────────────┐
│ Execute R1  ← Pinned references (WARN)   │
│ Execute R2  ← Assumption ledger (WARN)   │
│ Execute R3  ← Constraint drift (WARN)    │
│ Execute R4  ← Scope drift (WARN/>25%=ESC)│
│ Execute R5  ← Verification (BLOCK)       │
│ Execute R6  ← Execution integrity (WARN) │
│ Execute R7  ← Supply chain (WARN)        │
│ Execute R8  ← License contamination(WARN)│
│ Execute R9  ← Context boundaries (WARN)  │
│ Execute R10 ← Prompt residue (WARN)      │
└────────┬─────────────────────────────────┘
         │
         ▼
┌───────────────────────────────────────────────────┐
│ Determine verdict                                  │
│  • R5 FAIL?         → BLOCKED (archive halted)    │
│  • R4 drift >25%?   → ESCALATE (human required)   │
│  • Warnings only?   → APPROVED_WITH_WARNINGS       │
│  • All pass?        → APPROVED                     │
└────────┬──────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────┐
│ Output Artifacts        │
│ - audit_verdict.md      │
│ - verdict.json          │
│ - Traceability Matrix   │
└─────────────────────────┘

Output Artifacts

1. audit_verdict.md

# Audit Verdict: <DRR_ID>

**Date:** <ISO_TIMESTAMP>
**Auditor:** s3-audit (Zero Trust Gate)
**Status:** [APPROVED | APPROVED_WITH_WARNINGS | ESCALATE | BLOCKED]

## Executive Summary

- **Rule Violations:** <count>
- **Critical Issues:** <count>
- **Recommendation:** [PROCEED | PROCEED_NOTE_WARNINGS | HUMAN_REVIEW_REQUIRED | BLOCKED_RE_VERIFY]

## Traceability Matrix

| Rule | Check | Status | Evidence |
|------|-------|--------|----------|
| R1 | Pinned References | [PASS/WARN] | `<file>:<line> "<citation>"` |
| R2 | Assumption Ledger | [PASS/WARN] | `<count> OPEN, <count> implicit found` |
| R3 | Constraint Drift | [PASS/WARN] | `<diff_summary>` |
| R4 | Scope Drift | [PASS/WARN/ESCALATE] | `<pct>% (<out>/<total> files)` |
| R5 | Verification Evidence | [PASS/BLOCK] | `<verify.json status>` |
| R6 | Execution Integrity | [PASS/WARN] | `<execution.jsonl findings>` |
| R7 | Supply Chain | [PASS/WARN] | `<new packages checked>` |
| R8 | License Contamination | [PASS/WARN] | `<new code blocks reviewed>` |
| R9 | Context Boundaries | [PASS/WARN] | `<security constraints verified>` |
| R10 | Prompt Residue | [PASS/WARN] | `<patterns scanned>` |

## Detailed Findings

### R1: Pinned References
**Status:** [PASS/FAIL]

| Location | Issue | Citation |
|----------|-------|----------|
| `<file>:<line>` | [Generic ref/No version] | `"<text>"` |

### R2: Assumption Ledger
**Status:** [PASS/FAIL]

**Ledger State:**
| ID | Status | Evidence | Waiver |
|----|--------|----------|--------|

**Implicit Assumptions Detected:**
| Location | Assumption | Suggested Ledger Entry |
|----------|------------|------------------------|
| `<file>:<line>` | `<description>` | `A<N>: <description>` |

### R3: Constraint Drift
**Status:** [PASS/FAIL]

**DRR Constraints:**
| ID | Constraint | S2 Location | Status |
|----|------------|-------------|--------|
| C-F1 | `<text>` | spec.md:45 | [FOUND/MISSING] |

**Drift Summary:**
- Omissions: <count>
- Weakenings: <count>
- Evidence Gaps: <count>

### R4: Scope Drift
**Status:** [PASS/FAIL/ESCALATE]

**Calculation:**
- Total Modified Files: <n>
- Out-of-Scope Files: <n>
- Scope Drift: <pct>%

**DRR Scope Declaration:**
- IN-SCOPE: `<list>`
- OUT-OF-SCOPE: `<list>`

**File Categorization:**
| File | Category | Notes |
|------|----------|-------|
| `<path>` | [in-scope/out-of-scope/incidental] | |

**Threshold Check:**
- <= 10%: [PASS/FAIL]
- > 25%: [ESCALATE/N/A]

### R5: Verification Evidence
**Status:** [PASS/WARN]

**Evidence Files:**
| File | Exists | Status | Notes |
|------|--------|--------|-------|
| `verify.log` | [Y/N] | - | |
| `verification_result.json` | [Y/N] | `<status>` | |

**Test Results:**
| Type | Passed | Failed | Status |
|------|--------|--------|--------|
| Unit | <n> | <n> | [PASS/FAIL] |
| Integration | <n> | <n> | [PASS/FAIL] |

**Coverage:**
- Lines: <pct>% [PASS/FAIL]
- Branches: <pct>% [PASS/FAIL]

### R6: Execution Integrity
**Status:** [PASS/WARN]

**Execution Log:**
| File | Exists | Phases | Tool Errors |
|------|--------|--------|-------------|
| `execution.jsonl` | [Y/N] | `<count>` | `<count>` |

**Phase Sequence Check:**
| Skill | Expected | Actual | Status |
|-------|----------|--------|--------|
| S1-Quint | Q0→Q1→Q2→Q3→Q4→Q5 | `<sequence>` | [PASS/FAIL] |
| S2-OpenSpec | S0→S1→S2→S3→S5→S6→S7 | `<sequence>` | [PASS/FAIL] |

**Tool Failures:**
| Tool | Error | Timestamp |
|------|-------|-----------|
| `<tool_name>` | `<error>` | `<timestamp>` |

## Warnings

### <Rule_ID>: <Title>
- **Severity:** [HIGH | MEDIUM | LOW]
- **Description:** <what was found>
- **Evidence:** `<file>:<line> "<text>"`
- **Suggestion:** <how to improve next time>

## Notes

All warnings are advisory. Archive has proceeded.

2. verdict.json

{
  "audit_id": "<uuid>",
  "drr_id": "$ARGUMENTS",
  "timestamp": "<ISO_8601>",
  "status": "APPROVED | APPROVED_WITH_WARNINGS | ESCALATE | BLOCKED",
  "rules": {
    "R1": { "status": "PASS|WARN", "evidence": "..." },
    "R2": { "status": "PASS|WARN", "evidence": "..." },
    "R3": { "status": "PASS|WARN", "evidence": "..." },
    "R4": { "status": "PASS|WARN|ESCALATE", "drift_pct": 0.0, "evidence": "..." },
    "R5": { "status": "PASS|BLOCK", "evidence": "..." },
    "R6": { "status": "PASS|WARN", "phases_complete": true, "tool_errors": 0, "evidence": "..." },
    "R7": { "status": "PASS|WARN", "new_packages_checked": 0, "evidence": "..." },
    "R8": { "status": "PASS|WARN", "new_code_blocks_reviewed": 0, "evidence": "..." },
    "R9": { "status": "PASS|WARN", "security_constraints_verified": 0, "evidence": "..." },
    "R10": { "status": "PASS|WARN", "patterns_scanned": [], "evidence": "..." }
  },
  "warnings": [
    {
      "rule": "R<N>",
      "severity": "HIGH|MEDIUM|LOW",
      "description": "...",
      "suggestion": "..."
    }
  ],
  "summary": {
    "total_warnings": 0,
    "high_count": 0,
    "medium_count": 0,
    "low_count": 0,
    "recommendation": "PROCEED | PROCEED_NOTE_WARNINGS | HUMAN_REVIEW_REQUIRED | BLOCKED_RE_VERIFY"
  }
}

Finding Handling

Completion

S3 terminates after producing artifacts. It does not wait for user input.

Output:

• audit_verdict.md
• verdict.json

Verdict values:

• APPROVED → All rules pass. Proceed.
• APPROVED_WITH_WARNINGS → Advisory issues found, noted for improvement. Archive proceeded.
• ESCALATE → R4 scope drift >25%. Archive halted. Human review required before proceeding.
• BLOCKED → R5 verification evidence missing or failed. Archive halted. Re-verify and achieve PASS.