Experiment Harness

Architecture: Two-Layer Execution Model

Layer 1 (Deterministic): scripts/run_factorial.sh
  - Pure sky jobs launch calls in a loop
  - ANY researcher can run this without Claude Code
  - NO LLM involvement — reproducible, auditable

Layer 2 (Monitoring Harness): /experiment-harness → /factorial-monitor
  - Creates experiment XML (standardized template)
  - VALIDATES pre-launch gates (9 checks, 0 skips)
  - Creates report file BEFORE launching
  - Drives /factorial-monitor for monitoring + diagnosis
  - Updates report after EVERY state change
  - Captures compound learning (new tests, issues, observations)

CRITICAL SEPARATION: The .sh script is the PRODUCT. The harness is the DEVELOPER TOOL. Production runs use ONLY the .sh script.

Workflow (5 Phases)

Phase 1: GENERATE   → Create experiment XML from template     [protocols/generate.md]
Phase 2: VALIDATE   → Pre-launch gates (all must pass)        [protocols/validate.md]
Phase 3: EXECUTE    → Launch .sh + drive /factorial-monitor    [protocols/execute.md]
Phase 4: COMPOUND   → Tests, issues, observations, watchlist  [protocols/compound.md]
Phase 5: REFLECT    → Self-assess harness effectiveness       [protocols/reflect.md]

Phase 1: GENERATE

Create the experiment XML from templates/experiment-run.xml.

1. Reference prior pass: Read the most recent XML and its report. Extract failures, observations, and watchlist items.
2. Load context: /plan-context-load → navigator → domains → metalearning → registry
3. Fill template: All 8 required sections
4. Validate XML: yaml.safe_load() (NEVER regex) to verify structure
5. Security check (Bi et al. G1-G2): XML contains no hardcoded credentials, paths match actual config files, resource requests match declared limits

Cognitive engagement checkpoint (Shen et al. 2026): Before proceeding, articulate in the report: "WHY do I expect these specific watchlist items? WHAT might go wrong that prior passes haven't revealed?"

Output: docs/planning/v0-2_archive/original_docs/run-debug-factorial-experiment-{N}th-pass.xml

Phase 2: VALIDATE

Pre-launch gates that MUST ALL pass:

If ANY gate fails → DO NOT LAUNCH. Fix first.

YAML Contract Validation (Gate Detail)

The YAML contract gate (configs/cloud/yaml_contract.yaml) enforces:

1. GPU types: Only GPUs listed in the contract's allowed_accelerators per cloud provider. A100 was removed — 5.5x cost, never authorized.
2. Banned GPUs: T4, V100, K80, P100, P4 are globally banned.
3. Cloud providers: Only GCP and RunPod. No AWS, Lambda, UpCloud.
4. Schema drift: No unauthorized top-level or resource keys in SkyPilot YAML.
5. Factorial strict: train_factorial.yaml must have string-format accelerators (not dict/priority list), GCP only, spot only, Docker only.
6. Cross-file: SkyPilot YAML accelerators must match configs/cloud/*.yaml.
7. Cost estimate: Preflight prints estimated cost based on GPU type x job count.

The contract is the SINGLE SOURCE OF TRUTH. Claude Code must NEVER update it without explicit user instruction. See CLAUDE.md Rule 31.

Phase 3: EXECUTE

1. Report file MUST exist at <report-output> with headers + empty tables (H1)
2. Launch: bash scripts/run_factorial.sh <config.yaml>
3. Enter /factorial-monitor Phase 2 (MONITOR)
4. After EACH poll: Update report status matrix (H2)
5. After EACH terminal job: Update cost table, check watchlist items (H2)
6. During idle time (spot queuing, long training): Run deep exploration protocol → protocols/deep-exploration.md
7. After ALL terminal: Write observations section

Phase 4: COMPOUND

After all jobs are terminal, produce ALL of these (H3):

1. Updated report with timeline, cost, observations, test opportunities
2. New tests — at least 1 per cloud observation (not template-filling; each test must target a specific bug or edge case discovered during THIS pass)
3. GitHub issues for each unresolved problem
4. Updated watchlist for the NEXT pass (carry forward + new items)
5. Metalearning doc if any process failure occurred
6. Cumulative state update → append to outputs/harness-state.jsonl

Compounding gate: Session is NOT complete until all 6 artifacts exist.

Cognitive engagement checkpoint: Before writing tests, articulate: "WHAT did I observe that was NOT predictable from prior passes? WHY does this observation matter for production reliability?"

Phase 5: REFLECT (Memento-Skills Read-Execute-Reflect-Write)

Self-assess after every pass:

1. Compare metrics: This pass vs. prior pass (success rate, cost, observation quality)
2. Evaluate rules: Did H1-H5 prevent failures, or did they add friction without value?
3. Propose modifications: Write harness-reflection-{N}.md with suggested rule changes
4. Update harness-state.jsonl with pass metrics for trend analysis

This phase prevents the harness from calcifying around rules that no longer apply. Skills are living artifacts that self-improve from deployment (Zhou et al. 2026).

XML Template

See templates/experiment-run.xml — 8 required sections. Template is loaded on-demand (L3 progressive disclosure per Anthropic guide).

Non-Negotiable Rules

Inherits F1-F5 from /factorial-monitor, plus:

H7-H9 details:

• NEVER modify SkyPilot YAML accelerators, cloud, or resources without user instruction
• The golden contract (configs/cloud/yaml_contract.yaml) defines EXACTLY what is allowed
• If L4 is unavailable and the user wants A100 fallback, the user must SAY SO explicitly
• Dict-format accelerators ({L4: 1, A100: 1}) are BANNED in factorial YAML — they create non-deterministic GPU selection. Use string format (L4:1) for determinism.
• Cost guard: preflight estimates max cost and prints it. If A100 appears, cost 5.5x.
• See: .claude/metalearning/2026-03-24-unauthorized-a100-in-skypilot-yaml.md

Additionally inherits:

Anti-Patterns

Competency Questions (BDI validation)

After execution, these must all be answerable YES from artifacts:

1. Was the report file created BEFORE the first sky jobs launch?
2. Does every XML section contain non-placeholder content?
3. Did the cost table update after every terminal job?
4. Were at least N new tests written (N = number of cloud observations)?
5. Does the watchlist for the next pass carry forward ALL unresolved items?
6. Did the reflection phase identify at least 1 rule to keep and 1 to reconsider?

Cumulative State (cross-pass trend tracking)

Append to outputs/harness-state.jsonl after each pass:

{
  "pass": 6,
  "date": "2026-03-24",
  "branch": "test/run-debug-gcp-6th-pass",
  "jobs_total": 34,
  "jobs_succeeded": 34,
  "jobs_failed": 0,
  "cost_usd": 8.50,
  "observations": 8,
  "new_tests_written": 5,
  "issues_filed": 2,
  "watchlist_carried": 3,
  "watchlist_new": 2,
  "harness_version": "0.2.0",
  "normalized_gain": 0.15
}

This enables: "Is the harness getting better over time?" via duckdb-skills:query.

Measurement Protocol (SkillsBench normalized gain)

Track before/after per pass:

g = (pass_success_rate - prior_pass_success_rate) / (1 - prior_pass_success_rate)

A positive g means the harness + fixes improved outcomes. A negative g means the harness caused harm (bloated context, wrong rules, etc.) — trigger Phase 5 REFLECT with extra scrutiny.

Integration Points

References

• Bi et al. (2026). "Automating Skill Acquisition." arXiv:2603.11808 — S=(C,π,T,R) tuple, security gates
• Li et al. (2026). "SkillsBench." arXiv:2602.12670 — normalized gain, focused > comprehensive
• Zhou et al. (2026). "Memento-Skills." arXiv:2603.18743 — Read-Execute-Reflect-Write loop
• Shen & Tamkin (2026). "How AI Impacts Skill Formation." arXiv:2601.20245 — cognitive engagement
• Ye et al. (2026). "Meta Context Engineering via Agentic Skill Evolution." arXiv:2601.21557
• Anthropic (2026). "Guide to Building Skills for Claude." — progressive disclosure, patterns
• Compound Engineering Plugin — compounding principle