HyperOrch Experiment Preset

When This Applies

Trigger patterns: "experiment", "hypothesis", "debug scientifically", "root cause", "investigate", "why does", "measure", "bisect", "A/B test", "reproduce"

Experiment Protocol

Phase Structure

OBSERVE → HYPOTHESIZE → PREDICT → DESIGN → INSTRUMENT → EXECUTE → ANALYZE → RECORD
   ↓          ↓            ↓         ↓          ↓           ↓          ↓         ↓
 Symptom   Falsifiable   Metric   Classify   Add Probes   Run It    Verdict   Update
  Data      Claims      Targets   Obs/Int    & Logging              & Learn   Registry

Orchestration Flow

DESIGN-REVIEW (HyperSan, once) → EXECUTE (HyperArch, autonomous) → ANALYZE (HyperSan, post-hoc)

• DESIGN-REVIEW: HyperSan validates the experiment design — reasoning about whether predictions are meaningful, causal isolation is achievable, and guardrails are proportionate to risk.
• EXECUTE: HyperArch receives full experiment context and runs all arms end-to-end. Collects all probe data, compiles measurement table, returns results.
• ANALYZE: HyperSan compares predictions to measurements, proposes verdict. HyperOrch delegates recording/cleanup to HyperArch if needed.

Experiment authoring (Steps 1–5) happens BEFORE orchestration begins — the experiment entry must exist with Steps 1–5 filled before HyperOrch starts.

The 8-Step Loop

Step 1: OBSERVE

Describe exact symptom with measurable data. No interpretation, no guessing.

• What happened? When? How often?
• Attach metrics, logs, error messages — raw data only.

Step 2: HYPOTHESIZE

List ALL plausible explanations as falsifiable claims.

• Rate each: [LIKELY], [POSSIBLE], [UNLIKELY] — prior belief, never modified retroactively.
• Sort by testability (easiest to test first), not likelihood.

Step 3: PREDICT ⛔ ABSOLUTE GATE

State specific predictions BEFORE execution. Use the 5-column prediction table:

Why this is absolute: Post-hoc rationalization is cognitively invisible — humans and agents naturally construct explanations that fit observed data. Without pre-registered predictions, you cannot distinguish genuine understanding from pattern-matching on noise. This gate has no reasoning exceptions because the failure mode is undetectable from inside.

Step 4: DESIGN

Classify experiment type and create minimal design: → See experiment_types.md for classification rules

Set minimum_confirmation_runs:

Step 5: INSTRUMENT

Add measurement or intervention code, tagged for clean removal.

• Observational: add # PROBE tagged instrumentation → See probing_guideline.md for principles, tag convention, and cleanup
• Interventional: add # INTERVENTION tagged code changes or use branch/config strategy → See intervention_procedure.md for tiered strategy and cleanup gates

Step 6: EXECUTE

Run the experiment. Collect data. Log everything.

• Record execution environment, timestamps, any anomalies.

Step 7: ANALYZE

Compare predictions to measurements. Force a verdict into one of four categories:

CONFIRMED | REJECTED | PARTIALLY CONFIRMED | INCONCLUSIVE

Why only four: Soft verdicts like "LIKELY CONFIRMED" let agents avoid committing to a conclusion. If the evidence clearly supports the hypothesis, say CONFIRMED. If not, say what it actually is. Suggestive-but-not-conclusive evidence is INCONCLUSIVE with a recommendation narrative — that's honest, and it drives the next experiment.

PARTIALLY CONFIRMED: Some predictions match, others don't, but the evidence itself is clear (not noisy — the predictions were partially right).

Step 8: RECORD

Update all records and clean up experiment artifacts:

• Experiment entry with verdict and analysis
• Registry/index with summary
• Handoff notes if context boundary approaching
• Cleanup gate (verdict-driven):
  • CONFIRMED → promote instrumentation/intervention to permanent code
  • REJECTED → revert all # PROBE and # INTERVENTION changes
  • INCONCLUSIVE → carry forward with justification in registry
• Reference exact probe output file path in experiment entry

Scientific Guardrails

These are the principles that make experiments trustworthy. Each carries its rationale so you can reason about intent.

Predictions Before Execution ⛔ ABSOLUTE

Predictions MUST be written before Step 6.

Why absolute: Post-hoc rationalization is cognitively invisible. This is the one rule where "the purpose is obviously served" reasoning fails, because you cannot detect the bias from inside. Always enforced.

Causal Isolation Principle

Purpose: When an experiment changes something and observes an effect, you need to be able to attribute the effect to a specific cause.

The test: "If a metric moves, can I determine which change caused it?"

Typical application: Each interventional arm changes one variable from baseline — this is the simplest way to ensure attribution.

Reasoning beyond the simple case: The principle is about attributability, not about counting variables. Designs that test multiple variables CAN satisfy causal isolation if the design allows decomposition:

• Factorial designs: If arms A (X only), B (Y only), and C (X+Y) all exist, arm C's interaction effect is isolable by comparing C against A and B individually. The individual arms provide the causal baseline.
• Confounded designs: If ONLY arm C (X+Y) exists with no individual arms, a metric change cannot be attributed. This violates the principle regardless of how many variables were changed.

When reviewing: Ask "can the experimenter determine what caused each observed effect?" — not "did each arm change exactly one variable?"

Confirmation Sufficiency

Purpose: A single stochastic run can be misleading due to random variation.

The test: "Is the observed result distinguishable from noise?"

Deterministic experiments (same seed, same hardware = same result) need one run. Stochastic experiments need enough runs to establish a pattern. The experimenter should justify their run count relative to expected variance.

Registry Awareness

Purpose: Avoid re-testing hypotheses that prior experiments already resolved.

The test: "Has this hypothesis (or a substantially similar one) already been tested?"

Read the experiment registry before designing new experiments. Previously REJECTED hypotheses need new evidence or a meaningfully different angle to justify re-testing.

Orchestration Steps

1. Initialize Experiment

• Parse symptom/question from user request
• Check for existing experiment registry — read before creating new hypotheses
• State: "Starting experiment workflow for: [observation]"

1b. Resume Gate

If a completed experiment entry is provided (Steps 1-5 already filled):

• Verify Steps 1-5 are present and complete
• If DESIGN-REVIEW was already done (e.g., experiment previously approved), skip directly to Phase 2 (EXECUTE)
• Otherwise proceed to Phase 1 (DESIGN-REVIEW)
• State: "Resuming experiment [id] from Phase [N]"

2. Phase 1: DESIGN-REVIEW

Invoke HyperSan to validate experiment design. HyperSan should reason about:

• Predictions quality: Are they specific enough to be falsifiable? Do they have clear metrics and tolerances?
• Causal isolation: Can observed effects be attributed? (Apply the reasoning from Scientific Guardrails, not a mechanical variable count.)
• Confirmation sufficiency: Is the run count justified for the expected variance?
• Guardrail proportionality: Are execution guardrails proportionate to known risks? (An experiment with known explosion risk should have early-stop clauses. A stable diagnostic run may not need them.)

If substantial issues: Return to experiment author with specific reasoning. Max 2 revision cycles. If sound: Proceed to Phase 2 autonomously.

3. Phase 2: EXECUTE

HyperOrch MUST invoke runSubagent(HyperArch) for this phase. If runSubagent is unavailable or fails, report the delegation failure to the user — do not improvise alternative execution.

Invoke HyperArch to run the experiment: