Validate Behavior Change

The Scientific Method

1. Hypothesis: "If I add X to CLAUDE.md, Reeve will do Y under condition Z"
2. Test Design: Realistic scenarios that should/shouldn't trigger
3. Experiment: Isolated test pulses (fresh sessions)
4. Analysis: Did behavior occur? Consistently?
5. Iteration: If not, adjust and re-test

Key Insight: Behavior must be:

• Consistent - Works on repeated runs (stochasticity)
• Specific - Triggers when it should (positive cases)
• Bounded - Doesn't trigger when it shouldn't (negative cases)

Workflow

Phase 1: Clear the Runway

CRITICAL: Ensure no pulses are active before testing

list_upcoming_pulses(limit=10)

If pulses exist:

1. Wait for completion, OR
2. Cancel non-critical: cancel_pulse(pulse_id=X)
3. Document what was cancelled

Why: Active pulses may modify Desk, contaminating results.

Phase 2: Capture Baseline

git stash  # If uncommitted work
BASELINE=$(git log -1 --format="%H")
echo "$BASELINE" > /tmp/behavior-validation-baseline.txt
echo "Baseline: $BASELINE"

Phase 3: Define Expected Behavior

Articulate before writing tests:

**Change**: [What am I modifying?]
**Desired Behavior**: [What should happen?]
**Trigger Conditions**: [When should it activate?]
**Boundary Conditions**: [When should it NOT activate?]
**Observable Evidence**: [How do I verify it worked?]

Phase 4: Design Test Cases

Requirements:

• Minimum 2 positive cases (should trigger)
• Minimum 2 negative cases (should NOT trigger)
• Each case runs exactly 2 times

Hard Test Design:

CRITICAL: No Leakage Rule

Test prompts must NOT:

• Mention expected outcome
• Give hints about correct behavior
• Include "test", "validate", "experiment"
• Mention target files (Tasks/, Responsibilities/)

Only instruction: "RESPOND TO THIS MESSAGE NATURALLY."

Phase 5: Execute & Analyze (Fail-Fast)

5a. Apply change:

# Make changes, commit so test pulses see them
git add -A && git commit -m "VALIDATION: [description]"

5b. Schedule staggered pulses:

schedule_pulse(scheduled_at="now", prompt=test_1, tags=["validation", "pos_1", "run_1"])
schedule_pulse(scheduled_at="in 3 minutes", prompt=test_1, tags=["validation", "pos_1", "run_2"])
schedule_pulse(scheduled_at="in 6 minutes", prompt=test_2, tags=["validation", "neg_1", "run_1"])

5c. Analyze with subagents (protect context):

Use /session-analyzer patterns to check what the test pulse actually did:

• Tool usage (did it call Write on expected files?)
• Feedback signals (any corrections indicate confusion)
• Token consumption (unexpectedly high = potential issues)

Task(
    subagent_type="Explore",
    model="haiku",
    prompt="""Analyze validation test {test_id}, run {run}.
    Expected: {evidence}

    Check file changes:
    - Tasks/Open.md, Knowledge/Diary/, Responsibilities/

    Use session-analyzer to check:
    - Session JSONL for tool_use calls (Write, Edit)
    - Feedback signals in user messages

    Report: PASS or FAIL with evidence."""
)

5d. Fail-fast on failure:

1. Cancel remaining: cancel_pulse(pulse_id=X)
2. Diagnose from Knowledge/Diary/session logs
3. Fix the Desk change
4. Commit fix, restart from beginning

5e. Success criteria:

• Both runs PASS → Confirmed
• One PASS, one FAIL → Flaky (strengthen or accept)
• Both FAIL → Not working, fix required

Phase 6: Clean Reset

ALWAYS reset, regardless of outcome:

BASELINE=$(cat /tmp/behavior-validation-baseline.txt)
git reset --hard $BASELINE
git status
rm /tmp/behavior-validation-baseline.txt

If tests passed → Re-apply with proper commit message

Test Case Templates

Hard Positive Example

test_id: "pos_implied_action"