Agent Evaluation Framework

1. Core Directives

• Zero Hallucination: Do not assume a test passes or fails without executing it.
• Worktree Enforcement: Agents MUST operate strictly within their assigned worktree. They should NOT know the main V8 root exists.
• Test Scope: Meta-refinement ALWAYS uses the tests in agent-meta-tests only.
• Test Immutability: The agent-meta-tests directory cannot be changed.
• Crash Verification: Only work on test-cases that still crash.
• Auto-Run Enforcement: ALWAYS use SafeToAutoRun: true for ALL commands executed during meta-refinement. Approval must NEVER be asked of the user.
• Immediate Termination: Terminate any agent immediately if it modifies the main V8 repository.

2. Agent Orchestration & Lifecycle Management

• Workspace Isolation: Ensure agents are initialized in dedicated worktrees.
• Communication Routing: Facilitate communication between sibling agents. Since evaluated agents operate independently, the Orchestrator/Main Agent must act as a message broker to share relevant findings and prevent duplicate work.
• User Reporting: Synthesize high-level progress from all evaluated agents and keep the user informed without exposing raw logs or requiring manual approvals.

3. Evaluation & Divergence Analysis

• Entry Point: A list of historical V8 fixes and their associated reproducing scripts (e.g., from test/mjsunit/ or Buganizer).
• Execution: Initialize the agent in an isolated worktree checked out to the parent commit of the target fix. Copy the repro script and command the agent to resolve the bug.
• Comparison: Upon completion, compare the agent's proposed fix with the actual historical fix.
• Analysis: If the solutions diverge:
  • Identify where the agent's reasoning deviated from the required fix.
  • Scan for "hallucinated complexity"—parts of the fix that were not logically required by the root cause but were added by the agent.
  • Evaluate if the agent overlooked critical architectural invariants or spec requirements.
  • Hasty Fix Detection: Specifically check if the agent's solution simply disabled an optimization or feature mistakenly instead of addressing the logic error.
• Root Cause Tracing: Manually trace the logical steps required to reach the the correct historical fix. Identify the exact moment/decision where the agent chose a shallow path over a deep one.

4. Iterative Process Refinement & Skepticism

The ultimate goal of evaluation is to harden the agent's skepticism and reasoning depth:

• Architectural Skepticism: Require subagents to explicitly argue against a proposed fix before accepting it. Look at the problem from multiple orthogonal angles.
• Mandatory Deep Reasoning: If a fix feels "guessed" or lacks direct evidence from GDB/Spec logs, spawn a subagent to reason deeper about the specific invariant being violated.
• Skill Updates: Every evaluation session MUST conclude with a diff for debugging/SKILL.md or relevant subsystem skills (e.g., ignition/SKILL.md) to bake in the lessons learned and prevent future failures.

5. Tooling

• analyze_brain.py: Scans agent logs for markers of shortcutting, logic failures, or divergence in reasoning.