Reviewing Agent Output

Stage 1: Spec Compliance Review

Question: Did the agent do what was asked?

Checklist

• Task completed — all items from the original prompt are addressed
• Correct files — changes are in the expected files, no unexpected files modified
• Tests written — if the prompt asked for tests, they exist
• Tests passing — logs show test run with passes (not just "tests written")
• PR created — PR exists with descriptive title and body
• Branch correct — branched from the right base (usually main)
• Scope respected — no changes outside the specified scope

How to Check

From logs:

• Look for Bash calls with test commands → check output for PASS/FAIL
• Look for Edit/Write calls → verify correct files were modified
• Look for gh pr create → verify PR was created

From run result:

• Check pull_requests array for PR links
• Check summary for agent's own description of what it did

Spec Compliance Verdict

If PARTIAL: codegen_resume_run(run_id, prompt="You missed: [specific items]. Please complete them.")

If FAIL: Use debugging-failed-runs skill to diagnose, then prompt-crafting for a better prompt.

Stage 2: Code Quality Review

Question: Is the code good enough to merge?

Only proceed to Stage 2 if Stage 1 passed.

Quality Checklist

• No obvious bugs — logic errors, off-by-one, null handling
• Error handling — errors caught and handled appropriately, not swallowed
• No security issues — no hardcoded secrets, no SQL injection, no XSS
• Consistent style — matches project conventions (naming, structure, patterns)
• No unnecessary changes — no random reformatting, no unrelated refactors
• Tests are meaningful — not just "it exists", tests cover real behavior
• No debug artifacts — no console.log, no commented-out code, no TODOs

How to Check

From logs (quick check):

• Look for patterns suggesting issues:
  • console.log in Edit tool inputs
  • Hardcoded values that look like secrets
  • Very large diffs (agent may have over-modified)
  • Agent reverting its own changes (indecision)

From PR (thorough check):

• If PR URL is available, ask user to review the diff
• Or use gh pr diff <number> locally if repo is available
• Focus on: new code logic, test quality, error handling

Quality Verdict

If MINOR: codegen_resume_run(run_id, prompt="Quality issues to fix: [list specific issues]")

If MAJOR: Consider whether a new run or local fix is more efficient.

Review Report Format

Present review results to the user as:

Agent Output Review — Run #<id>
================================

Stage 1: Spec Compliance — [PASS/PARTIAL/FAIL]
  [✓] Task completed
  [✓] Correct files modified
  [✓] Tests written and passing
  [✓] PR created: <link>
  [✗] Missing: <what's missing, if any>

Stage 2: Code Quality — [APPROVE/MINOR/MAJOR/REJECT]
  [✓] No obvious bugs
  [✓] Error handling present
  [✓] Consistent style
  [!] Minor: <issue description>

Recommendation: [Merge / Fix then merge / Rework / Reject]

Review in Multi-Task Context

When reviewing as part of executing-via-codegen (between tasks):

1. Quick review — Stage 1 only for intermediate tasks (keep momentum)
2. Full review — Both stages for the final task or critical tasks
3. Batch review — After all tasks complete, full review of all PRs together

The goal is efficiency: don't spend 5 minutes reviewing every trivial task, but don't skip review for complex or risky tasks.

Task Risk Assessment

Integration with Code Review Skills

If superpowers requesting-code-review skill is available:

• Use it for DEEP review of critical agent PRs
• It dispatches a code-reviewer subagent for thorough analysis
• Best for: final PR, complex changes, security-sensitive code

For routine reviews, this skill's two-stage process is sufficient and faster.

Remember

• Two stages: spec compliance first, code quality second
• Logs are your primary source — read them before making judgments
• Compare against the original prompt — that's the spec
• Don't merge without review — even if the agent says "all tests pass"
• Resume for minor fixes — cheaper than creating a new run
• New run for major issues — stale context in a failed run hurts more than it helps
• Risk-based depth — full review for risky tasks, quick check for simple ones
• Report clearly — the user needs to make the merge decision