Track the accuracy of arbiter rulings over time. After fixes are applied, record whether upheld findings actually needed fixes and whether dismissed findings were correctly dismissed.
Track the effectiveness of arbiter rulings over time. After fixes are applied, record:
This is automatically triggered after /adversarial-review - the system will prompt for tracking after fixes are applied.
This creates a feedback loop that improves future arbiter decisions.
Automatic prompting is enabled by default - just run the command and it will prompt for pending rulings.
# Track all pending rulings (prompts for each)
/arbiter-outcome-tracker --session-id "{uuid}"
# Quick track all with defaults
/arbiter-outcome-tracker --session-id "{uuid}" --track-all
# View accuracy stats
/arbiter-outcome-tracker --stats
# Single ruling
/arbiter-outcome-tracker --session-id "{uuid}" --finding-id "LG-001"
fixed | partially_fixed | not_fixed | wont_fixtrue | false# Query KB for the review session
kb_search(
query: "arbiter-review {session_id}",
artifact_type: "arbiter-review",
limit: 1
)
→ Returns session with all rulings
══════════════════════════════════════════════════════════
TRACK RULING: {finding_id}
══════════════════════════════════════════════════════════
Ruling: {ruling} | Severity: {severity}
Description: {short_description}
Outcome? [1] Fixed [2] Partial [3] Won't fix [4] Not real [s] Skip
══════════════════════════════════════════════════════════
Quick keyboard shortcuts:
1-4 - Select outcomes - Skip this onea - Mark all as fixedq - Quit trackingkb_get( artifact_type: "arbiter-review", story_id: "ARBITER-REVIEW-{session_id}" )
→ Returns session with all rulings
### Step 2: For Each Untracked Ruling
Present to user:
══════════════════════════════════════════════════════════ RULING OUTCOME: {finding_id} ══════════════════════════════════════════════════════════ Ruling: {ruling} | Severity: {severity} Source: {source_role} ({source_model}) Description: {finding_description}
Was this ruling correct? [1] Yes - fix worked [2] Partial [3] Won't fix [4] No issue ══════════════════════════════════════════════════════════
### Step 3: Record Outcome (with Model Attribution)
Record outcome with model context for optimization:
kb_add_lesson(