Calibration Trainer

Step 2: Mode Selection

Ask the user:

Would you like to practice interval estimation (90% confidence intervals), binary probability assessment, or both?

If no preference, default to both.

Briefly explain each mode:

• Interval: You'll estimate ranges for factual questions. The goal is for 90% of true answers to fall within your ranges.
• Binary: You'll assess the probability of true/false statements being correct.

Step 3: Question Count

Ask:

How many questions would you like? (default: 10, minimum: 5)

Before presenting questions, check the available pool in .measure/question_bank/. Count unanswered questions (where "asked": false) for the selected mode(s).

• If fewer unanswered questions remain than requested, inform the user:

  "Only N unanswered [type] questions remain. Would you like to proceed with N, or reset the question pool? Resetting marks all questions as unanswered — your calibration history is preserved."

• If zero unanswered questions remain for a type, offer to reset that pool.

When mode is "both," split questions roughly evenly between interval and binary.

Step 4: Explain the 90% Confidence Interval

This step is critical, especially for first-time users. Before presenting any interval questions, explain:

About 90% confidence intervals: For each question, give a LOW and HIGH value such that you are 90% confident the true answer falls between them. This means:

• Only about 1 in 10 answers should fall outside your range
• Most people naturally give ranges that are much too narrow (closer to 50% confidence)
• The equivalent bet test: Would you bet $1,000 at 9:1 odds that the answer is inside your range? If that bet feels risky, widen your range.

If the user has calibration history showing overconfidence (hit rate < 80%), add:

Your calibration history suggests your ranges tend to be too narrow. Try making your ranges about 30-50% wider than feels natural.

Step 5: Present Questions

IMPORTANT: Present exactly ONE question per message, then STOP and wait for the user's response before presenting the next question. Do not list multiple questions. Do not batch questions. One question, one message, wait for the answer.

Do NOT reveal the correct answer after each question — wait until all questions are answered to avoid anchoring effects.

For each question, send a single message in this format:

Interval:

Question N of M: What is [question]?

Give your 90% confidence interval:

• Low (5th percentile — 95% chance the true answer is above this):
• High (95th percentile — 95% chance the true answer is below this):

Binary:

Question N of M: True or false: [statement]

What is your probability (0-100%) that this statement is true?

Then stop and wait for the user to respond. After they answer, present the next question. Repeat until all questions are answered.

Step 6: Score

Construct a JSON payload with the user's responses:

{
  "interval_responses": [
    {"question": "...", "answer": <true_answer>, "user_low": <low>, "user_high": <high>}
  ],
  "binary_responses": [
    {"statement": "...", "answer": <true/false>, "user_probability": <0-100>}
  ],
  "history": [<previous sessions from .measure/calibration.json>]
}

Run the scoring script:

echo '<json_payload>' | uv run python .claude/skills/calibration-trainer/scripts/score_calibration.py

Parse the JSON output.

Step 7: Present Results

Interval results:

Interval Score: N out of M correct (X% hit rate). Target: 90%.

• If hit rate < 80%: "You're overconfident — your ranges are too narrow. [count] of your answers fell outside your ranges."
• If hit rate > 95%: "You're underconfident — your ranges are wider than necessary. You could narrow them and still capture 90%."
• If 80-95%: "Good calibration! Your ranges are well-sized."

Show average CI width and surprise index (how far misses were from the range).

Binary results:

Brier Score: X (0 = perfect, 0.25 = random guessing, lower is better)

• Below 0.10: "Excellent probability assessment"
• 0.10-0.15: "Good, better than most untrained assessors"
• 0.15-0.25: "Fair, room for improvement"
• Above 0.25: "Below random guessing — your confidence levels are systematically off"

Per-question reveal: Show each question with the correct answer, the user's response, and whether it was a hit or miss.

Trend (if calibration history exists):

Previous session: X% hit rate. This session: Y%. Over N sessions: [improving / stable / declining].

If this is the first session:

This is your first calibration session. We're establishing your baseline.

Step 8: Update Storage

1. In .measure/question_bank/, mark each presented question as "asked": true.
2. Append the new session to .measure/calibration.json history array:

   {
     "session_id": "<from scoring output>",
     "timestamp": "<from scoring output>",
     "type": "<interval|binary|both>",
     "questions": [<per-question details from scoring output>],
     "session_hit_rate": <hit_rate or null>,
     "brier_score": <brier_score or null>
   }
   

3. Update .measure/calibration.json summary:
   • Increment total_questions_answered
   • Update interval/binary subtotals, assessment, and trend
   • Set last_session to the new session timestamp

Step 9: Guidance

Based on results, provide targeted advice:

If overconfident (hit rate < 80%):

To improve: Before finalizing each range, apply the equivalent bet test — would you bet $1,000 at 9:1 odds that the answer is inside your range? If not, widen it.

Also try thinking about reasons you might be wrong: "What if my assumptions are completely off? What would the answer be then?"

If underconfident (hit rate > 95%):

To improve: You know more than you think. Try starting with your best guess, adding a range that feels "just barely wide enough," then leaving it there instead of widening further.

If well-calibrated:

Your calibration is good. This means your uncertainty estimates on real measurement projects will be more reliable, and value-of-information calculations will be more accurate.

For deeper methodology, read references/scoring-methodology.md.

Next Steps

• If your calibration needs improvement, run another session with calibration-trainer.
• If you're ready to start a measurement project, use the measurement-definition skill.
• To run the full guided workflow, use the measure skill.