Evaluate Model Calibration

Steps

Step 1: Understand the Concepts

A model can have perfect discrimination (AUROC=1.0) but terrible calibration (all predictions are 0.99 or 0.01). And vice versa.

Step 2: Create a Reliability Diagram

Using provided predictions and outcomes (or your own dataset):

import numpy as np
from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

# Sample: model predictions and true outcomes
y_true = [...]   # 0 or 1
y_pred = [...]   # predicted probabilities (0.0 to 1.0)

# Create calibration curve
prob_true, prob_pred = calibration_curve(y_true, y_pred, n_bins=10)

# Plot
plt.plot(prob_pred, prob_true, marker='o', label='Model')
plt.plot([0, 1], [0, 1], linestyle='--', label='Perfect calibration')
plt.xlabel('Mean predicted probability')
plt.ylabel('Observed frequency')
plt.title('Reliability Diagram')
plt.legend()
plt.savefig('calibration_plot.png')

Step 3: Calculate ECE and Brier Score

from sklearn.metrics import brier_score_loss

# Brier Score (lower is better, 0 is perfect)
brier = brier_score_loss(y_true, y_pred)

# Expected Calibration Error (lower is better)
def expected_calibration_error(y_true, y_pred, n_bins=10):
    bins = np.linspace(0, 1, n_bins + 1)
    ece = 0
    for i in range(n_bins):
        mask = (y_pred >= bins[i]) & (y_pred < bins[i+1])
        if mask.sum() > 0:
            bin_acc = y_true[mask].mean()
            bin_conf = y_pred[mask].mean()
            ece += mask.sum() / len(y_true) * abs(bin_acc - bin_conf)
    return ece

ece = expected_calibration_error(np.array(y_true), np.array(y_pred))

Step 4: Interpret Clinically

For your model, answer:

• Where is the model over-confident? (predicts high but observed frequency is lower)
• Where is the model under-confident? (predicts low but observed frequency is higher)
• At what probability thresholds does this matter clinically? (e.g., the 15% threshold for CT scanning in TBI)
• What's the clinical harm of miscalibration at this threshold? (missed scans vs unnecessary scans)

Step 5: Recalibration Options

If calibration is poor, common fixes:

1. Platt Scaling — Fit a logistic regression on model outputs
2. Isotonic Regression — Non-parametric calibration mapping
3. Temperature Scaling — Single parameter adjustment (common for neural networks)

Recalibrate on a held-out calibration set, never on the test set.

Artifacts

1. Reliability Diagram — Calibration plot for one health AI model (from data or literature)
2. ECE and Brier Score Calculation — Code + results with interpretation
3. Clinical Interpretation — 1-page memo explaining what the calibration means for clinical decision-making at specific thresholds
4. Recalibration Recommendation — If needed, which method and why

Assessment Criteria

Common Mistakes

• Using too few bins (< 5) or too many bins (> 20) for the reliability diagram
• Confusing calibration with discrimination — a well-calibrated model can still have mediocre AUROC
• Recalibrating on the test set (data leakage — always use a separate calibration set)
• Ignoring calibration because "the AUROC is high" — a common and dangerous practice in health AI

Related Skills

• run-tripod-ai-checklist — Item 13a specifically addresses calibration reporting
• decision-curve-analysis — Clinical utility assessment that accounts for calibration
• fairness-audit — Calibration may differ across demographic subgroups
• model-card-generator — Include calibration metrics in model documentation

References

• Niculescu-Mizil & Caruana (2005). Predicting Good Probabilities: https://doi.org/10.1145/1102351.1102430
• Van Calster et al. (2019). Calibration: the Achilles heel of predictive analytics: https://doi.org/10.1186/s12916-019-1466-7
• sklearn Calibration Guide: https://scikit-learn.org/stable/modules/calibration.html