Microcalibrate

pip install microcalibrate

What MicroCalibrate Does

Calibration problem: You have survey data with initial weights, and you know certain population totals (benchmarks). Calibration adjusts weights so weighted survey totals match benchmarks.

Example:

from microcalibrate import Calibration
import numpy as np
import pandas as pd

# Survey data (1,000 households)
weights = np.ones(1000)  # Initial weights

# Estimates (how much each household contributes to targets)
estimate_matrix = pd.DataFrame({
    'total_income': household_incomes,      # Each household's income
    'total_employed': household_employment  # 1 if employed, 0 if not
})

# Known population targets (benchmarks)
targets = np.array([
    50_000_000,  # Total income in population
    600,         # Total employed people
])

# Calibrate
cal = Calibration(
    weights=weights,
    targets=targets,
    estimate_matrix=estimate_matrix,
    l0_lambda=0.01  # Sparsity penalty
)

# Optimize weights
new_weights = cal.calibrate(max_iter=1000)

# Check results
achieved = (estimate_matrix.values.T @ new_weights)
print(f"Target: {targets}")
print(f"Achieved: {achieved}")
print(f"Non-zero weights: {(new_weights > 0).sum()} / {len(weights)}")

L0 Regularization for Sparsity

Why sparsity matters:

• Reduces dataset size (fewer households to simulate)
• Faster PolicyEngine calculations
• Easier to validate and understand

L0 penalty:

# L0 encourages many weights to be exactly zero
cal = Calibration(
    weights=weights,
    targets=targets,
    estimate_matrix=estimate_matrix,
    l0_lambda=0.01  # Higher = more sparse
)

To see impact:

# Without L0
cal_dense = Calibration(..., l0_lambda=0.0)
weights_dense = cal_dense.calibrate()

# With L0
cal_sparse = Calibration(..., l0_lambda=0.01)
weights_sparse = cal_sparse.calibrate()

print(f"Dense: {(weights_dense > 0).sum()} households")
print(f"Sparse: {(weights_sparse > 0).sum()} households")
# Sparse might use 60% fewer households while matching same targets

Automatic Hyperparameter Tuning

Find optimal l0_lambda:

from microcalibrate import tune_hyperparameters

# Find best l0_lambda using cross-validation
best_lambda, results = tune_hyperparameters(
    weights=weights,
    targets=targets,
    estimate_matrix=estimate_matrix,
    lambda_min=1e-4,
    lambda_max=1e-1,
    n_trials=50
)

print(f"Best lambda: {best_lambda}")

Robustness Evaluation

Test calibration stability:

from microcalibrate import evaluate_robustness

# Holdout validation
robustness = evaluate_robustness(
    weights=weights,
    targets=targets,
    estimate_matrix=estimate_matrix,
    l0_lambda=0.01,
    n_folds=5
)

print(f"Mean error: {robustness['mean_error']}")
print(f"Std error: {robustness['std_error']}")

Interactive Dashboard

Visualize calibration: