Heterogeneity Analysis

Limitation: Underpowered with few studies, overpowered with many.

2. I² (I-squared)

What it is: Percentage of variability due to heterogeneity rather than chance.

Interpretation Guidelines (Cochrane):

Key Teaching Points:

• I² is a proportion, not an absolute measure
• Overlapping ranges are intentional—context matters
• Always consider clinical and methodological diversity

Socratic Questions:

• "If I² is 75%, what does that tell us about the studies?"
• "Can we still do a meta-analysis with high I²?"
• "What might cause studies to have different true effects?"

3. Tau² (Tau-squared)

What it is: Estimated variance of true effects across studies.

Interpretation:

• Tau² = 0 → No heterogeneity (all studies estimate same effect)
• Larger tau² → Greater spread of true effects
• Tau (square root) is on same scale as effect size

Advantage: Absolute measure, unlike I² which is relative.

4. Prediction Interval

What it is: Range where we expect the true effect of a NEW study to fall.

Why it matters:

• Wider than confidence interval
• Shows practical implications of heterogeneity
• Critical for clinical decision-making

Example:

Pooled effect: OR = 0.70, 95% CI [0.55, 0.89]
Prediction interval: [0.35, 1.40]

Interpretation: While the average effect favors treatment,
a new study might find effects ranging from strongly 
beneficial (0.35) to slightly harmful (1.40).

R Code for Heterogeneity Assessment

Basic Heterogeneity Statistics

library(metafor)

# Fit random-effects model
res <- rma(yi = yi, sei = sei, data = dat, method = "REML")

# View heterogeneity statistics
print(res)
# Look for: tau², I², H², Q, p-value

# Extract specific values
res$tau2   # tau-squared
res$I2     # I-squared (as proportion)
res$QE     # Q statistic
res$QEp    # p-value for Q test

Confidence Intervals for I²

# Get confidence interval for I²
confint(res)

# Output includes:
#        estimate   ci.lb   ci.ub
# tau^2    0.0234  0.0012  0.1456
# I^2(%)  62.4000 12.3000 89.2000

Prediction Interval

# Calculate prediction interval
predict(res)

# Or manually:
pi_lower <- res$beta - qt(0.975, res$k-2) * sqrt(res$tau2 + res$se^2)
pi_upper <- res$beta + qt(0.975, res$k-2) * sqrt(res$tau2 + res$se^2)

Visualizing Heterogeneity

# Forest plot with prediction interval
forest(res, 
       slab = dat$study,
       addpred = TRUE,  # Adds prediction interval
       header = TRUE)

# Baujat plot (identifies outliers)
baujat(res)

# GOSH plot (sensitivity to study inclusion)
gosh_res <- gosh(res)
plot(gosh_res)

Teaching Framework

Step 1: Report the Statistics

"Let's look at your heterogeneity results:

• Q = 24.5, p = 0.003 (significant)
• I² = 67% [42%, 82%]
• Tau² = 0.08"

Step 2: Interpret in Context

"This suggests substantial heterogeneity. About 67% of the variation we see is due to real differences between studies, not just chance."

Step 3: Discuss Implications

"With this level of heterogeneity, we should:

1. Still report the pooled effect, but with caution
2. Explore sources of heterogeneity
3. Consider subgroup or meta-regression analysis
4. Report the prediction interval"

Step 4: Investigate Sources

"Let's think about what might cause these differences:

• Different populations (age, severity)?
• Different interventions (dose, duration)?
• Different outcome measures?
• Different study designs?"

Decision Framework

I² Assessment
    │
    ├── I² < 40%
    │   └── Heterogeneity likely unimportant
    │       → Proceed with pooled estimate
    │
    ├── I² 40-75%
    │   └── Moderate heterogeneity
    │       → Report pooled estimate
    │       → Explore sources (subgroups)
    │       → Report prediction interval
    │
    └── I² > 75%
        └── Substantial heterogeneity
            → Question if pooling is meaningful
            → Mandatory exploration of sources
            → Consider narrative synthesis
            → Always report prediction interval

Common Misconceptions

1. "High I² means we can't do meta-analysis"

   • Reality: High I² means we need to investigate and interpret carefully
   • Pooling may still be appropriate with proper caveats

2. "Non-significant Q means no heterogeneity"

   • Reality: Q test has low power with few studies
   • Always report I² and tau² alongside Q

3. "I² tells us about clinical importance"

   • Reality: I² is statistical, not clinical
   • A small I² can hide clinically important variation

Assessment Questions

1. Basic: "What does I² = 50% mean?"

   • Correct: About half the observed variation is due to true differences between studies

2. Intermediate: "Q test is non-significant but I² = 45%. How do you interpret this?"

   • Correct: Q test may be underpowered; moderate heterogeneity may still exist

3. Advanced: "Pooled OR = 0.6 [0.4, 0.9] but prediction interval is [0.3, 1.2]. What's the clinical implication?"

   • Correct: While average effect is beneficial, a new setting might see no effect or even harm

Related Skills

• meta-analysis-fundamentals - Understanding pooled effects
• forest-plot-creation - Visualizing heterogeneity
• publication-bias-detection - Another source of concern

Adaptation Guidelines

Glass (the teaching agent) MUST adapt this content to the learner:

1. Language Detection: Detect the user's language from their messages and respond naturally in that language
2. Cultural Context: Adapt examples to local healthcare systems and research contexts when relevant
3. Technical Terms: Maintain standard English terms (e.g., "forest plot", "effect size", "I²") but explain them in the user's language
4. Level Adaptation: Adjust complexity based on user's demonstrated knowledge level
5. Socratic Method: Ask guiding questions in the detected language to promote deep understanding
6. Local Examples: When possible, reference studies or guidelines familiar to the user's region

Example Adaptations:

• 🇧🇷 Portuguese: Use Brazilian health system examples (SUS, ANVISA guidelines)
• 🇪🇸 Spanish: Reference PAHO/OPS guidelines for Latin America
• 🇨🇳 Chinese: Include examples from Chinese medical literature