Grade Assessment

Starting Point

Factors That Lower Certainty (Downgrade)

1. Risk of Bias

What to assess:

• Randomization and allocation concealment
• Blinding of participants, personnel, outcome assessors
• Incomplete outcome data
• Selective reporting
• Other biases

When to downgrade:

• Serious limitations → Down 1 level
• Very serious limitations → Down 2 levels

Socratic Questions:

• "Were the studies properly randomized?"
• "Could the lack of blinding have affected results?"
• "Was there substantial loss to follow-up?"

2. Inconsistency (Heterogeneity)

What to assess:

• Point estimates vary widely
• Confidence intervals show minimal overlap
• I² is high
• Studies show different directions of effect

When to downgrade:

• Unexplained heterogeneity with I² > 50%
• Studies show conflicting results
• Prediction interval crosses null

Key Teaching Point: "Inconsistency is different from imprecision. Inconsistency means studies disagree; imprecision means we're uncertain about each estimate."

3. Indirectness

Types of indirectness:

When to downgrade:

• Important differences between evidence and question
• Surrogate outcomes used instead of patient-important outcomes

4. Imprecision

What to assess:

• Wide confidence intervals
• Small sample size / few events
• Optimal Information Size (OIS) not met

Rules of thumb:

• Binary: < 300 events total → consider downgrading
• Continuous: < 400 participants total → consider downgrading
• CI crosses thresholds of clinical importance

When to downgrade:

• CI includes both appreciable benefit and appreciable harm
• CI includes no effect and appreciable benefit (or harm)

5. Publication Bias

What to assess:

• Funnel plot asymmetry
• Egger's test significant
• Industry funding with positive results only
• Small study effects

When to downgrade:

• Strong suspicion of missing studies
• Trim-and-fill suggests meaningful impact

Factors That Raise Certainty (Upgrade)

Only for observational studies starting at Low

1. Large Effect

2. Dose-Response Gradient

• Clear relationship between dose/exposure and outcome
• Biological plausibility

3. Plausible Confounding

• All plausible confounders would reduce the effect
• Yet effect is still observed

GRADE Assessment Process

Step 1: Define the Question (PICO)
    │
Step 2: Identify Study Designs
    │
    ├── RCTs → Start at HIGH
    └── Observational → Start at LOW
    │
Step 3: Assess Downgrade Factors
    │
    ├── Risk of Bias?
    ├── Inconsistency?
    ├── Indirectness?
    ├── Imprecision?
    └── Publication Bias?
    │
Step 4: Assess Upgrade Factors (if observational)
    │
    ├── Large Effect?
    ├── Dose-Response?
    └── Confounding?
    │
Step 5: Determine Final Rating
    │
Step 6: Write Certainty Statement

Summary of Findings Table

Template

Creating Absolute Effects

# From relative risk
baseline_risk <- 0.20  # 20% in control group
RR <- 0.75
RR_lower <- 0.60
RR_upper <- 0.94

# Absolute risk reduction
ARR <- baseline_risk * (1 - RR)  # 5% = 50 per 1000
ARR_lower <- baseline_risk * (1 - RR_upper)
ARR_upper <- baseline_risk * (1 - RR_lower)

Certainty Statements

High certainty: "We are very confident that the true effect lies close to that of the estimate of the effect."

Moderate certainty: "We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different."

Low certainty: "Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect."

Very low certainty: "We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect."

Teaching Framework

Step 1: Establish the Question

"What exactly are we trying to answer? Let's define:

• Population
• Intervention
• Comparator
• Outcomes"

Step 2: Identify the Evidence

"What studies do we have?

• How many RCTs vs observational?
• What's our starting point?"

Step 3: Systematic Assessment

"Let's go through each GRADE domain:

1. First, risk of bias..."
2. Then, inconsistency..." [Continue through all domains]

Step 4: Make Judgments

"Based on our assessment:

• We downgraded for [reasons]
• Final certainty: [level]"

Step 5: Write the Statement

"Now let's write what this means for decision-makers..."

Common Mistakes to Avoid

1. Double-counting

   • Don't downgrade for both heterogeneity AND wide CIs if they're related

2. Automatic downgrading

   • Not every limitation requires downgrading
   • Consider impact on the effect estimate

3. Ignoring context

   • A "large" CI depends on clinical context
   • What difference matters to patients?

4. Forgetting outcomes

   • GRADE is assessed per outcome, not per review

Assessment Questions

1. Basic: "RCTs start at what GRADE certainty level?"

   • Correct: High

2. Intermediate: "I² = 70% with studies showing effects in opposite directions. Which domain is affected?"

   • Correct: Inconsistency

3. Advanced: "Studies are in adults but your question is about children. The intervention and outcomes are the same. What domain and how much to downgrade?"

   • Correct: Indirectness (population); typically down 1 level for serious indirectness

Related Skills

• meta-analysis-fundamentals - Understanding effect sizes
• heterogeneity-analysis - Assessing inconsistency
• publication-bias-detection - One of the GRADE domains

Adaptation Guidelines

Glass (the teaching agent) MUST adapt this content to the learner:

1. Language Detection: Detect the user's language from their messages and respond naturally in that language
2. Cultural Context: Adapt examples to local healthcare systems and research contexts when relevant
3. Technical Terms: Maintain standard English terms (e.g., "forest plot", "effect size", "I²") but explain them in the user's language
4. Level Adaptation: Adjust complexity based on user's demonstrated knowledge level
5. Socratic Method: Ask guiding questions in the detected language to promote deep understanding
6. Local Examples: When possible, reference studies or guidelines familiar to the user's region

Example Adaptations:

• 🇧🇷 Portuguese: Use Brazilian health system examples (SUS, ANVISA guidelines)
• 🇪🇸 Spanish: Reference PAHO/OPS guidelines for Latin America
• 🇨🇳 Chinese: Include examples from Chinese medical literature