Meta-Analyst Skill

Quick Start

1. Receive question — free-text clinical question from the user (e.g., "Do SGLT2 inhibitors reduce mortality in HFrEF?")
2. Run Phase 1 — formalise PICO, build and approve search strategy, execute searches, screen abstracts, generate PRISMA flow
3. Run Phase 2 — extract data, assess RoB 2.0, assemble characteristics table
4. Run Phase 3 — compute effect sizes, pool (DL random-effects), assess heterogeneity, run sensitivity analyses, test publication bias, rate GRADE, assemble report
5. Export report — save as meta_analysis_[question_slug].md and meta_analysis_[question_slug].json
6. Reply with brief summary — pooled estimate + GRADE certainty + path to exported files

Read the phase reference documents before executing each phase. The agent reads them to resolve any ambiguity.

Pipeline Architecture

User question (free-text clinical question)
  │
  ▼
PHASE 1 — IDENTIFY
  Stage 1.1  PICO Formalization          [LLM: extract P, I, C, O with MeSH + synonyms]
  Stage 1.2  Search Strategy             [LLM: Cochrane MECIR + RCT filter → human ✓]
  Stage 1.3  Search Execution            [API: PubMed E-utilities, CENTRAL, CTG v2 → dedup]
  Stage 1.4  Abstract Screening          [LLM: include/exclude/uncertain → human ✓]
  Stage 1.5  PRISMA Flow                 [Python: prisma_flow_svg()]
  │
  ▼
PHASE 2 — APPRAISE
  Stage 2.1  Data Extraction             [LLM: 3× majority-vote extraction from full text]
  Stage 2.2  Risk of Bias Assessment     [Evidence Evaluator + fallback RoB 2.0 checklist]
  Stage 2.3  Characteristics Table       [Python: format_characteristics_table()]
                                         [→ human ✓ before synthesis]
  │
  ▼
PHASE 3 — SYNTHESIZE
  Stage 3.1  Effect Size Computation     [Python: effect_sizes.py]
  Stage 3.2  Meta-Analytic Pooling       [Python: pooling.py — DL default, FE comparison]
  Stage 3.3  Heterogeneity              [Python: heterogeneity.py — Q, I², tau², PI]
  Stage 3.4  Sensitivity Analyses        [Python: sensitivity.py — LOO, high-RoB, FvR]
  Stage 3.5  Publication Bias            [Python: publication_bias.py — Egger's (k≥10)]
  Stage 3.6  GRADE Certainty             [Python + LLM: 5 domains]
  Stage 3.7  Report Assembly             [Python: report.py — Markdown + JSON]
  │
  ▼
Output: meta_analysis_[slug].md + meta_analysis_[slug].json

Phase 1 — Identify

Reference document: references/phase_1_identify.md

Read this document before starting Phase 1. It defines the exact I/O contracts, API endpoints, deduplication rules, inclusion/exclusion criteria, and human checkpoint scripts.

Stage 1.1 — PICO Formalization

Extract Population, Intervention, Comparator, and Outcome from the user's question. Assign MeSH terms and synonyms to each element. Construct the pico_search_string (OR within concepts, AND across P/I/C).

Output: PICO JSON with pico_search_string.

Stage 1.2 — Search Strategy Construction

Build Boolean queries for PubMed, Cochrane CENTRAL, and ClinicalTrials.gov following Cochrane MECIR standards. Append the Cochrane Highly Sensitive Search Strategy RCT filter to the PubMed query.

Human Checkpoint #1: Present the proposed queries to the user. Do not execute searches until the user approves or edits.

Stage 1.3 — Search Execution

Execute the approved queries:

• PubMed: esearch.fcgi → efetch.fcgi (E-utilities)
• Cochrane CENTRAL: REST API or web interface
• ClinicalTrials.gov: https://clinicaltrials.gov/api/v2/studies (paginate via nextPageToken)

Deduplicate by PMID, DOI, then title fuzzy match (≥0.90 normalised similarity). Log per-database hit counts.

Stage 1.4 — Abstract Screening

Classify each deduplicated record as include / exclude / uncertain against the PICO inclusion criteria (RCT design, population match, intervention match, comparator match, outcome reported).

Human Checkpoint #2: Present the shortlist of included + uncertain papers. Do not proceed to Phase 2 until approved.

Stage 1.5 — PRISMA Flow

from pipeline.visualizations import prisma_flow_svg

svg = prisma_flow_svg({
    "db_pubmed": n, "db_central": n, "db_ctgov": n,
    "identified": n, "duplicates_removed": n, "screened": n,
    "excluded_screening": n, "eligible": n,
    "excluded_eligibility": n, "included": n,
})

Phase 2 — Appraise

Reference document: references/phase_2_appraise.md

Read this document before starting Phase 2.

Stage 2.1 — Data Extraction

Run 3× independent extraction passes per study. Apply majority vote on numeric fields. Flag low_confidence_fields when all three passes disagree. Extract the full standardised form (study ID, design, population, N per arm, events per arm for binary outcomes, mean/SD/N for continuous outcomes, follow-up duration, blinding, allocation concealment).

Stage 2.2 — Risk of Bias Assessment

Primary approach: Invoke the Evidence Evaluator skill for each included study.

npx skills add SciSpark-ai/evidence_evaluator

Extract per-domain RoB 2.0 judgments (D1: Randomisation, D2: Deviations, D3: Missing data, D4: Measurement, D5: Reporting) and the overall judgment.

Fallback: If Evidence Evaluator is unavailable, apply the streamlined 5-domain RoB 2.0 checklist defined in references/phase_2_appraise.md.

Generate the traffic light SVG:

from pipeline.visualizations import rob_traffic_light_svg

svg = rob_traffic_light_svg([
    {"study": "McMurray 2019", "domains": [
        {"domain": "D1: Randomisation", "judgment": "low"},
        {"domain": "D2: Deviations",    "judgment": "low"},
        {"domain": "D3: Missing data",  "judgment": "low"},
        {"domain": "D4: Measurement",   "judgment": "low"},
        {"domain": "D5: Reporting",     "judgment": "some concerns"},
    ]}
])

Stage 2.3 — Characteristics Table

from pipeline.report import format_characteristics_table

table_md = format_characteristics_table(table_studies)
# table_studies: list of dicts with keys:
# first_author, year, n_intervention, n_control,
# intervention_description, comparator_description,
# followup_duration, rob_overall

Human Checkpoint #3: Present the extracted data and RoB summary for review. Do not begin Phase 3 until approved.

Phase 3 — Synthesize

Reference document: references/phase_3_synthesize.md

Read this document before starting Phase 3. It contains complete code invocation examples for every stage.

Stage 3.1 — Effect Size Computation

from pipeline.effect_sizes import compute_log_or, compute_md, compute_smd, zero_cell_correction

Apply zero_cell_correction before any log-transform. Choose the effect size type based on outcome type (binary → log OR or log RR; continuous same scale → MD; continuous different scales → SMD/Hedges' g).

Stage 3.2 — Meta-Analytic Pooling

from pipeline.pooling import pool_random_effects_dl, pool_fixed_effect_iv, pool_mantel_haenszel

Default: DerSimonian-Laird random-effects. Run fixed-effect IV as sensitivity comparison. Use Mantel-Haenszel as a cross-check for sparse binary data. Back-transform log OR/RR to original scale before reporting.

Stage 3.3 — Heterogeneity

from pipeline.heterogeneity import cochrans_q, i_squared, tau_squared_dl, prediction_interval

Report Q (with p-value), I² (with Cochrane Handbook interpretation), tau², and the prediction interval (when k ≥ 3). I² interpretation thresholds: <40% low, 40–60% moderate, 60–75% substantial, ≥75% considerable.

Stage 3.4 — Sensitivity Analyses

from pipeline.sensitivity import leave_one_out, exclude_high_rob, fixed_vs_random_comparison

Run all three analyses for each pooled outcome. Flag any study whose leave-one-out exclusion changes direction or significance.

Stage 3.5 — Publication Bias

from pipeline.publication_bias import eggers_test, funnel_plot_data

k < 10 guard: If fewer than 10 studies, skip Egger's test entirely and note: "Formal test for funnel asymmetry not conducted (k < 10; Cochrane Handbook 10.4.3.1)."

If k ≥ 10 and Egger's p < 0.10, report suspected publication bias and apply GRADE downgrade of −1.

Stage 3.6 — GRADE Certainty

from pipeline.grade import (
    assess_risk_of_bias, assess_inconsistency, assess_imprecision,
    assess_publication_bias, compute_grade, grade_summary_row
)

Assess all 5 GRADE domains for each outcome:

Start at High certainty for RCTs. Apply all downgrades:

certainty = compute_grade("High", {
    "rob": rob_downgrade,
    "inconsistency": inconsistency_downgrade,
    "indirectness": indirectness_downgrade,   # agent-assigned
    "imprecision": imprecision_downgrade,
    "publication_bias": pub_bias_downgrade,
})

Stage 3.7 — Report Assembly

from pipeline.report import assemble_report, format_characteristics_table, format_grade_sof_table

report = assemble_report(
    pico=pico_json,
    search={"query": query, "date_range": date_range, "databases": [...]},
    prisma_counts=prisma_counts,
    characteristics_table=format_characteristics_table(table_studies),
    rob_summary="Narrative RoB summary.",
    outcomes=[...],          # list of outcome dicts with pooling, heterogeneity, grade
    sensitivity={...},       # dict keyed by outcome name
    publication_bias={...},  # dict keyed by outcome name
    prisma_svg=None,         # auto-generated if None
    rob_svg=rob_svg,
)

The function returns {"markdown": str, "json": dict}. Write both to disk:

import json
with open("meta_analysis_output.md", "w") as f:
    f.write(report["markdown"])
with open("meta_analysis_output.json", "w") as f:
    json.dump(report["json"], f, indent=2)

Human Checkpoints

Three checkpoints require explicit user approval before continuing:

Never skip a checkpoint. If the user is not available, pause and state which checkpoint is pending.

Output Format

Markdown Report Structure

The assembled report (report["markdown"]) contains these sections in order:

1. Title and metadata (question, date, databases)
2. PRISMA Flow Diagram (embedded SVG)
3. Search Strategy (query text, date range)
4. Characteristics of Included Studies (Markdown table)
5. Risk of Bias Summary (traffic light SVG + narrative)
6. Results (per-outcome subsections):
   • Forest plot SVG
   • Pooled estimate + 95% CI + p-value
   • Heterogeneity: I², Q, tau², prediction interval
7. Sensitivity Analyses (leave-one-out, fixed vs random, high-RoB exclusion)
8. Publication Bias (funnel plot SVG + Egger's result or skip note)
9. GRADE Summary of Findings (Markdown table with certainty symbols)
10. Narrative Summary (agent-generated)
11. Disclaimer

JSON Export Structure

{
  "pico":             { "population": "...", "intervention": "...", "comparator": "...", "outcome": "..." },
  "search":           { "query": "...", "date_range": "...", "databases": [...] },
  "prisma":           { "identified": N, "included": N, ... },
  "studies":          [ { ... } ],
  "outcomes": [
    {
      "name":          "...",
      "pooling":       { "pooled": ..., "ci_lower": ..., "ci_upper": ..., "p_value": ... },
      "heterogeneity": { "i2": ..., "q": ..., "tau_sq": ..., "prediction_lower": ..., "prediction_upper": ... },
      "sensitivity":   { ... },
      "grade":         { "certainty": "High|Moderate|Low|Very Low", "certainty_symbols": "⊕⊕⊕⊕", "downgrade_reasons": [...] }
    }
  ],
  "grade_summary":    [ { "outcome": "...", "certainty": "...", "symbols": "..." } ],
  "publication_bias": { ... }
}

Disclaimer (always include): The Markdown report footer contains: "This meta-analysis was generated by an AI agent skill. Results should be verified by qualified researchers before use in clinical decision-making."

Composition with Evidence Evaluator

The meta-analyst composes with the Evidence Evaluator skill for per-study RoB 2.0 assessment (Stage 2.2).

Install:

npx skills add SciSpark-ai/evidence_evaluator

Usage: For each included study, invoke Evidence Evaluator with the full text or abstract + methods. Request Stage 4 (bias risk) output specifically to extract the 5 RoB 2.0 domain judgments and the overall judgment.

Fallback behavior: If Evidence Evaluator is not installed, or returns no output for a study, apply the streamlined 5-domain RoB 2.0 checklist defined in references/phase_2_appraise.md. Document which studies were assessed via Evidence Evaluator vs the fallback checklist.

Reference Documents

Read the appropriate reference document before executing each phase or when resolving any ambiguity:

Key Design Principles

• Cochrane Handbook methodology throughout. DL random-effects is the default; fixed-effect is always run as a sensitivity comparison, not a primary result.
• All statistical computation is deterministic Python. LLM handles screening, extraction, GRADE indirectness, and narrative. Python handles all numbers.
• Three human checkpoints prevent irreversible errors. Searching, extraction, and synthesis each require user sign-off before proceeding.
• Prediction intervals are mandatory when k ≥ 3. A significant pooled estimate with a prediction interval crossing the null signals important real-world heterogeneity.
• k < 10 guard on Egger's test. Never report or act on Egger's results with fewer than 10 studies.
• GRADE indirectness requires documented reasoning. Agent must explain why the PICO elements of each study match or diverge from the research question.
• Report is the primary deliverable. The Markdown file is comprehensive and auditable. The chat response is a brief summary pointing to it.