Causal Inference Advisor

Step 1: Understand the Causal Question

Ask the user:

1. What is the treatment/exposure (X)?
2. What is the outcome (Y)?
3. What is the unit of analysis (individual, firm, country, time period)?
4. Was the treatment randomly assigned? (Were users randomly split into groups, or did you just observe what happened naturally?)
5. What data do you have? (cross-sectional = snapshot of many units at one time / panel = same units tracked over time / time-series = one unit measured repeatedly)

Step 2: Method Selection Decision Tree

Based on the answers above, route to the correct method:

Was treatment randomly assigned?
├── YES → Was there non-compliance or contamination?
│   │         (Some users didn't actually receive what they were assigned)
│   ├── YES → Intention-to-Treat + IV/LATE for complier effect
│   │         (Read experiment-designer/references/rct-analysis.md)
│   └── NO → Is there interference between units?
│       │     (Can one user's treatment affect another user's outcome?)
│       ├── YES → Cluster/switchback design needed
│       │         (Read references/interference-networks.md)
│       └── NO → RCT Analysis
│            ├── Simple: Compare group averages directly
│            ├── Better: Adjust for pre-experiment covariates (Lin estimator)
│            │   (Read experiment-designer/references/rct-analysis.md)
│            ├── Small sample (<200/arm): Permutation test
│            │   (Read experiment-designer/references/small-sample-inference.md)
│            └── For subgroup effects → Read references/hte-estimation.md
│
└── NO → Is there a natural experiment or policy change?
    ├── YES → What kind?
    │   ├── Abrupt cutoff → Regression Discontinuity (RDD)
    │   │   (e.g., students just above/below a score threshold get different treatment)
    │   │   (Read references/rdd-guide.md)
    │   ├── Policy change at known time → Difference-in-Differences (DiD)
    │   │   (compare the affected group to a similar unaffected group, before and after)
    │   │   ├── Few treated units → Synthetic Control / Synthetic DiD
    │   │   │                        (Read references/synthetic-control.md)
    │   │   └── Many treated units → Standard DiD / Staggered DiD
    │   │                             (Read references/did-guide.md)
    │   └── Random encouragement → Instrumental Variables
    │       (something that nudges people toward treatment without directly affecting outcome)
    │       (e.g., a mailer encouraging sign-up affects enrollment but not outcomes directly)
    │       (Read references/iv-late.md)
    │
    └── NO → Purely observational data
        ├── Can you draw a causal diagram (DAG) showing what causes what?
        │   ├── YES → Adjust for confounders (regression, matching, IPW, AIPW)
        │   │          (Read references/matching-weighting.md)
        │   └── NO → Help user construct a DAG (see Step 3)
        │
        └── Is there likely an unmeasured factor affecting both treatment and outcome?
            └── YES → Sensitivity analysis REQUIRED
                       (Read references/sensitivity-analysis.md)

NEVER skip the identification step. If the user cannot explain why their estimate is causal (not merely a correlation), SAY SO EXPLICITLY:

"⚠️ Without an identification strategy (a clear argument for why this is cause-and-effect, not just correlation), this analysis estimates an association, not a causal effect. Proceed with correlational language only."

Step 3: DAG Construction (if needed)

Guide the user through building a Directed Acyclic Graph:

1. List all variables the user has or knows about
2. Identify the treatment (X) and outcome (Y)
3. Ask about confounders: "What variables affect BOTH the treatment and the outcome?" (Example: income might affect both whether someone uses a new budgeting app AND their savings rate. If you don't adjust for income, the app looks more effective than it is.)
4. Ask about mediators: "Does the treatment affect the outcome through any intermediate step?" (Example: a training program → increases skills → increases wages. Skills is the mediator. Usually do NOT adjust for mediators unless you specifically want to decompose the pathway.)
5. Ask about colliders: "Are there variables caused by BOTH the treatment and the outcome?" (Example: being hospitalized might be caused by both the treatment and bad health. Conditioning on a collider creates a spurious association. Do NOT adjust for these.)
6. Ask about instruments: "Is there a variable that affects the treatment but has no direct path to the outcome?" (Example: distance to a college affects whether someone attends college but doesn't directly affect their earnings except through college.)

Use the DAG to determine the adjustment set (the variables you need to control for to isolate the causal effect).

Step 4: Check Assumptions

For every method, the user MUST verify assumptions before estimation:

When assumptions FAIL:

• Parallel trends fails → try synthetic control, or use matching with pre-treatment outcomes as covariates
• Weak instrument (F < 10) → do NOT use IV; look for a stronger instrument or use a different approach
• Poor overlap in matching → trim the sample or use AIPW (doubly robust) instead of matching
• McCrary density test fails in RDD → manipulation likely; the RDD is not valid

If assumptions are violated, WARN and suggest alternatives or sensitivity analysis.

Step 5: Estimation

Guide the user to the appropriate estimation approach:

For Average Treatment Effect (ATE):

• Regression with controls (if linear, few confounders)
• Matching (find similar people in both groups and compare them)
• Inverse Probability Weighting / IPW (reweight the data so it looks like a randomized experiment)
• Augmented IPW / Doubly Robust / AIPW (combines regression AND weighting for extra protection if one is wrong) — PREFERRED for robustness

For Heterogeneous Treatment Effects (HTE, CATE): Read references/hte-estimation.md for:

• Meta-learners: S-learner, T-learner, X-learner
• DR-learner (doubly robust)
• Causal Forest (generalized random forest)
• When to use each approach

For Quantile Treatment Effects (QTE): When you care about the effect on the distribution, not just the mean (e.g., does the treatment help the bottom 10%? does it reduce variability?):

• Use quantile regression with treatment indicator
• QTE is identified in RCTs under treatment rank invariance
• In observational studies, combine with propensity score methods

Step 6: Refutation Tests (MANDATORY)

After estimation, run ALL applicable refutation tests. Read references/sensitivity-analysis.md and use scripts/refutation_tests.py:

1. Placebo treatment: Assign a random fake treatment → effect should be ~0
2. Placebo outcome: Use an unrelated outcome → effect should be ~0
3. Subset validation: Estimate on random subsets → effect should be stable
4. Add random common cause: Add a random confounder → effect should not change
5. Bootstrap refutation: Resample and re-estimate → check stability
6. Sensitivity to unmeasured confounding:
   • For matching/weighting: Rosenbaum bounds
   • For general settings: E-value

If ANY refutation test fails, WARN:

"🔴 Refutation test failed: [test name]. The causal estimate may not be reliable. Investigate before reporting."

Step 7: Reporting

Generate results with:

1. Point estimate with 95% confidence interval
2. Effect size in interpretable units (not just coefficient)
3. Identification strategy clearly stated
4. Assumptions listed with verification evidence
5. Refutation test results summarized
6. Limitations explicitly stated

Common Mistakes to PREVENT

• NEVER claim causation without an identification strategy (a clear argument for why it's causal)
• NEVER condition on a collider or post-treatment variable
• NEVER ignore unmeasured confounding in observational studies
• NEVER report DiD without checking parallel trends
• NEVER use IV with a weak instrument (first-stage F < 10)
• NEVER skip refutation tests — they are not optional
• NEVER interpret ATT (effect on the treated) as ATE (effect on everyone) without justification