The Ladder of Causation

Most RCA lives implicitly at rung 3 — every "the root cause was X" statement is a counterfactual — but most RCA tools can only support rung 1. The gap is where spurious root causes come from.

Structural Causal Models (SCMs)

An SCM is a triple (U, V, F) where:

• U — exogenous variables (unobserved background conditions, modeled as independent random variables)
• V — endogenous variables (observed system variables)
• F — a set of structural equations, one per endogenous variable, expressing how each variable is generated from its parents and from exogenous noise

Example for a simple latency incident:

U = {u_traffic, u_cpu_noise, u_lat_noise}
V = {Traffic, CPU, Latency}
F: Traffic := u_traffic
   CPU     := 0.7 * Traffic + u_cpu_noise
   Latency := 0.3 * CPU + 0.1 * Traffic + u_lat_noise

This SCM encodes three claims: Traffic is exogenous, CPU is caused by Traffic plus noise, and Latency is caused by both CPU (primary) and Traffic (direct effect, e.g., queueing). A directed acyclic graph (DAG) corresponds one-to-one:

Traffic → CPU → Latency
       ↓________↑

Why the DAG matters

The DAG encodes which variables can be confounders. Pearl's do-calculus operates on DAGs to answer: "Given the DAG structure and observational data, can I compute the effect of do(X=x) on Y without running an experiment?" The answer depends on whether a valid adjustment set exists — a set of variables that, when conditioned on, blocks all backdoor paths from X to Y.

Backdoor criterion

A set Z satisfies the backdoor criterion for (X, Y) if:

1. No node in Z is a descendant of X.
2. Z blocks every path between X and Y that contains an arrow into X.

If such a Z exists, then:

P(Y | do(X=x)) = Σ_z P(Y | X=x, Z=z) P(Z=z)

This is the adjustment formula. It says: to compute the causal effect of X on Y, stratify by Z and average.

Frontdoor criterion

When no backdoor adjustment set exists (unmeasured confounders), the frontdoor criterion may still rescue you: find a set M such that M is on every directed path X→Y, M is unconfounded with X, and no unblocked backdoor path from M to Y exists. Then:

P(Y | do(X=x)) = Σ_m P(m | x) Σ_{x'} P(Y | m, x') P(x')

Counterfactual computation

Given an SCM and observed evidence e = {X=x, Y=y}, compute "would Y have been y' had X been x'?":

1. Abduction — use evidence e to update the distribution over U.
2. Action — replace the structural equation for X with X := x' (mutilate the graph).
3. Prediction — compute P(Y=y') under the modified SCM and updated U.

This is the only known formal procedure for computing individual-level counterfactuals. Every "the root cause was X" claim you want to defend mathematically reduces to a counterfactual computation in a specified SCM.

Bayesian Networks for fault diagnosis

A Bayesian network is a DAG with a conditional probability table (CPT) at each node. When the CPTs encode causal rather than merely probabilistic dependencies, the network functions as an SCM for computing effects and counterfactuals.

Fault-diagnosis workflow

1. Define the variables: symptoms, components, environmental factors.
2. Draw the DAG from domain knowledge (or learn it — see structure learning below).
3. Estimate CPTs from historical incident data or expert elicitation.
4. At incident time, instantiate observed variables as evidence.
5. Compute posterior P(fault | evidence) using variable elimination, junction tree, or MCMC.
6. Rank candidate faults by posterior probability × remediation cost.

Empirical performance

Paper 2 of our mathematical-foundations research: a comparative evaluation of Bayesian network fault diagnosis across six industrial case studies found classification accuracy of 85–94% when network structure matches the physical process, dropping to 60–70% under structure misspecification. The lesson: the DAG is the hard part. A correct DAG with mediocre CPTs outperforms a wrong DAG with perfect CPTs.

Structure learning

When you don't know the DAG a priori:

• Constraint-based (PC, FCI algorithms by Spirtes, Glymour, Scheines): test conditional-independence claims in the data and rule out edges that contradict them.
• Score-based (GES, BIC-scored search): score candidate DAGs by fit + complexity penalty, hill-climb to a local optimum.
• Hybrid (MMHC): constraint-based skeleton then score-based orientation.

Structure learning from observational data identifies the DAG only up to Markov equivalence — it cannot distinguish X→Y from Y→X without intervention or temporal order. Use it as a hypothesis generator, not as ground truth.

Graph-theoretic fault localization

When you have a service dependency graph (or any causal graph) and want to localize a fault to a node, centrality measures help.

Centrality measures

Paper 3 finding (our research)

Graph centrality alone is insufficient for fault localization in microservices — it identifies structurally important nodes, but the structurally important node is often not where the fault is. The best-performing methods combine centrality with runtime metrics: weight edges by current request rate and latency, then rank by weighted centrality. This marries the static architectural view with the dynamic operational view.

Information-theoretic methods

When you have high-frequency time series but no reliable DAG, information-theoretic measures can detect statistical coupling without assuming a generative model.

Transfer entropy

TE(X → Y) = H(Y_{t+1} | Y_t^k) - H(Y_{t+1} | Y_t^k, X_t^l)

Transfer entropy measures how much the future of Y is predictable from the past of X beyond what's already predictable from Y's own past. It's directional, nonparametric, and detects nonlinear coupling.

Mutual information

I(X; Y) = Σ_{x,y} P(x,y) log [P(x,y) / (P(x)P(y))]

Symmetric measure of statistical dependence. Not causal by itself — X and Y can have high MI because both depend on a common driver — but useful as a screening tool.

Paper 4 finding

Transfer entropy substantially outperformed lagged correlation for fault localization in a 24-node industrial chemical process. The key advantage: transfer entropy detects nonlinear coupling that linear correlation misses entirely.

Granger causality

X Granger-causes Y if past values of X help predict Y beyond what past Y values alone provide. Formally, a VAR model:

Y_t = Σ_i a_i Y_{t-i} + Σ_j b_j X_{t-j} + ε_t

Reject the null that all b_j = 0 using an F-test. If rejected, X Granger-causes Y.

Caveats

• Granger causality is predictive, not interventional. It can be misled by a common cause with different lags to X and Y.
• It assumes linearity and stationarity unless you use nonlinear or time-varying variants.
• It works best when combined with physical domain knowledge to prune spurious hits.

Putting it all together — a causal-inference workflow for RCA

Input: an incident with rich telemetry (metrics, traces, logs, events)

Step 1. Scope the question as a causal query.
        "Did X cause Y?" or "What happens to Y under do(X=x)?"

Step 2. Construct (or learn) a DAG of the system.
        — Start from known topology (service map, code dependencies)
        — Augment with structure-learning over telemetry time series
        — Validate against domain experts

Step 3. Check identifiability.
        — Is there a valid backdoor adjustment set for (X, Y)?
        — Is there a valid frontdoor set?
        — If neither, an observational answer is impossible — need an experiment.

Step 4. Estimate the causal effect.
        — Fit CPTs / structural equations from historical data
        — Use doubly-robust estimators when unsure about model form
        — Report uncertainty bands, not point estimates

Step 5. Compute counterfactuals for each candidate root cause.
        — Abduct → Act → Predict
        — Rank candidates by the counterfactual effect on the outcome

Step 6. Validate with an intervention if possible.
        — Canary, chaos experiment, or A/B test
        — Compare observed effect to the SCM's prediction
        — If they disagree, the DAG is wrong

Tools and libraries

Common pitfalls

Conditioning on a collider

A collider is a node C with parents X and Y pointing into it (X → C ← Y). Conditioning on C opens a path between X and Y even though they're independent marginally. This is how Berkson's paradox and selection bias arise.

Practical example: You want to know if slow database queries cause user churn. You condition on "users who filed a support ticket" — but both slow queries and other frustrations cause support tickets. Conditioning on support-ticket-filers introduces spurious correlation.

Mistaking a mediator for a confounder

If M is on the causal path X → M → Y, conditioning on M blocks the effect of X on Y and makes it look like X has no effect. This is the opposite mistake from confounder-adjustment.

Data dredging for DAGs

Running structure-learning across many variables and picking the DAG with the best score will almost always produce an overfit structure. Pre-register your DAG from domain knowledge before looking at the data.

Temporal ordering violations

If you use cross-sectional data (one snapshot in time), you cannot distinguish X → Y from Y → X. Use panel data or explicit time stamps.

When causal inference is the wrong tool

Causal inference works best when:

• The system has many degrees of freedom and you want a quantitative estimate.
• You have rich observational data.
• You can formalize the question.
• The question is about populations or rates, not a single incident.

It's often the wrong tool for:

• Single-event failure investigation (use CAST or classical methods).
• Safety-critical certification (use FTA, STPA — regulators want interpretable structures).
• Fast-moving incidents (use tracing tools, escalate to causal inference in the postmortem).

Checklist before closing a causal-inference RCA

• The causal query is stated formally.
• The DAG is specified before looking at the data and annotated with domain justification.
• Identifiability is verified (backdoor / frontdoor / neither).
• Uncertainty is reported (not a point estimate).
• Sensitivity analysis: how robust is the conclusion to unmeasured confounders?
• A validation intervention is proposed (or the reason it is impossible is documented).
• The counterfactual rationale for each claimed root cause is explicit.

References

• Pearl, J. (2009). Causality: Models, Reasoning, and Inference (2nd ed.). Cambridge University Press.
• Pearl, J., & Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.
• Pearl, J., Glymour, M., & Jewell, N. P. (2016). Causal Inference in Statistics: A Primer. Wiley.
• Spirtes, P., Glymour, C., & Scheines, R. (2000). Causation, Prediction, and Search (2nd ed.). MIT Press.
• Granger, C. W. J. (1969). Investigating causal relations by econometric models and cross-spectral methods. Econometrica, 37(3), 424–438.
• Schreiber, T. (2000). Measuring information transfer. Physical Review Letters, 85(2), 461–464.
• Koller, D., & Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques. MIT Press.
• Sharma, A., & Kiciman, E. (2020). DoWhy: An end-to-end library for causal inference. arXiv:2011.04216.