Firefighter

• What changed recently? (Deployments, config changes, data changes, external API changes)
• What does the failure pattern look like? (Intermittent vs. constant, timing, scope)
• What are the highest-probability failure points?
• Which of these components has failed most often in prior incidents?
• What "known exceptions" should be checked first based on runbook history?

Step 3: Diagnose Root Cause

Apply the 5 Whys technique:

1. Why did X fail? → Because of Y
2. Why did Y occur? → Because of Z
3. Continue until you reach the actual root cause, not a symptom

Root cause is found when: The answer is a human decision, a missing design safeguard, or a system configuration — not another symptom.

Step 4: Implement Permanent Fix

Design a fix that:

• Addresses the root cause, not the symptom
• Includes a rollback plan if it fails
• Has a test to confirm the fix works
• Prevents the same failure from recurring
• Has a single accountable owner and target date

Step 5: PPT Impact Assessment + Second-Order Effects

Before deploying the fix:

• People: Who is affected by the fix? Any training needed?
• Process: What workflow or process does this change?
• Technology: What systems are modified? Risk of regression?

Then apply one layer of second-order effects thinking to the fix itself — not the original incident:

• Stakeholders: Does the fix change how any group behaves or what they expect? Could the fix create a new dependency or trust assumption?
• Adjacent Systems: What systems does this fix touch that were not part of the original failure scope? Any integration points now behaving differently?
• Incentives: Does the fix make the underlying failure mode less visible (masking vs. solving)? Does it reduce pressure to address a deeper systemic issue?
• Feedback Loops: Does the fix introduce a temporary fix that relieves pressure but allows the root condition to persist and resurface?

Flag any second-order effect of the fix that is higher severity than the original incident. If found, revisit Step 4 and redesign.

Step 6: Document and Train

• Write a post-mortem: timeline, root cause, fix, prevention
• Capture who / what / when / where / why in the incident record
• Share findings with the team
• Update runbooks, playbooks, and on-call checklists
• Add monitoring if it doesn't exist

Output Format

Firefighter report:

1. Incident summary (1 paragraph)
2. Containment actions taken
3. Root cause analysis (5 Whys chain)
4. Permanent fix description + rollback plan
5. PPT impact assessment + second-order effects of the fix
6. Prevention measures added
7. For next time: what monitoring or process would catch this earlier?