Post Mortem

Post-Mortem Template

1. Summary (2-3 sentences)

What happened, when, impact.

2. Timeline

3. Impact

• Duration: How long?
• Users affected: How many?
• Revenue impact: If applicable
• Data loss: Any?
• Reputation: Customer communications?

4. Root Cause(s)

Not "human error"—go deeper:

• Why was the error possible?
• What safeguards didn't catch it?
• What systemic conditions contributed?

5. What Went Well

• Detection time
• Response coordination
• Communication
• Recovery speed

6. What Could Be Improved

• Missing alerts
• Documentation gaps
• Process friction
• Tool limitations

7. Action Items

Rule: Every action item has an owner and date. No orphan items.

Facilitation Guide

Before the Meeting

1. Gather timeline from logs, chat, alerts
2. Identify all participants (responders, stakeholders)
3. Set expectation: learning, not blame
4. Share draft timeline for review

During the Meeting

Opening (5 min)

"We're here to understand what happened and improve. This is blameless—we assume everyone acted reasonably with the info they had. Focus on systems and processes, not individuals."

Timeline Walk-through (20 min)

• Go chronologically
• Ask: "What did you know at this point?"
• Ask: "What options did you see?"
• Fill in gaps in understanding

Root Cause Discussion (15 min)

• Use 5 Whys technique
• Look for systemic issues
• Avoid stopping at "human error"

Action Items (15 min)

• Prioritize by impact and effort
• Assign owners IN the meeting
• Set realistic due dates
• Limit to 3-5 meaningful items (not 20 small ones)

Closing (5 min)

"Thank you for the candid discussion. We'll share the write-up for review. Any final thoughts?"

After the Meeting

1. Write up within 24 hours
2. Circulate for factual corrections
3. Publish to team/org
4. Track action items to completion
5. Review effectiveness in 30 days

Anti-Patterns to Avoid

Questions That Unlock Learning

• "What information would have helped at that moment?"
• "What made sense to do at the time?"
• "Where did our mental model differ from reality?"
• "What surprised you?"
• "What was harder than expected?"
• "If this happens again, what would we do differently?"
• "What's the smallest change that would have prevented this?"

Severity Classification

Connecting to Prevention

Post-mortems feed into:

• Runbooks: Better playbooks for next time
• Monitoring: New alerts for early detection
• Testing: New test cases for CI/CD
• Architecture: Design changes to prevent recurrence
• Training: Skills gaps identified

The Phoenix Post-Mortem

A personal example from Alex's own evolution...

What we learned from the Phoenix incident:

• Two sources of truth caused confusion
• Testing in production (Master Alex) was risky
• Kill switches need multiple layers
• Document decisions as you make them

These lessons became: ADR-006, RISKS.md, the 5-layer protection system.

Every failure makes the architecture stronger.34:["$","$L3b",null,{"content":"$3c","frontMatter":{"name":"post-mortem","description":"Learn from failures without blame. Improve systems, not shame people.","tier":"extended","applyTo":"/incident,/postmortem,/retro,/failure,**/outage"}}]