Incident Response

Diagnosis

• Check what changed recently: deployments, configuration changes, dependency updates, infrastructure changes.
• Check monitoring dashboards: error rates, latency, resource utilization, queue depths.
• Check logs for the first occurrence — when did this start?
• Use correlation IDs to trace failing requests end-to-end.
• Narrow scope: is it application, infrastructure, database, or external dependency?

Post-Incident

After the incident is resolved:

Timeline Reconstruction

• When was the problem introduced?
• When was it detected?
• When was it mitigated?
• When was it fully resolved?

Root Cause

• MUST identify the root cause, not just the proximate cause.
• "The deployment broke it" is the proximate cause. "The deployment had no health check, so a broken build reached production" is the root cause.

Action Items

• Categorize actions as: prevent (stop it from happening), detect (catch it faster), mitigate (reduce blast radius).
• Each action MUST have an owner and a deadline.
• Prioritize actions that prevent recurrence over actions that detect faster.

Blameless Analysis

• Focus on systems and processes, not individuals.
• Ask "why did the system allow this?" not "who did this?"
• Identify process gaps, tooling gaps, and missing safeguards.

Output

For active incidents: triage assessment, immediate mitigation recommendation, diagnosis steps. For post-mortems: timeline, root cause, prioritized action items.