Chaos Engineer

Load detailed guidance based on context:

Safety Checklist

Non-obvious constraints that must be enforced on every experiment:

• Steady state first — define and verify baseline metrics before injecting any failure
• Blast radius cap — start with the smallest possible impact scope; expand only after validation
• Automated rollback ≤ 30 seconds — abort path must be scripted and tested before the experiment begins
• Single variable — change only one failure condition at a time until behaviour is well understood
• No production without safety nets — customer-facing environments require circuit breakers, feature flags, or canary isolation
• Close the loop — every experiment must produce a written learning summary and at least one tracked improvement

Output Templates

When implementing chaos engineering, provide:

1. Experiment design document (hypothesis, metrics, blast radius)
2. Implementation code (failure injection scripts/manifests)
3. Monitoring setup and alert configuration
4. Rollback procedures and safety controls
5. Learning summary and improvement recommendations

Concrete Example: Pod Failure Experiment (Litmus Chaos)

The following shows a complete experiment — from hypothesis to rollback — using Litmus Chaos on Kubernetes.

Step 1 — Define steady state and apply the experiment

# Verify baseline: p99 latency < 200ms, error rate < 0.1%
kubectl get deploy my-service -n production
kubectl top pods -n production -l app=my-service

Step 2 — Create and apply a Litmus ChaosEngine manifest

# chaos-pod-delete.yaml
apiVersion: litmuschaos.io/v1alpha1