Rollback Recovery

Patterns & examples

Rollback Decision Criteria:

• Critical Failure: Core functionality is broken for all users.
• Widespread Regressions: Multiple non-critical but important features are broken.
• Data Corruption: A change is causing incorrect data to be written.
• Performance Collapse: Service response times are making the system unusable.

Rollback Sequence:

1. Identify: Recognise the failure via monitoring or user reports.
2. Evaluate: Quickly decide if "fixing forward" or rolling back is the safest path.
3. Execute: Perform the rollback procedure (e.g., git revert, helm rollback, or blue/green toggle).
4. Verify: Ensure service is restored and no new issues are introduced by the rollback itself.

Git Revert vs. Reset Pattern:

# ✅ Correct: Use git revert for shared history to maintain a clear audit trail
git revert <commit_hash>
git push origin main

# ❌ Wrong: Using git reset --hard on a shared branch can break other developers' local copies
# git reset --hard <previous_commit_hash>
# git push origin main --force

Anti-patterns to avoid

• ❌ "Hope as a strategy" — Deploying changes without a clear, documented rollback plan.
• ❌ Ignoring data rollbacks — Failing to consider how to revert database migrations or schema changes.
• ❌ Manual-only rollbacks — Relying on complex, manual steps to revert a change during an emergency.

KB Reference

~/vaults/baphled/3. Resources/Knowledge Base/AI Development System/Skills/DevOps-Operations/Rollback Recovery.md

Related skills

• incident-response — Coordinating mitigation and response
• release-management — Managing the delivery lifecycle
• monitoring — Detecting failures and verifying recovery
• feature-flags — Disabling features without re-deploying
• devops — Core infrastructure and deployment patterns