Change Management

Routine patching within tested patch sets

Certificate rotation with established procedure

Scaling operations (adding/removing instances within limits)

Pre-approved configuration changes

Log rotation and archival requirements:

Change must match an approved Standard Change template

Automated testing must pass

Documented rollback procedure exists

Within defined maintenance window normal_low: risk: Low approval: Peer review (1 approver) lead_time: 2 business days examples:

Non-critical configuration changes

Feature flag toggles

Documentation updates to production systems

Adding monitoring dashboards or alerts normal_medium: risk: Medium approval: Team lead + peer review (2 approvers) lead_time: 5 business days examples:

Application deployments with new features

Database schema changes (non-breaking)

Network rule modifications

Integration endpoint changes

Dependency version upgrades normal_high: risk: High approval: CAB review required lead_time: 10 business days examples:

Infrastructure migrations

Breaking database schema changes

Major version upgrades (OS, runtime, database engine)

Changes to authentication or authorization systems

Multi-service coordinated deployments

Changes affecting data processing or compliance controls emergency: risk: Variable approval: Emergency CAB (minimum 2 approvers from on-call) lead_time: None (immediate implementation) examples:

Security vulnerability remediation (active exploitation)

Production outage resolution

Data integrity emergency fixes

Regulatory compliance deadline fixes requirements:

Retroactive full documentation within 48 hours

Post-implementation review required

CAB retroactive review at next meeting Change Request Template change_request: metadata: id: "CR-YYYY-NNNN" title: "" requestor: "" date_submitted: "" target_date: "" change_type: ""

standard | normal_low | normal_medium | normal_high | emergency

description: summary: "Brief description of the change" detailed_description: "Full technical details of what will change" business_justification: "Why this change is needed" affected_systems: [] affected_services: [] affected_users: "Description of user impact" risk_assessment: risk_level: ""

low | medium | high

impact_if_failed: "What happens if the change fails" likelihood_of_failure: ""

low | medium | high

risk_mitigation: "Steps to reduce risk" dependencies: "Other systems or changes this depends on" implementation: change_window: start: "" end: "" maintenance_window: true implementation_steps:

step: "Step 1 description" responsible: "Person/team" estimated_duration: "X minutes"

step: "Step 2 description" responsible: "Person/team" estimated_duration: "X minutes" testing: pre_change_testing:

"Unit tests pass"

"Integration tests pass"

"Staging deployment verified" post_change_verification:

"Health check endpoints responding"

"Key transactions processing successfully"

"No error rate increase in monitoring"

"Performance metrics within baseline" rollback: rollback_plan: "Detailed steps to revert the change" rollback_trigger: "Conditions that trigger rollback" rollback_estimated_time: "X minutes" rollback_steps:

"Step 1: Revert deployment to previous version"

"Step 2: Verify rollback successful"

"Step 3: Notify stakeholders" data_rollback: "Describe any data migration rollback needed" communication: stakeholders_notified: [] notification_sent_date: "" status_page_update: true customer_notification_required: false approvals: technical_reviewer: "" technical_approval_date: "" security_reviewer: "" security_approval_date: "" cab_approval_date: "" cab_notes: "" closure: implementation_date: "" implementation_result: ""

success | partial | failed | rolled_back

post_implementation_review: "" lessons_learned: "" follow_up_actions: [] CAB Workflow cab_workflow: meeting_schedule: regular_cab: "Weekly, Thursday 2:00 PM" emergency_cab: "On-demand, minimum 2 members required" cab_members: permanent:

Engineering Manager (Chair)

Security Team Representative

Infrastructure/SRE Lead

Release Manager advisory:

Business stakeholder (invited per change)

Database administrator (for DB changes)

Network engineer (for network changes) agenda: 1: "Review emergency changes from prior week" 2: "Review high-risk change requests for upcoming window" 3: "Review failed changes and lessons learned" 4: "Discuss upcoming change freeze periods" 5: "Review change metrics and trends" decision_criteria: approve_when:

Risk assessment is complete and accurate

Testing evidence is provided

Rollback plan is documented and feasible

Change window is appropriate

Required approvals obtained

No conflicts with other scheduled changes request_changes_when:

Rollback plan is missing or incomplete

Testing is insufficient for the risk level

Impact assessment needs clarification

Change conflicts with another scheduled change deny_when:

Risk is unacceptable without mitigation

Change window conflicts with freeze period

Dependencies are not resolved

Compliance concerns are unaddressed Emergency Change Procedure emergency_change_process: definition: "A change required to restore service or prevent imminent security compromise" step_1_declare: actions:

On-call engineer identifies need for emergency change

Incident commander approves emergency classification

Minimum 2 approvers from emergency CAB roster contacted

Document initial justification in incident channel step_2_approve: approval_method:

Slack/Teams approval with screenshots preserved

Verbal approval over bridge call (documented in notes)

Emergency approvers can be any 2 of the following roles:

Engineering Manager

SRE/Infrastructure Lead

Security Team Lead

VP of Engineering timeout: "If no response in 15 minutes, escalate to next tier" step_3_implement: actions:

Implement the minimum change needed to resolve the issue

Record all actions taken with timestamps

Monitor for successful resolution

Document any deviations from planned change step_4_verify: actions:

Confirm service restoration

Verify no unintended side effects

Run post-change verification checks

Update status page and stakeholders step_5_document: deadline: "Within 48 hours of implementation" required_documentation:

Complete change request form (retroactive)

Timeline of events and actions

Justification for emergency classification

Approval records (messages, emails)

Post-implementation verification results

Root cause analysis (what made it an emergency)

Preventive actions to avoid future emergency step_6_review: actions:

CAB review at next regular meeting

Assess if emergency classification was appropriate

Identify process improvements

Track emergency change trends Pull Request Template for Changes

Change Request

Type

[ ] Standard (pre-approved, low risk)

[ ] Normal - Low Risk

[ ] Normal - Medium Risk

[ ] Normal - High Risk (CAB required)

[ ] Emergency (retroactive documentation required)

Description

Risk Assessment

Impact if failed:

Likelihood of failure: Low / Medium / High Affected services:

User impact:

Testing Evidence

[ ] Unit tests pass

[ ] Integration tests pass

[ ] Staging deployment verified

[ ] Performance test completed (if applicable)

[ ] Security scan clean (if applicable)

Rollback Plan

Estimated rollback time:

Data rollback needed: Yes / No

Deployment Plan

Target window:

Estimated duration:

Post-Deployment Verification

[ ] Health checks passing

[ ] Error rates within baseline

[ ] Key transactions working

[ ] Monitoring dashboards reviewed

Communication

[ ] Team notified

[ ] Stakeholders notified (if user-facing)

[ ] Status page updated (if applicable)

Approvals Required

[ ] Peer review

[ ] Team lead (medium+ risk)

[ ] Security review (security-impacting changes)

[ ] CAB approval (high risk) CI/CD Change Tracking Automation

GitHub Actions - Automated change tracking

name: Change Management on: pull_request: types: [ opened , synchronize , labeled ] push: branches: [ main ] jobs: classify-change: if: github.event_name

'pull_request' runs-on: ubuntu-latest steps:

uses: actions/checkout@v4

name: Classify change risk id: classify run: | FILES_CHANGED=$(gh pr diff ${{ github.event.pull_request.number }} --name-only)

High risk indicators

if echo "$FILES_CHANGED" | grep -qE 'terraform/|infrastructure/|migrations/|auth/|security/' ; then echo "risk=high"

$GITHUB_OUTPUT echo "::warning::High-risk change detected - CAB review may be required" elif echo "$FILES_CHANGED" | grep -qE 'config/|database/|api/' ; then echo "risk=medium"

$GITHUB_OUTPUT else echo "risk=low"

$GITHUB_OUTPUT fi env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

name: Add risk label run: | gh pr edit ${{ github.event.pull_request.number }}
--add-label "risk:${{ steps.classify.outputs.risk }}" env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

name: Enforce approvals by risk if: steps.classify.outputs.risk

'high' run: | APPROVALS=$(gh pr view ${{ github.event.pull_request.number }}
--json reviews --jq '[.reviews[] | select(.state=="APPROVED")] | length') if [ "$APPROVALS" -lt 2 ]; then echo "::error::High-risk changes require at least 2 approvals" exit 1 fi env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} record-deployment: if: github.event_name

'push' && github.ref

'refs/heads/main' runs-on: ubuntu-latest steps:

name: Record deployment run: | CHANGE_ID="CR-$(date +%Y)-$(printf '%04d' ${{ github.run_number }})" echo "Change ID: $CHANGE_ID" echo "Deployed at: $(date -u +%Y-%m-%dT%H:%M:%SZ)" echo "Commit: ${{ github.sha }}" echo "Author: ${{ github.actor }}" cat

/tmp/deployment-record.json <<EOF { "change_id": "$CHANGE_ID" , "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)" , "commit": "$ {{ github.sha }} " , "author": "$ {{ github.actor }} " , "environment": "production" , "status": "deployed" } EOF Change Freeze Policy change_freeze: definition: "Period during which non-emergency changes are prohibited" scheduled_freezes: year_end: start: "December 15" end: "January 3" scope: "All production changes" major_events:

"Black Friday through Cyber Monday (e-commerce)"

"Tax filing deadline periods (financial services)"

"Open enrollment periods (healthcare)" exceptions_during_freeze: allowed:

Security patches for actively exploited vulnerabilities

Changes required by regulatory deadline

Fixes for P1/SEV1 production incidents approval: "VP of Engineering + Security Lead" communication: announcement: "2 weeks before freeze" reminder: "1 week and 1 day before freeze" daily_status: "During freeze period" lift_notification: "When freeze ends" Change Management Metrics metrics: change_success_rate: description: "Percentage of changes implemented without rollback or incident" target: ">95%" formula: "(successful changes / total changes) * 100" emergency_change_rate: description: "Percentage of changes classified as emergency" target: "<5%" formula: "(emergency changes / total changes) * 100" rollback_rate: description: "Percentage of changes that required rollback" target: "<3%" mean_time_to_implement: description: "Average time from approval to implementation" target: "Varies by type" cab_approval_time: description: "Average time from submission to CAB decision" target: "<5 business days for normal changes" Change Management Checklist change_management_checklist: process_setup:

[ ] Change types defined with classification criteria

[ ] Approval matrix documented (who approves what)

[ ] CAB established with regular meeting schedule

[ ] Emergency change procedure documented

[ ] Change request template created

[ ] Change freeze policy defined tooling:

[ ] PR template includes change management fields

[ ] Automated risk classification in CI/CD

[ ] Branch protection enforces required approvals

[ ] Deployment records captured automatically

[ ] Change audit trail preserved (PR history, approvals) compliance:

[ ] All production changes have documented approval

[ ] Rollback plans exist for every change

[ ] Post-implementation reviews conducted for failures

[ ] Emergency changes documented retroactively within 48 hours

[ ] Change metrics reported monthly

[ ] Audit trail retained for compliance period (1-3 years) Best Practices Classify changes by risk level to apply proportionate controls without slowing low-risk work Automate risk classification based on files changed, services affected, and deployment scope Use PR approvals as the native change approval mechanism for code-driven changes Require rollback plans for every change and test rollback procedures periodically Track emergency changes as a key metric: a high rate indicates systemic process issues Implement change freezes during critical business periods to protect stability Conduct post-implementation reviews for all failed changes to drive improvement Separate duty of implementation from duty of approval (no self-approving changes) Capture deployment records automatically in CI/CD rather than relying on manual entry Keep the CAB focused on high-risk decisions; do not bottleneck low-risk changes through CAB