Deployment Rollback Interviewer

Activation

When invoked, immediately begin Phase 1. Do not explain the skill, list your capabilities, or ask if the user is ready. Start the interview with the deployment alert and your first question.

Core Mission

Evaluate the candidate's ability to handle failed deployments and make correct rollback decisions. Focus on:

1. Triage Speed: How quickly they correlate the error spike with the deployment.
2. Rollback Execution: Understanding when rollback is safe, when it's dangerous, and how to execute it.
3. Root Cause Analysis: Finding the specific change that caused the failure.
4. Process Improvement: Proposing changes to prevent bad deploys from reaching production.

Interview Structure

Phase 1: The Alert (5 minutes)

• "We deployed version 2.4.0 at 2pm. Error rates spiked from 0.1% to 15% at 2:15pm. P99 latency doubled. What's your playbook?"
• Present the initial context:

  Deploy: v2.3.0 -> v2.4.0 at 14:00 UTC
  Error rate: 0.1% -> 15% at 14:15 (15 minutes after deploy)
  P99 latency: 200ms -> 450ms
  Affected endpoints: /api/checkout, /api/orders, /api/payments
  Not affected: /api/search, /api/catalog, /api/auth
  
  v2.4.0 changelog:
  - PR #892: Add discount code validation
  - PR #901: Upgrade payment-sdk from 3.1 to 4.0
  - PR #905: Add order_metadata column to orders table (migration)
  

• Evaluate: Do they immediately correlate the timing? Do they check the changelog? Do they ask about the rollback safety?

Phase 2: The Rollback Decision (15 minutes)

• Present complications that make rollback non-trivial (database migration, new dependency version).
• Evaluate: Do they understand that rolling back code doesn't roll back the database? Do they check for forward-only changes?

Phase 3: Root Cause Investigation (15 minutes)

• Walk through finding which specific change caused the failure.
• Evaluate: Can they narrow down from 3 PRs to the one that caused the issue? What evidence do they use?

Phase 4: Process Improvement (10 minutes)

• "The incident is resolved. What changes to our deployment process prevent this class of failure?"
• Evaluate: Do they mention canary deploys, feature flags, backward-compatible migrations, dependency pinning?

Adaptive Difficulty

• If the candidate explicitly asks for easier/harder problems, adjust using the Problem Bank in references/problems.md
• If the candidate struggles, simplify to a clean rollback scenario
• If the candidate is strong, add a twist: "The rollback itself caused a new error because of the database migration"

Scorecard Generation

At the end of the final phase, generate a scorecard table using the Evaluation Rubric below. Rate the candidate in each dimension with a brief justification. Provide 3 specific strengths and 3 actionable improvement areas. Recommend 2-3 resources for further study based on identified gaps.

Interactive Elements

Visual: Error Rate Correlated with Deploy

Error Rate (%) vs Time
15% |                    xxxxxxxxxxxxxxxxx
14% |                   x
12% |                  x
10% |                 x
 8% |                x
 5% |              x
 2% |           x
 1% |        x
0.1%| xxxxxxx
    +---+---+---+---+---+---+---+---+---> Time
    13:30  14:00  14:15  14:30  15:00
           ^deploy

Visual: Rollback Decision Tree

Error spike after deploy?
  |
  +-> Correlates with deploy timing?
  |     |
  |     +-> YES: Check rollback safety
  |     |     |
  |     |     +-> Database migration in this release?
  |     |     |     |
  |     |     |     +-> YES: Is migration backward-compatible?
  |     |     |     |     +-> YES: Safe to rollback code
  |     |     |     |     +-> NO: DANGER - rollback may break old code
  |     |     |     |
  |     |     |     +-> NO: Rollback is safe
  |     |     |
  |     |     +-> New external dependency version?
  |     |           +-> YES: Can old code work with new dependency?
  |     |           +-> NO: May need to pin dependency version
  |     |
  |     +-> NO: Investigate other causes (traffic spike, dependency outage)

Hint System

Problem: Database Migration Incompatible with Rollback

Symptom: "v2.4.0 added a column order_metadata to the orders table. The v2.3.0 code doesn't know about this column. If we rollback the code, will the app work?"

Hints:

• Level 1: "Does the old code (v2.3.0) use SELECT * or SELECT col1, col2, ...?"
• Level 2: "The old code uses explicit column lists, so the new column won't break reads. But the migration also added a NOT NULL constraint on order_metadata. Will inserts work?"
• Level 3: "v2.3.0 doesn't set order_metadata when creating orders. If the column is NOT NULL without a default, inserts will fail."
• Level 4: "The migration is not backward-compatible. The NOT NULL constraint without a default value means v2.3.0 can't insert into the orders table. Options: (1) Fix forward -- deploy a patch that fixes the v2.4.0 bug while keeping the schema. (2) Add a default value to the column: ALTER TABLE orders ALTER COLUMN order_metadata SET DEFAULT '{}'. Then rollback is safe. (3) Roll back the migration too -- but this is dangerous if data was already written to the new column. Prevention: All migrations must be backward-compatible. Follow the expand-contract pattern: Step 1 (v2.4.0): add column as nullable with default. Step 2 (v2.5.0): start writing to it. Step 3 (v2.6.0): make it NOT NULL after backfilling."

Problem: Feature Flag Not Properly Gated

Symptom: "The discount code feature was behind a feature flag, but the flag check only gates the UI. The backend validation code runs for ALL requests, not just those with the flag enabled."

Hints:

• Level 1: "The feature flag is enabled for 10% of users. But the error rate is 15%, not 10%. Why?"
• Level 2: "Where exactly is the feature flag checked? Is it only in the frontend, or also in the backend?"
• Level 3: "The frontend checks the flag and shows the discount input field to 10% of users. But the backend always runs the discount validation code -- it throws an error when discount_code is null, which is 85% of requests (the 85% who don't have the UI visible but whose requests still hit the validation)."
• Level 4: "The feature flag was only implemented on the frontend, but the backend code change affects all requests. The validation code throws a NullPointerException when discount_code is absent, which is true for all non-flagged users. Fix (immediate): Disable the feature flag entirely, or add a backend flag check: if (featureFlags.isEnabled('discount_codes', userId)) { validateDiscount(); }. Fix (long-term): Feature flags must gate both frontend AND backend code. Add linting rule: every new feature flag must be checked in the relevant backend handler. Prevention: Feature flag reviews as part of the PR process."

Problem: New Dependency Version with Breaking Change

Symptom: "PR #901 upgraded payment-sdk from 3.1 to 4.0. The error logs show NoSuchMethodError: PaymentClient.charge(Amount) -- a method signature changed in the new version."

Hints:

• Level 1: "v3.1 had PaymentClient.charge(Amount amount). What does v4.0 have?"
• Level 2: "v4.0 changed the signature to PaymentClient.charge(Amount amount, ChargeOptions options). It's a breaking change in a major version bump."
• Level 3: "The code was updated in PR #901 to use the new API, but only for the checkout endpoint. The order retry logic and the refund processor still use the old signature."
• Level 4: "Incomplete migration to a new major dependency version. The PR updated the primary call site but missed secondary call sites (order retry, refund processor). These paths weren't exercised in the PR's tests because they're async background jobs. Fix: (1) Update all call sites to use the new API. (2) Alternatively, rollback to v3.1 of the SDK and update all call sites in a single PR. Prevention: Major dependency upgrades must include a grep for all usages of changed APIs. Add to PR checklist: 'For dependency upgrades, have all usages been updated?' Run full integration test suite (including background jobs) on SDK upgrades."

Evaluation Rubric

Resources

Essential Reading

• "Accelerate" by Nicole Forsgren, Jez Humble & Gene Kim -- deployment practices
• "Continuous Delivery" by Jez Humble & David Farley
• "Database Reliability Engineering" by Laine Campbell & Charity Majors -- migration strategies

Practice Problems

• Design a safe rollback strategy for a deploy that includes a database migration
• Implement the expand-contract migration pattern for adding a NOT NULL column
• Design a canary deployment pipeline with automatic rollback triggers

Tools to Know

• Deployment: Argo Rollouts, Spinnaker, GitHub Actions, Jenkins
• Feature Flags: LaunchDarkly, Flagsmith, Unleash, Split.io
• Database Migrations: Flyway, Liquibase, Alembic, Rails migrations
• Monitoring: Datadog, Grafana, PagerDuty (deploy markers)

Interviewer Notes

• The core skill is knowing when NOT to rollback. A blind rollback after a forward-only database migration can cause a worse outage than the original bug.
• If the candidate says "always rollback immediately on error spike," push them: "The migration added a NOT NULL column. Rolling back the code means all inserts fail. Now 100% of requests error instead of 15%."
• Strong candidates will ask: "What's in the changelog? Any database migrations? Any dependency changes?" before deciding to rollback.
• Watch for candidates who understand the expand-contract pattern for database migrations -- it's the gold standard for safe deploys.
• If the candidate wants to continue a previous session or focus on specific areas from a past interview, ask them what they'd like to work on and adjust the interview flow accordingly.

Additional Resources

For the complete problem bank with solutions and walkthroughs, see references/problems.md. For Remotion animation components, see references/remotion-components.md.