Bulk workspace contract delete (soft → hard) mitigation

When to use / not use

Expected behavior (code)

Soft delete maps to ContractStatus.DELETED:

        if update_status == UpdateStatus.DELETE:
            return ContractStatus.DELETED

Hard delete requires current status DELETED:

        if update_status == UpdateStatus.HARD_DELETE:
            if current_status != ContractStatus.DELETED:
                raise ContractHardDeleteNotAllowed(
                    "Only deleted contracts can be permanently deleted."
                )
            return ContractStatus.HARD_DELETED

After HARD_DELETED, _handle_post_hard_delete_actions runs (relations, contract events, packet removal, etc.) — see contracts_v3/services/contract_service.py.

ContractV3Service.update_contract_status bulk-updates search and, for DELETE only, calls _post_contract_delete_actions — not for HARD_DELETE (post-actions differ at ContractService layer).

Mitigation script pattern (django-rest-api)

Reference implementation (adjust IDs, WORKSPACE_ID, USER_EMAIL, DRY_RUN):

• scripts/oncall_mitigations/phonepe_bulk_delete_contracts.py

Constants:

• CONTRACT_IDS: explicit list (count and IDs verified in Metabase on contracts_v3_contractv3: created_by_workspace_id = {workspace_id} and id in (...)).
• USER_EMAIL: must resolve to a User who has an OrganizationUser for that workspace (used for updated_by / audits). Example from incident: {user_email} → confirm via Metabase on org user / auth user tables.
• STATUS: UpdateStatus.DELETE for phase 1; UpdateStatus.HARD_DELETE for phase 2.

Auth context:

• Build SDBaseServiceAuthContext with org_user=OrganizationUserService.get_org_user_by_user_id_and_workspace(user_id=user.id, workspace_id=workspace.id) — do not pass org_user=None with a non-None context ( ensure_object_permission asserts). Alternative: auth_context=None on update_contract_status (skips object-level permission enforcement; workspace scoping still applies on contract fetch).

Dry run: set DRY_RUN = True; script should not call update_contract_status — only validates user, workspace, contract IDs, and prints intent.

Real run: DRY_RUN = False; prefer transaction.atomic() around the service call.

Verification (Metabase / SQL)

Before soft delete: IDs exist and belong to workspace:

• Filter: created_by_workspace_id = {workspace_id}, id in ({contract_id_list}).

Before hard delete: all targets status = 'DELETED' (or run fails with ContractHardDeleteNotAllowed).

After hard delete: status = 'HARD_DELETED' for all IDs.

Use saved questions or inline filters; incident used metabase.in.spotdraft.com question links with database: 2, source-table: 75 (table id for Contract V3 — confirm in your env if schema differs).

Available MCP tools

Slack

• slack_read_channel — incident channel (e.g. #incident-*).
• slack_read_thread — review script approval / Metabase links in threads.

Jira (Atlassian MCP)

• Example: SPD-43674 — search “bulk delete”, “PhonePe”, “contract”.

BigQuery

• Mirror: spotdraft-prod.prod_{region}_db.public_contracts_v3_contractv3 with created_by_workspace_id and status.
• Map {region} to cluster (india, eu, usa, mea).

Groundcover

• Not usually required unless update_contract_status or Celery follow-up tasks fail; filter by contract_id, workspace_id, error text.

Investigation workflow

1. Confirm scope: workspace id, cluster, exact count of IDs, customer approval, Jira / Rootly link.
2. Resolve attribution email with CS — must be a valid org user for the workspace.
3. Metabase: all IDs belong to created_by_workspace_id; note any VOIDED / non-deletable states — may require soft delete first.
4. Phase 1: STATUS = DELETE, DRY_RUN true → then false; re-check Metabase DELETED.
5. Phase 2: STATUS = HARD_DELETE, dry run → execute; verify HARD_DELETED.
6. Attach shell transcript / Metabase screenshots to Rootly; request reviewer to close.

Known issue patterns

Escalation

• Management command requested for recurring bulk deletes — track as product/CLI work (see incident thread from Gourav Singh on 2026-04-14).
• Data recovery after HARD_DELETED — not reversible via normal product flows; involve DBA / backup policy.

Related incidents / links

• Rootly 3116 — PhonePe bulk delete, 196 contracts, workspace 5092.
• Jira SPD-43674.
• Prior art Slack: C0A35A98UKU (referenced in channel for similar task).

Connections to other skills

• Contract search / ES issues after delete — use contract search resync skills if UI still shows deleted contracts.
• Void contracts — may block or complicate transitions; align with get_status_update_for_contract and workflow state (escalate if not straightforward DELETE).