Executed Contract Blocked by Pending PRE_SEND Approval

Do not use for:

• upload flow re-collecting approvals on V4/V5 contracts: approvals-reset-on-upload-v4-triage
• stale assignee / manual review drift in contract search only: contract-search-stale-assignee-manual-review
• bulk reassignment of active in-flight approvals or V5 restrict-action updates: approval-v5-bulk-reassign-download-restriction-backend

Key identifiers to collect

• {workspace_id}
• {cluster}
• {contract_id} for one impacted example
• {required_approval_id} / {active_approval_id} once queried
• {org_user_email} for the support actor if a Django-shell mitigation is needed
• optional: {request_id} if correlating upload/download requests

Known historical example only: Jira SPD-42470, workspace 179471, cluster IN, sample contract 436143, dry-run count 332.

Environment-to-dataset mapping

Use the same production mapping used in other approval / contract skills:

Replace {project}, {dataset}, and {prefix} in the queries below.

Investigation workflow

1. Confirm the incident signature

Expected symptom bundle:

• contract status is COMPLETED / executed
• execution_date is populated
• customer cannot download the contract
• a PRE_SEND approval is still pending

Slack / Jira clues from the reference incident:

• no active workflow path to finish the approval
• approver was deleted
• customer reported multiple contracts, not just one

2. Quantify blast radius in BigQuery

Run sequentially.

SELECT
  cra.id AS required_approval_id,
  cra.approval_name,
  cra.contract_id,
  cra.org_user_id,
  cra.role_id,
  cra.sent_for_approval,
  c.status AS contract_status,
  c.execution_date,
  c.created_by_workspace_id,
  ca.id AS active_approval_id,
  ca.status AS active_approval_status
FROM `{project}.{dataset}.{prefix}contracts_v3_contractrequiredapprovalv2` AS cra
JOIN `{project}.{dataset}.{prefix}contracts_v3_contractv3` AS c
  ON c.id = cra.contract_id
LEFT JOIN `{project}.{dataset}.{prefix}contracts_v3_contractapprovalv2` AS ca
  ON ca.id = cra.active_approval_id
WHERE c.created_by_workspace_id = {workspace_id}
  AND c.status = 'COMPLETED'
  AND c.execution_date IS NOT NULL
  AND cra.approval_type = 'PRE_SEND'
  AND cra.is_deleted = FALSE
  AND cra.sent_for_approval = TRUE
  AND (ca.status IS NULL OR ca.status NOT IN ('APPROVED', 'SKIPPED'))
ORDER BY cra.contract_id, cra.id;

If this returns rows, the issue class is confirmed.

Use a single-contract drilldown when validating a customer-reported sample:

SELECT
  cra.id AS required_approval_id,
  cra.contract_id,
  cra.approval_name,
  cra.org_user_id,
  cra.role_id,
  ca.id AS active_approval_id,
  ca.status AS active_approval_status,
  c.status AS contract_status,
  c.execution_date
FROM `{project}.{dataset}.{prefix}contracts_v3_contractrequiredapprovalv2` AS cra
JOIN `{project}.{dataset}.{prefix}contracts_v3_contractv3` AS c
  ON c.id = cra.contract_id
LEFT JOIN `{project}.{dataset}.{prefix}contracts_v3_contractapprovalv2` AS ca
  ON ca.id = cra.active_approval_id
WHERE cra.contract_id = {contract_id}
  AND c.created_by_workspace_id = {workspace_id}
  AND cra.approval_type = 'PRE_SEND'
  AND cra.is_deleted = FALSE
ORDER BY cra.id;

3. Validate the backend restriction path

The block is expected from current code behavior:

• contracts_v3/contract_approvals/data/db_repo.py
  • are_approvals_in_progress(...) only checks for non-deleted, sent_for_approval=True, non-terminal approvals for the contract/workspace.
  • It does not short-circuit for COMPLETED contracts.
• sd_auth/permissions.py
  • IsActionRestrictedDueToActiveApproval maps both download and upload views to restricted actions.
  • If compute_restricted_actions_use_case returns DOWNLOAD, the user sees the action blocked.
• sd_organizations_v2/org_user/domain/org_user_can_be_deleted_or_deactivated_check_use_case.py
  • org-user deletion/deactivation checks SCIM keys and external credentials only.
  • It does not inspect or clean up pending approval assignments.

This means a deleted approver can leave behind orphaned PRE_SEND approvals that still gate completed contracts.

4. Rule out the wrong mitigation path

Do not assume the stock reassign_contract_approvals managerie fixes this issue.

Why:

• contracts_v3/management/commands/reassign_contract_approvals.py filters ContractRequiredApprovalV2 with contract__status=ContractStatus.ACTIVE
• the command is meant for reassignment of active in-flight approvals before or during approver changes
• completed/executed contracts in this incident class are outside that command’s write scope

This is the main operational trap. If the contract is already completed, reassignment is usually the wrong end state anyway; the approval should be terminalized so download is unblocked.

5. Logs / telemetry expectations

• Groundcover / app logs are usually low-signal here
• there is often no exception
• treat this as a data-state incident first, not a crash / deployment regression

If logs are required, use them only to correlate the upload/download timeline or identify a request_id.

Mitigation

Correct mitigation shape for completed contracts

The reference incident was resolved with a one-off Django-shell script, not with the stock managerie command.

Your mitigation should be workspace-scoped and should:

1. start from the verified BQ list of matching required_approval_id / contract_id rows
2. create a new terminal ContractApprovalV2 row with status SKIPPED for each affected PRE_SEND approval
3. update each ContractRequiredApprovalV2.active_approval to the new skipped approval
4. create approval-skipped audits attributed to a valid workspace user
5. partially resync affected contracts in search

Operational notes from the incident:

• dry run first
• run in non-peak hours if the workspace has a large number of affected contracts
• re-run the verification query after execution until it returns zero rows

What to ask engineering / support to prepare

If no reusable script already exists for the workspace, ask for a Django-shell script that is explicitly scoped to:

• workspace_id={workspace_id}
• contract__status=COMPLETED
• execution_date IS NOT NULL
• approval_type='PRE_SEND'
• sent_for_approval=True
• active approval status not in APPROVED/SKIPPED

Expected script behavior from the incident:

• dry-run mode prints affected approvals and contracts
• write mode bulk-skips the pending PRE_SEND approvals
• search resync runs for affected contracts after the DB update

Post-mitigation verification

Re-run the workspace blast-radius query above. Success condition:

• zero matching rows remain
• customer confirms download is re-enabled

Confirm / disprove matrix

Escalation triggers

• large workspace blast radius or repeated recurrence across customers
• missing contract version / audit creation failures during mitigation
• uncertain ownership of the one-off script or no safe support actor for audit attribution
• need for permanent product fix:
  • cleanup of pending approvals when approver is deleted
  • completed-contract guard in are_approvals_in_progress(...)

Code paths to reference

• contracts_v3/contract_approvals/data/db_repo.py
• sd_auth/permissions.py
• sd_organizations_v2/org_user/domain/org_user_can_be_deleted_or_deactivated_check_use_case.py
• contracts_v3/management/commands/reassign_contract_approvals.py

Related incidents / tickets

• Jira: SPD-42470
• Rootly: incident 3005 (Probo)
• Slack: channel C0AMYDFLLBA
• Supporting context: later reassignment incidents confirmed reassign_contract_approvals is intended for active approvals before deletion / role changes, not for already completed contracts

Connections to other skills

• contract-lookup for baseline contract / workspace investigation
• approvals-reset-on-upload-v4-triage when upload behavior itself is disputed
• approval-v5-bulk-reassign-download-restriction-backend for active approval reassignment and V5 restriction edits