Bulk CONTRACT_EXECUTED webhook re-trigger — timeout / partial completion

Use when:

• Bulk re-trigger CONTRACT_EXECUTED after an incident or backfill
• Logs show SoftTimeLimitExceeded (or similar worker kill) during the run
• You have an explicit pending contract_id list and need safe catch-up
• Discussion references management command re_trigger_contract_executed_webhook

Do not use for:

• Clickwrap / Tray / Salesforce KP Created retrigger → clickwrap-tray-salesforce-sync-triage
• CLICKWRAP download_link 404 right after create → contract-lookup
• Single-contract webhook debugging with no bulk timeout story → general webhook / integration triage

Code paths (django-rest-api)

Payload parity with management command: Both build DomainEventPayload.from_current_context(ContractExecutedEventData(contract=ContractResponseV2.from_orm(contract)), user_id=contract.created_by_id, workspace_id=contract.creator_party.id) then call execute_send_contract_executed_event_use_case.

Investigation workflow

Step 1: Confirm partial completion pattern

• Get total vs completed vs pending IDs from the run log or ticket.
• If failure timestamp aligns with worker/task time limit, treat as timeout tail, not random drops.

Step 2: Identify which entrypoint was used

• Management command: python manage.py re_trigger_contract_executed_webhook {workspace_id} {executed_before_date} [batch_size] [webhook_trigger_delay] ...
• On-call script: scripts/oncall_mitigations/re_trigger_contract_executed_webhook_for_pending_ids.py — intended for only pending IDs after a failed bulk run.

Step 3: Validate pending IDs against DB (read-only)

In manage.py shell (or SQL), for each {contract_id}:

• Row exists on ContractV3
• contract_roles include workspace_id = {workspace_id} if you enforce tenant guard
• Contract is in the executed / completed state you expect for a legitimate CONTRACT_EXECUTED resend (product judgment — do not resend for wrong lifecycle)

Step 4: Dry run before side effects

Run mitigation execute(dry_run=True) first. Expected behavior:

• Same batch loop and payload construction as live run
• Logs WOULD_TRIGGER / WOULD_FAIL per id
• Does not call execute_send_contract_executed_event_use_case when dry run

Then execute(dry_run=False) for real sends.

Run context: Use python manage.py shell so Django is configured; do not run the script as a bare python file without django.setup().

Step 5: If timeouts persist

• Reduce batch_size or increase delay only if rate limits matter; primary fix is smaller total work per worker invocation (chunk ID list across multiple runs) or raise task soft time limit for a dedicated maintenance task (platform change — not ad hoc).
• Long-term: extend re_trigger_contract_executed_webhook with optional --contract-ids / file input so support does not rely on a one-off script (product engineering follow-up).

Mitigation summary

1. Do not re-run the full date-range command for all {total} if {completed} already succeeded — risk duplicate webhook noise downstream.
2. Do re-trigger only {pending_contract_ids} using the on-call script pattern or an equivalent small batch.
3. Always dry_run=True first; then dry_run=False.
4. Keep workspace_id guard aligned with WORKSPACE_ID in script or pass workspace_id= into execute().

Escalation / follow-ups

• Duplicates in downstream: If customer already dedupes by contract_id + event id, confirm; if not, coordinate before mass retrigger.
• Command hardening: File ticket to add ID list / resume cursor to re_trigger_contract_executed_webhook and/or run bulk retrigger as chunked Celery tasks with per-task time limits.

Related skills

• clickwrap-tray-salesforce-sync-triage — different webhook family (KP Created / Tray), similar “retrigger after gap” idea
• contract-lookup — contract shape, workspace fields, Tray tables

Incident input note

Slack: Incident thread may live at https://spotdraft.slack.com/archives/C0AQ293P40L (channel id C0AQ293P40L). This skill is generalized from bulk CONTRACT_EXECUTED retrigger + SoftTimeLimitExceeded + pending ID list pattern; replace placeholders with values from the thread, logs, and Metabase.