Az Pipelines Debug

Step 1: Verify Auth, Build State, and Approval Gate

printenv AZURE_DEVOPS_EXT_PAT | wc -c
az pipelines build show --id <build_id> --organization https://dev.azure.com/speediedan --project interpretune -o table
curl -sS -u ":${AZURE_DEVOPS_EXT_PAT}" \
  "https://dev.azure.com/speediedan/interpretune/_apis/pipelines/approvals?state=pending&api-version=7.1-preview.1"

Interpretation:

• If the build is notStarted and approvals are pending, approve the run before touching the agent
• If no approval is pending, then inspect queue backlog, worker dispatch, and agent availability next

Step 2: Release the Queued Run

Approve the pending gate directly from the shell:

curl -sS -X PATCH -u ":${AZURE_DEVOPS_EXT_PAT}" \
  -H "Content-Type: application/json" \
  -d '[{"approvalId":"<approval_id>","status":"approved","comment":"Approved via CLI for self-hosted GPU validation."}]' \
  "https://dev.azure.com/speediedan/interpretune/_apis/pipelines/approvals?api-version=7.1-preview.1"

Step 3: Monitor Job Dispatch and Runner Activity

watch -n 30 'az pipelines build show --id <build_id> --organization https://dev.azure.com/speediedan --project interpretune --query "{status:status,result:result,startTime:startTime,finishTime:finishTime}" -o json'
tail -f /opt/az_pipeline_agent/_diag/Agent_*.log
ls -1t /opt/az_pipeline_agent/_diag/Worker_*.log | head
az pipelines agent list --organization https://dev.azure.com/speediedan --pool-id 1 -o table

Interpretation:

• If the agent log only shows keepalive polling and no new worker log appears, the run is still blocked upstream
• If a new worker log appears, switch to that log immediately for step-level failure details
• If the agent is offline or disabled, fix that before editing the pipeline or tests

Step 4: Triage Failure Class

Queue / approval failures

• Build stays notStarted
• No worker log is created
• Approval query returns a pending approval

Action:

• Approve the run first
• Only restart the agent if approvals are clear and the pool still is not dispatching work

Infrastructure or runner failures

• Worker log starts, then dies before pytest output
• Agent log shows docker socket, SSL, or shutdown issues

Action:

• Follow the documented rootless Docker and agent restart flow in the Azure pipeline docs
• Recheck /var/run/docker.sock symlink handling and agent service health

Test-memory failures

• Step exits 137
• Job receives shutdown signal during a heavy pytest phase
• GPU or host memory ramps sharply during a single phase

Action:

• Confirm the phase split is intact before changing memory limits
• Keep baseline coverage CPU-only
• Prefer isolating CUDA-gated tests into IT_RUN_CUDA_TESTS=1, standalone, and profile_ci slices
• Increase swap only after verifying the phase split and fixture retention are not the main cause

Step 5: Local Reproduction Strategy

Use the local Azure reproduction flow in distributed-insight to recreate the containerized runner context. Start with the same phase that failed remotely.

Useful commands:

CUDA_VISIBLE_DEVICES='' python -m pytest --cov=src/interpretune --cov-append --cov-report= src/interpretune tests -v --reruns 2 --reruns-delay 5
IT_RUN_CUDA_TESTS=1 python -m pytest --cov=src/interpretune --cov-append --cov-report= tests -v --durations=50 --reruns 2 --reruns-delay 5
bash ./tests/special_tests.sh --mark_type=standalone
bash ./tests/special_tests.sh --mark_type=profile_ci

Step 6: Fixture and Scope Triage for CUDA-Involved Tests

When CUDA tests still carry too much memory:

• Read fixture_usage.instructions.md before changing test fixtures
• Prefer existing generated fixtures over bespoke fixture builders
• Do not add application-code workarounds for test-environment problems
• Reduce fixture retention in tests before changing product code
• For analysis-heavy classes, prefer the shared helpers in tests/analysis_resource_utils.py
• Use analysis_fixture_scope() so low-RAM runners can fall back to function scope while higher-RAM runners keep class reuse
• Prefer AnalysisExtractionMixin and declarative AnalysisFixtureSpec entries over parity-local extraction helpers
• Narrow heavyweight fixture scope only for the classes or aliases that are actually forcing retention across methods

Step 7: What Not to Do

• Do not permanently disable GPU coverage for the self-hosted pipeline just to avoid OOMs
• Do not assume a build-level queue.name of Azure Pipelines means the YAML job pool changed
• Do not restart the agent before checking for a pending approval
• Do not degrade application code to compensate for fixture or CI-environment issues

Expected Outcome

After following this skill you should know:

• Whether the run was blocked by approval, queueing, agent health, or test memory
• Which Azure worker log corresponds to the active run
• Whether the correct fix belongs in pipeline structure, agent operations, or test fixture scope