Babysit Job

Example resubmit command: uv run iris --config lib/iris/examples/marin.yaml job run --no-wait --extra marin:tpu --tpu v5litepod-16 -- python experiments/tutorials/train_tiny_model_tpu.py

If any required field is missing, ask for it before proceeding.

Scope

• Recovery is stop then resubmit at the job level.
• Cluster-level actions are out of scope. Do not restart, recreate, or otherwise mutate the cluster unless the user gives explicit consent in the current thread.
• For TPU bad-node errors, escalate to debug-tpu.

Monitoring Ownership and Duration

• Assign a single monitoring owner when the loop starts.
• Keep this loop running until one of the following:
  • the job reaches a terminal state and the user has acknowledged next action
  • a user-specified stopping point is reached
  • an unrecoverable error is found and reported to the user
• Do not stop early after seeing first loss lines, first eval, or first W&B link.
• Expect monitoring to commonly take 4-5 hours for ferry-scale runs.
• For GPT Codex specifically: unless otherwise directed, do not end your turn just to give a status update; keep monitoring until terminal state or until the user's goal is reached.
• If the user requests continuous monitoring, do not end the turn while monitoring is active; continue until terminal state, a user-specified stopping point, or an unrecoverable error.
• If handoff is needed, transfer ownership explicitly with: current job_id, latest error/signal, W&B link(s), and resubmission metadata.

Cadence and Tooling Notes

• Cadence default after startup stabilization is sleep 570.
• Startup stabilization sequence (after submit/resubmit):
  • once the job is submitted, sleep 120 and check for immediate failure
  • if still alive, switch to the normal 570 cadence
• Tool-runtime workaround:
  • keep one long-running monitor process/session
  • poll the same session in ~30 second chunks as needed by tool runtime limits
  • repeated no-output polls are expected while waiting for the next 570-second check
• Single monitor process rule:
  • run only one active monitor loop per job to avoid duplicate SSH tunnel and port-binding conflicts
• Sleep must be foreground (max ~10 min due to tool timeout).
• Loop control is at agent level, not bash.

State File

Write to scratch/<create_timestamp>_monitoring_state.json, create the scratch directory if needed. <create_timestamp> should have format YYYYMMDD-HHMM. Track restart_count to detect flapping. State file allows resume after context reset.

{
  "ts": <timestamp_ms>,
  "job_id": "<JOB_ID>",
  "config": "<IRIS_CONFIG_PATH>",
  "resubmit_command": "<IRIS_JOB_RUN_COMMAND_WITH_NO_WAIT>",
  "restart_count": 0
}

Loop

1. SLEEP
   - if just submitted/restarted: sleep 120 once
   - otherwise: sleep 570

2. CHECK LOGS
   uv run iris --config <CONFIG> job logs --since-seconds 900 --include-children <JOB_ID> | rg -i -e "loss|error|traceback|exception|resource_exhausted|oom|compiler_base\.cc:2587|program hbm requirement|largest program allocations|ownerdiederror|dead node|node death|autoscaler unsatisfied resources|no accelerator found|failed_precondition|device or resource busy"

3. CHECK STATUS
   uv run iris --config <CONFIG> job list --json --prefix <JOB_ID>

   Terminal success: JOB_STATE_SUCCEEDED
   Terminal non-success: JOB_STATE_FAILED, JOB_STATE_KILLED, JOB_STATE_WORKER_FAILED, JOB_STATE_UNSCHEDULABLE
   Non-terminal: JOB_STATE_PENDING, JOB_STATE_BUILDING, JOB_STATE_RUNNING

   If `pending_reason` indicates worker scale-up/capacity wait, treat as scheduler
   capacity wait — do not run cluster update/recreate/restart actions. Continue
   waiting on cadence, or stop+resubmit only if user explicitly asks.

   Treat RUNNING as controller-level signal only; confirm allocation via expected
   W&B run when possible.

3a. ON TERMINAL STATE / OOM-LIKE SIGNAL — get a structured per-task summary
   (final state, exit, duration, peak memory) instead of grepping logs:

   uv run iris --config <CONFIG> job summary --json <JOB_ID>

   Fast postmortem: e.g. "13/14 shards peaked near the container memory limit
   and failed with exit 137" → cgroup OOM, raise `--memory` on resubmit.

4. PRINT W&B RUN IDS/LINKS (once per training run)
5. REPORT PROGRESS (format: ~<current>/<exact_max>)
6. EVALUATE (terminal? error? stalled? -> recover or continue)

7. RECOVER (STOP -> RESUBMIT)
   - If current job is still non-terminal, stop it first:
     uv run iris --config <CONFIG> job stop <JOB_ID>
   - Then resubmit:
     <RESUBMIT_COMMAND>
   - Capture `job_id` from output (line like `Job submitted: /<user>/<job>`).
   - Iris nuance:
     - if `resubmit_command` omits `--job-name`, Iris auto-generates a fresh id each resubmission.
     - if `resubmit_command` uses a fixed `--job-name`, Iris may reuse the same id
       after terminal completion by replacing the finished job.
   - Update state file: `job_id=<NEW_JOB_ID>`, `restart_count += 1`.
   - Go to step 1.

Fixing Small Bugs

When EVALUATE detects an error, before recovery:

1. Analyze logs — look for Traceback, Error, Exception. Identify file and line.
2. If small fix (typo, missing import, wrong variable name): fix it, then RECOVER.
3. If complex (architectural, unclear cause, broad investigation): report to user, exit loop.

Small-fix examples: NameError, ImportError, SyntaxError, obvious KeyError.

Complex examples: OOM, TPU/XLA HBM exhaustion, distributed training failures, data loading issues, unclear multi-file stack traces.

Error Patterns

• Treat TPU/XLA HBM reports as failure even without literal OOM:
  • Program hbm requirement ...
  • Largest program allocations in hbm
• If progress stalls across multiple intervals with OwnerDiedError, dead node, or unsatisfied resources -> mark degraded and notify user.
• If same error repeats after one fix attempt, do not retry blindly; report to user.

When to Escalate

• Debug Zephyr pipeline issues -> debug-zephyr-job
• Debug TPU bad-node errors -> debug-tpu
• Debug running tasks with iris task exec -> debug-iris-job

Notes

• Iris job list --prefix requires canonical job names (/<user>/<job>), not short names.
• Iris monitoring is job-level; cluster updates are not part of normal recovery.