Babysit Training

2. Log Output Analysis

Read the latest training log:

# Find active log
ls -lt output/*.txt | head -3
# Read last 50 lines
tail -50 <latest_log>

Extract and report:

• Current epoch N / max_epochs
• Latest train_loss value
• Latest eval metrics (Record F1, Record EM, SQL EM) if available
• Any error messages or tracebacks

Red flags (act immediately):

• NaN or inf in loss: training diverged. Kill and restart with lower LR.
• CUDA out of memory / OutOfMemoryError: OOM. Kill, reduce batch size or disable auto_batch_size, restart.
• RuntimeError tracebacks: read full traceback, diagnose, fix code, restart.
• Loss increasing for >5 consecutive epochs: possible LR too high or data issue. Flag to user.

3. Metric Trending

Compare current metrics against earlier epochs in the same log:

grep -E "F1 =|train loss =" <log_file> | tail -20

Flag if:

• Record F1 has not improved in the last patience_epochs eval cycles (early stopping should handle this, but verify)
• Train loss plateaued (< 0.1% change over 5+ epochs)
• Dev loss diverging from train loss (overfitting signal)

4. GPU & System Health

nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv,noheader,nounits
df -h / | tail -1

Flag if:

• GPU utilization < 30% for a running training process (possible data loading bottleneck)
• GPU memory usage near 100% of total (OOM risk on next batch size increase)
• GPU temperature > 85C (thermal throttling risk)
• Disk usage > 90% (checkpoint saves will fail silently)

5. Checkpoint & W&B Integrity

# Find the active run directory
ls -lt output/ | head -5
# Check checkpoints exist and are recent
ls -la output/<run_dir>/checkpoints/
# Check W&B is syncing (not offline/crashed)
ls -lt output/<run_dir>/wandb/latest-run/

Flag if:

• No checkpoint files in run directory after >1 epoch
• Checkpoint files are 0 bytes (write failure)
• W&B latest-run/ directory missing or no recent sync files
• wandb-offline-run-* directories exist (W&B fell back to offline mode)

6. STOP File Check

ls -la STOP 2>/dev/null

If STOP file exists, verify training acknowledged it (log should show "Stop file detected"). If training is still running without acknowledgment after 2+ epochs, the stop check may be broken.

Restart Procedure

When a crash is detected and restart is needed:

1. Identify the crash cause from the log tail
2. Find the latest run directory with saved training state:

   ls -lt output/ | grep -E "t5_ft|t5_scr" | head -5
   

3. Check for saved training state:

   ls output/<run_dir>/checkpoints/training_state.pt
   

4. Restart with resume:

   PYTHONUNBUFFERED=1 nohup python3 train_t5.py --resume output/<run_dir> > output/<descriptive_log>.txt 2>&1 &
   

5. Verify restart by checking the new log after ~30 seconds:

   sleep 30 && tail -20 output/<descriptive_log>.txt
   

6. Document the crash as an issue (see Issue Documentation below)

For sweep restarts, the sweep agent handles trial-level recovery automatically. Only restart the sweep process itself if it crashed:

PYTHONUNBUFFERED=1 nohup python3 part1/sweep.py --budget 1.5 --max-hours 12 > output/sweep_restart.txt 2>&1 &

Issue Documentation

When an issue is found, create issues/<descriptive-name>.md:

---