Standardized tmux experiment monitoring with progress parsing, CSV log reading, and status reporting
Standardized workflow for checking all running ML experiments via tmux sessions.
tmux ls
Report each session: name, creation time, attached/detached status.
For each session, capture the last 30-50 lines:
tmux capture-pane -t <session-name> -p | tail -50
Look for common progress indicators in the output:
Epoch X/Y or epoch: XStep X/Y or X/Y iterationsXX.X%loss: X.XXXX, val_loss: X.XXXXaccuracy, f1, aucIf experiments write to CSV logs, read the latest entries:
# Find recent CSV logs
find . -name "*.csv" -newer <start-time> -type f
# Read last few rows
tail -5 <log-file.csv>
Scan output for common failure patterns:
CUDA out of memoryRuntimeErrorTracebackErrorKilled or OOMOutput a structured report:
## Experiment Status Report
| Session | Status | Progress | Metric | ETA |
|---------|--------|----------|--------|-----|
| sweep-lr | Running | 45/100 epochs | loss: 0.234 | ~2h |
| ablation-1 | Running | 78% | acc: 0.891 | ~30min |
| baseline | Completed | 100/100 | loss: 0.198 | - |
| sweep-wd | Error | 23/100 | CUDA OOM | - |
If experiments.md exists in the project, update it with:
/check-experiments commandexperiment-monitor.js hook shows brief summary at session startstop-summary.js hook shows running session count at session end