Monitor

Step 1: Validate Inputs

1. Verify each log file path exists (or its directory exists — file may not be created yet)
2. Verify corresponding experiment scripts are running:

   # Check if training process is still running
   ps aux | grep "<exp_id>" | grep -v grep
   

   Or check for PID files in <exp_root>/logs/<round_dir>/<exp_id>/pid

Step 2: Poll Loop

For each monitoring cycle:

2a: Read Latest Log Content

For each log file:

# Read the last N lines of the log file
tail -100 <exp_root>/logs/<round_dir>/<exp_id>/train.log

2b: Parse Metrics

Parse the log content for the watched metric:

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/parse_logs.py <exp_root>/logs/<round_dir>/<exp_id>/train.log

Extract the metric trajectory (all values of the watched metric over time).

2b.1: Metric Name Fallback

If the watched metric is not found in the parsed records, attempt auto-detection:

1. Case-insensitive match: Try metric_to_watch.lower() against all keys lowercased
2. Prefix variants: Try train_<metric>, val_<metric>, <metric>_train (e.g., loss → train_loss, val_loss) Disambiguation: If multiple prefix variants match (e.g., both train_loss and val_loss), prefer val_<metric> — validation loss is a better divergence signal than training loss. Log which variant was selected to dev_notes.
3. Substring match: Look for any key containing metric_to_watch as a substring (e.g., "loss" matches "total_loss")
4. Report missing: If no match found after all fallbacks, log a warning and report the available metric names to the orchestrator. Do not treat this as divergence. Return status unmonitored with the list of available metrics. The orchestrator handles this status by continuing without divergence checks but with a hard timeout fallback.

2c: Check for Divergence

Run divergence detection on the extracted trajectory, passing lower_is_better and model-category-aware thresholds:

python3 -c "
import json, sys
# sys.path: add the plugin's scripts/ directory
from detect_divergence import check_divergence, get_thresholds_for_category
from parse_logs import parse_log, extract_metric_trajectory

records = parse_log('<exp_root>/logs/<round_dir>/<exp_id>/train.log')
values = extract_metric_trajectory(records, '<metric>')
kwargs = get_thresholds_for_category('<model_category or None>')
kwargs['lower_is_better'] = <lower_is_better>
result = check_divergence(values, **kwargs)
print(json.dumps(result))
"

Alternatively, use the CLI with the --model-category flag:

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/detect_divergence.py '<json_values>' --model-category <model_category>

This applies category-specific thresholds automatically (e.g., RL uses explosion_threshold=20.0 to avoid false positives on reward spikes).

2d: Overfitting Check (if overfitting_check provided)

If overfitting_check was provided in the inputs:

1. Extract both metric trajectories from the log:

   • Train metric: extract_metric_trajectory(records, overfitting_check["train_metric"])
   • Val metric: extract_metric_trajectory(records, overfitting_check["val_metric"])

2. Run overfitting detection:

   python3 ${CLAUDE_PLUGIN_ROOT}/scripts/detect_divergence.py --check-overfitting '<train_json>' '<val_json>' [--model-category <category>]
   

3. If overfitting detected:

   • Report status "overfitting_warning" (do NOT kill the process — overfitting is a warning, not a hard failure)
   • Log to error tracker: category: "overfitting", severity: "warning"
   • Log to behavioral memory: ${CLAUDE_PLUGIN_ROOT}/scripts/goal_memory.py <exp_root> log-behavior training_insight '{"insight":"Overfitting detected: <severity> at step <step>","source":"monitor"}'

2e: Take Action on Divergence

If divergence is detected:

1. Kill the training process:

   # Option 1: PID file (preferred — most reliable)
   kill $(cat <exp_root>/logs/<round_dir>/<exp_id>/pid) 2>/dev/null
   
   # Option 2: Safe pattern match — verify process is a training process before killing
   # First find candidates, then verify cmdline contains python/train before killing
   for pid in $(pgrep -f "<exp_id>"); do
     cmdline=$(cat /proc/$pid/cmdline 2>/dev/null | tr '\0' ' ')
     if echo "$cmdline" | grep -qE 'python|train'; then
       kill "$pid"
     fi
   done
   

   Warning: Never use bare pkill -f "<exp_id>" — it could match unrelated processes.

2. Record the divergence:

   • Read the current experiment result file (if it exists)
   • Ownership check: If the file already has status: "completed" or status: "failed", do NOT overwrite — the experiment finished first. Log to dev_notes: "Monitor detected divergence for <exp_id> but experiment already completed with status '<status>' — skipping overwrite." Skip to step 3.
   • If the file does not exist, or has status: "running": update status to "diverged" and add divergence details to notes:

     {
       "status": "diverged",
       "notes": "Divergence detected: <reason> at step <step>"
     }
     

   • Write the updated result using the Write tool
   • Validate the written file:

     python3 schema_validator.py <exp_root>/results/<exp_id>.json result
     

     If validation fails, fix the JSON before continuing.

3. Log the event: Append to <exp_root>/dev_notes.md:

   ## Divergence Detected
   - Experiment: <exp_id>
   - Reason: <reason>
   - Step: <step>
   - Action: Training process killed
   

4. Log to error tracker:

   python3 ${CLAUDE_PLUGIN_ROOT}/scripts/error_tracker.py <exp_root> log '{"category":"divergence","severity":"warning","source":"monitor","message":"<divergence reason>","exp_id":"<exp_id>","context":{"divergence_type":"<nan|explosion|plateau|drift>","step":<step>,"metric_to_watch":"<metric>"}}'
   

5. Log divergence pattern to behavioral memory:

   When an experiment is killed due to divergence, also log the pattern to behavioral memory:

   python3 ${CLAUDE_PLUGIN_ROOT}/scripts/goal_memory.py <exp_root> log-behavior divergence_pattern '{"description":"<metric> diverged at step <N> with config <config>","affected_branches":["<branch>"],"threshold":{"parameter":"<hp>","value":<val>},"source":"monitor"}'
   

   This helps future hp-tune iterations avoid configs that trigger divergence.

2f: Report Status

For healthy experiments, report status:

Monitoring status:
- exp-001: healthy (loss=0.45 at step 500, trending down)
- exp-002: DIVERGED (NaN at step 350) - process killed
- exp-003: healthy (loss=0.52 at step 480, trending down)

Step 3: Completion

The monitor exits when:

1. All experiments have completed (log files stop being updated)
2. All experiments have been killed (divergence)
3. The orchestrator signals to stop

Monitoring Heuristics

Check frequency

• First 100 steps: check every 10 seconds (early divergence is common)
• Steps 100-1000: check every 30 seconds
• After step 1000: check every 60 seconds

Divergence parameters (defaults, adjust based on model type)

• NaN/Inf detection: Always enabled, immediate kill
• Explosion threshold: 5x rolling average over 10-step window
• Plateau patience: 20 evaluation checkpoints with min_delta=1e-6

See also hp-tune/references/tuning-strategy.md for per-model-type HP guidance that informs threshold selection.

Recommended thresholds by model type

RL Model Monitoring

When model_category = "rl", apply these adjustments:

When divergence_metric is a reward metric (lower_is_better = False):

• Reward plateau patience should be higher (50+ episodes) — RL is inherently noisy
• Do NOT kill on short-term reward drops (exploration causes temporary dips)
• Only flag divergence if reward collapses to <10% of historical max sustained over 50+ episodes
• NaN/Inf detection still applies normally

Common divergence patterns

• NaN in first 10 steps: Learning rate too high. Note this for hp-tune.
• Loss explosion after good start: Possible learning rate schedule issue or gradient accumulation bug.
• Slow plateau: Model capacity may be insufficient, or learning rate too low.

Error Handling

• Log file doesn't exist yet: Wait up to 60 seconds for it to appear (or 180 seconds if the experiment has a code_branch — worktree setup, dataset downloads, and dependency resolution add significant startup time). If the log file still doesn't exist after the wait period, check whether the experiment process is still alive (via PID file at <exp_root>/logs/<round_dir>/<exp_id>/pid). If the process is alive, extend the wait by another 60 seconds. If the process is dead or no PID file exists, report error immediately.
• Log file format unrecognized: Try all parsers, report if none work
• Process already dead: Check exit code, mark as failed if non-zero
• Permission errors: Report and skip that experiment
• Log file exists but is empty: If the log file has 0 bytes after 5 minutes of monitoring, report status "no_output" with reason "Log file empty after 5 minutes — training may have stalled". Log to error tracker with category: "training_failure", severity: "warning", source: "monitor".

Output

Return to the orchestrator a dict per experiment:

• exp_id: Experiment identifier
• status: One of healthy, diverged, completed, failed, no_output
• reason: Divergence reason (if diverged) or null
• step: Step at which divergence was detected (if diverged) or -1
• latest_metrics: Dict of most recent metric values
• metric_trajectory: List of watched metric values over time

When the watched metric is not found after fallbacks (Step 2b.1):

• metric_trajectory: [] (empty — watched metric was never parsed)
• latest_metrics: All other available metrics from the final log line (the watched metric will be absent from this dict)
• reason: "Watched metric '<name>' not found; available: [<list>]"

Important: These status values (healthy, no_output) are internal monitor output for the orchestrator only. They must NOT be written to experiment result JSON files. Result files use: completed, failed, diverged, timeout.