Sync

Sync Experiment Status

Synchronize local experiment tracking with actual SLURM job states.

Steps

1. Load tracked experiments using list_experiments MCP tool, filtering for non-terminal states (status = queued, running)

2. For each active experiment, check its SLURM status:

   • Use job_info with the experiment's cluster and job_id
   • Parse the state: PENDING → queued, RUNNING → running, COMPLETED → completed, FAILED → failed, CANCELLED → cancelled, TIMEOUT → failed

3. For completed experiments, try to extract metrics:

   • Use find_logs to locate the output log
   • Use tail_log to read the last ~50 lines
   • Look for common metric patterns: "val_acc", "best_metric", "loss", "accuracy", "eval"
   • Update the experiment's best_metric field

4. For failed experiments: