Autonomous Research

1. Audits architecture and reads prior experiment history
2. Searches literature for better approaches
3. Proposes and ranks changes
4. Implements behind CLI flags
5. Runs training (autonomous) or provides commands (collaborative)
6. Validates measurement system before trusting metrics
7. Diagnoses results — interprets metric deltas to identify root causes
8. Searches literature for the specific diagnosed problem
9. Designs isolation experiments when bundled changes produce unexpected results
10. Accumulates cross-experiment evidence — patterns across multiple trials
11. Recognizes when to stop — analytical derivation or removal vs. more training
12. Reprioritizes based on diagnostic evidence and literature
13. Journals everything and loops

Required Inputs

1. Model code path: Where model definition and training code live
2. Model objective: What the model does and what downstream consumers need
3. Training command template: Base command (agent appends flags)
4. Target metrics with thresholds: What "good enough" looks like
5. Iteration budget: Max experiment iterations (default: 10)
6. Mode: Autonomous or Collaborative

Autonomy Principles

Decision Authority

• Autonomous mode: NEVER asks permission — decides based on metrics, evidence, rules
• Collaborative mode: Proposes experiments with full commands, waits for results, solicits feedback
• Uncertain between options → pick stronger literature support. Equal → pick simpler

Conservative Code Changes

• Every change behind a CLI flag, existing behavior as default
• ONE hypothesis per iteration (unless designing isolation experiment)
• Changes breaking output compatibility are FORBIDDEN
• Crashes → revert immediately, journal the failure

Evidence-Based Decisions

• Continue only if metrics trend in hypothesized direction
• Stop early if: NaN/Inf, gradient explosion, 20%+ worse after warmup, collapsed outputs
• SUCCESS: ≥5% improvement on primary metric vs best-so-far
• FAILED: primary metric regresses or flat
• INCONCLUSIVE: some metrics improve, others regress

Measurement-First Principle

Before trusting ANY metric: Is it computed? Computed correctly? Does it measure what we think? When paradoxical → investigate measurement code before model.

Automatic Escalation

Decide what to investigate based on evidence:

• Loss curves → optimization → param tuning
• Contradicting losses → loss redesign
• Probe shows info exists but model can't use → architecture
• Output worse than random → analytical derivation or removal
• Literature shows clearly better approach → architecture change

Budget Management

• Front-load high-impact changes (architecture, loss) over incremental tuning
• Reserve ≥2 iterations for validation of best config
• Isolation experiments count against budget

Procedure

Phase 0 — Bootstrap

1. Read prior state (parallel sub-agents):

   • A: Experiment journal → last trials, best metrics, known failures
   • B: Architecture file at /memories/repo/architecture.md
   • C: Model code + training script → CLI flags, components
   • D: Ideas Inbox at /home/elerson/Documents/Obsidian Vault/Fovea/Training models - Ideas.md

2. Validate measurement pipeline: For each target metric, trace computation → storage → reporting. Check polarity/encoding. Check init values. Fix bugs first.

3. Establish baseline: Use recent results (< 24h) or run default-config training. Record ALL metrics.

4. Architecture audit: Document strengths, weaknesses, unused components, bottlenecks, capacity, inductive bias.

5. Literature survey (parallel sub-agents):

   • A: Time-series encoding SOTA
   • B: Multi-task learning best practices
   • C: Representation learning for RL
   • D: Problem domain (LOB prediction, financial forecasting)

6. Read Ideas Inbox: Parse ideas, score by (impact × feasibility) / complexity, mark as QUEUED/TESTED/DEFERRED/OUT_OF_SCOPE.

7. Build research agenda: Rank all candidates. Interleave inbox items with literature/diagnosis ideas.

8. Initialize working file → See collaborative-and-templates.md for template.

Phase 1 — Research Cycle (repeats)

Step 1 — Select Next Experiment

1. Re-read working file
2. Re-read Ideas Inbox for new entries
3. Pick top-priority untried experiment
4. If agenda empty → generate new hypotheses from accumulated results
5. If no hypotheses → Final Report

Step 2 — Implement the Change

1. Write code behind CLI flag (default = existing behavior)
2. Add flag to argparse with help text
3. Guard code with if args.flag:
4. Create config entry in configuration_suggestions.md
5. Update architecture file if architecture change
6. Verify default path not broken

Framework checks (JAX/Flax/JIT):

• New boolean params in static_argnums?
• JIT-traced dict structure changed?
• Flax initializers only run during model.init(), not model.apply()
• Count positions for static_argnums carefully
• Build before running

Step 3 — Run Training

Autonomous mode:

1. Kill existing processes: pkill -f "train_" 2>/dev/null; sleep 2
2. Launch: {command} --{flag} {value} > /tmp/run_{N}.log 2>&1 &
3. Record start time, PID, flags

Collaborative mode:

1. Present command + hypothesis + metrics to watch (with Research Progress Summary if not first trial → see collaborative-and-templates.md)
2. Wait for results

Step 4 — Monitor Training (autonomous only)

Poll log every 60-90 seconds:

• Extract key metrics
• STOP: NaN/Inf, gradient explosion, 20%+ worse after warmup, collapsed outputs
• CONTINUE: Trending right, too early, neutral
• COMPLETE: Script exited or wall-clock exceeded

Step 5 — Evaluate Results

1. Compare all metrics against baseline and best-so-far
2. Classify: SUCCESS (≥5% improvement) / PARTIAL / FAILED / CRASHED
3. → Read diagnostic-investigation.md and execute Steps 5b.1-5b.8
4. Collaborative mode only: → Execute Step 5c from collaborative-and-templates.md

Step 6 — Journal & Iterate

→ Use journal entry template from collaborative-and-templates.md.

1. Write to experiment journal
2. Update configuration_suggestions.md
3. Update architecture file Change Log
4. Update working file (budget, agenda, findings)
5. Check termination:
   • All targets met → Final Report
   • Budget exhausted → Final Report
   • No hypotheses → Final Report
   • Otherwise → Step 1

Final Report

→ Use final report template from collaborative-and-templates.md.

Monitoring Protocol

Log Parsing Patterns

loss_total:, IC, BSS, RegAcc, Dir Acc, grad_norm, NaN/inf, Epoch, SpreadRatio, AUC

Health Checks

• Process running (kill -0 $PID)
• GPU memory not exhausted
• Disk space sufficient
• Log still being written to

Safety Rules

1. All changes behind flags — default path NEVER modified
2. One hypothesis per iteration — no bundling
3. Output compatibility — downstream consumers must work
4. No destructive actions — no deleting checkpoints, no force-push
5. Budget enforcement — hard stop at limit
6. Crash recovery — fix bug, retry once; two crashes = abandon
7. Revert path — no flag = original behavior
8. Journal everything — every decision, metric, hypothesis recorded

Context About This Project

• Asset: ES (S&P 500 E-mini futures)
• Input: LOB snapshots + Renko regime features
• Outputs: Return predictions (RetIC), regime classification, win probability, direction, volatility
• Metrics: RetIC, Direction Accuracy, Regime BSS, volume prediction quality
• Architecture: projects/training_lab_jax/models/hierarchical_transformer.py
• Training: projects/training_lab_jax/experiments/train_hierarchical.py
• Journal: projects/training_lab_jax/experiments/experiment_journal.md
• Config: configuration_suggestions.md
• Framework: JAX + Flax
• Current arch: PatchTST → shared trunk → specialized heads (~700K-800K params)