Report Research

3) Extract metrics for the report

For each run you report, include at least:

• corr_mean
• bmc_mean
• bmc_last_200_eras.mean
• avg_corr_with_benchmark (from the BMC summary)

Prefer a single markdown table with one row per model.

4) Write a full report in experiment.md

Update/create experiment.md with these sections (keep it crisp but complete):

• Title + Date
• Abstract (what was tested + headline result)
• Hypothesis / Motivation (why this should help BMC)
• Method (data split, CV, feature set, model type/hparams, any transforms)
• Experiments run (one subsection per config that actually ran; include output artifacts)
• Results (the metrics table; mention best run + trade-offs)
• Standard plot (embed the PNG and include the generating command)
• Decisions made (what you chose and why; e.g., per-era vs global, feature set choice, sweep choices)
• Stopping rationale (why you stopped iterating; e.g., plateau after N rounds, confirmatory scale step, diminishing returns)
• Findings (what worked / didn’t; interpret the plot)
• Next experiments (2–5 concrete follow-ups)
• Repro commands (train + plot commands from repo root)

5) Generate the standard plot(s) and link them

Default standard plot (baseline = benchmark predictions):

PYTHONPATH=numerai python3 -m agents.code.analysis.show_experiment benchmark <best_model_results_name> \
  --base-benchmark-model v52_lgbm_ender20 \
  --benchmark-data-path numerai/v5.2/full_benchmark_models.parquet \
  --start-era 575 --dark \
  --output-dir numerai/agents/experiments/<experiment_name> \
  --baselines-dir numerai/agents/baselines

Then embed it in experiment.md with a relative link:

![benchmark vs best model](plots/<generated_plot_name>.png)

If you have multiple candidate models, either:

• generate one plot with multiple experiment models, or
• generate one plot per candidate (and link all of them).

6) Final checks

• Plot files exist under plots/.
• experiment.md links resolve (use relative paths).
• Metrics table matches results/*.json.
• Report clearly states what was run vs what is only planned/configured.