Results Analysis

Analysis rules

• Do not call a result "better" until the comparison is direct.
• If the run count, seed coverage, or sample size is weak, say the evidence is exploratory.
• Separate:
  • observed metric difference
  • statistical or practical importance
  • causal interpretation
• Keep negative results visible.
• Prefer simple, defensible figures over decorative plots.
• Prefer tables over inline arrow prose for high-signal multi-run trends.
• Every markdown table in analysis_report.md, stats_appendix.md, or figure_catalog.md must have a numbered caption immediately above it.
• Table captions must state the comparison scope, the relevant experiment or artifact set, and the key variables shown.
• Prefer run-root plots for per-run evidence and reserve analysis-generated figures for cross-run comparisons that need a shared visual.
• Use repo-root paths such as /external/... or /research/... when linking or embedding figures in Markdown artifacts.
• Embed only the smallest set of generated runtime figures needed to support the argument. Link the rest from the figure catalog.
• If an eligible completed run is missing its canonical plots, backfill them before calling the figure unavailable.
• If a figure cannot be generated, specify exactly what should be plotted and from which files.
• Do not replace a background-analysis report with an experiment-only schema.
• Do not replace a materially superseded live interpretation without freezing the prior three-file analysis bundle in research/analysis/history/.
• Keep the live appendix and figure catalog current-focused; put historical copies in frozen snapshot bundles instead of accumulating history sections there.
• Treat frozen snapshot bundles as immutable historical records, not as files to normalize back into the latest live schema.

Output

The analysis bundle should answer:

• What was compared?
• Which comparisons are valid?
• What improved, regressed, or failed?
• How strong is the evidence?
• Which figures support the comparison, and were they reused or backfilled?
• What still needs verification or approval?

References and templates

• Use assets/analysis-report-template.md.
• Use assets/stats-appendix-template.md.
• Use assets/figure-catalog-template.md.
• Use references/analysis-checklist.md.