Ai Ml Experiment Rigor

• Separate pilot runs from paper runs.
• Keep one source of truth for config values.
• Log enough to reconstruct a run without guessing.
• Prefer a smaller, repeatable study over a larger but irrecoverable search.

Statistical Guardrails

• Do not use a small p-value as a substitute for effect size or practical value.
• Report spread and uncertainty, not only the best run.
• State the unit of variation: seeds, folds, tasks, or users.
• Explain unfair comparisons when budgets or pretrained assets differ.

Output Contract

Return one or more of:

• experiment_spec: ready-to-run design
• comparison_table: baselines, metrics, and fairness notes
• reproducibility_risks: likely failure points
• compute_budget: staged plan with must-have and optional runs

References

• Use references/experiment-checklists.md for the experiment-spec template, run checklist, and common failure modes.