Researchstack Experiment Design

Produce an experiment matrix with columns:

• claim,
• metric,
• workload or dataset,
• baseline,
• ablation,
• expected outcome,
• failure interpretation.

Default rigor checks:

• fair baseline tuning and budget disclosure,
• scale sensitivity,
• compute or hardware cost,
• robustness to parameter changes,
• tail or worst-case behavior if relevant,
• negative results worth reporting.

Flag common paper-killing problems:

• main claim depends on one cherry-picked setup,
• ablations do not isolate causal factors,
• systems speedups ignore resource cost,
• ML comparisons use stale or weak baselines,
• networking evaluation omits adverse or dynamic conditions.

When the session establishes a durable evaluation rule, baseline policy, or failed direction that should not be relearned next time, propose a project-memory entry for it.