Experiment Designer

Use:

python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute

4. Prioritize experiments with ICE

• Impact: potential upside
• Confidence: evidence quality
• Ease: cost/speed/complexity

ICE Score = (Impact * Confidence * Ease) / 10

5. Launch with stopping rules

• Decide fixed sample size or fixed duration in advance
• Avoid repeated peeking without proper method
• Monitor guardrails continuously

6. Interpret results

• Statistical significance is not business significance
• Compare point estimate + confidence interval to decision threshold
• Investigate novelty effects and segment heterogeneity

Hypothesis Quality Checklist

• Contains explicit intervention and audience
• Specifies measurable metric change
• States plausible causal reason
• Includes expected minimum effect
• Defines failure condition

Common Experiment Pitfalls

• Underpowered tests leading to false negatives
• Running too many simultaneous changes without isolation
• Changing targeting or implementation mid-test
• Stopping early on random spikes
• Ignoring sample ratio mismatch and instrumentation drift
• Declaring success from p-value without effect-size context

Statistical Interpretation Guardrails

• p-value < alpha indicates evidence against null, not guaranteed truth.
• Confidence interval crossing zero/no-effect means uncertain directional claim.
• Wide intervals imply low precision even when significant.
• Use practical significance thresholds tied to business impact.

See:

• references/experiment-playbook.md
• references/statistics-reference.md

Tooling

scripts/sample_size_calculator.py

Computes required sample size (per variant and total) from:

• baseline rate
• MDE (absolute or relative)
• significance level (alpha)
• statistical power

Example:

python3 scripts/sample_size_calculator.py \
  --baseline-rate 0.10 \
  --mde 0.015 \
  --mde-type absolute \
  --alpha 0.05 \
  --power 0.8