Ab Test Design

Decision Classification

• MECHANICAL: Sample size calculation, statistical significance thresholds, guardrail metric selection for known risks
• TASTE: Which design question to test first, minimum detectable effect size (what improvement is worth shipping?), test duration vs speed of learning trade-off
• USER_CHALLENGE: Choosing between conflicting test results and business goals, testing changes that affect revenue, ending tests early

Safety Mechanisms

• One variable per test — multi-variable tests produce uninterpretable results
• Decision rules written before test starts — no moving goalposts
• Guardrail metrics are mandatory — every test must define what must not regress
• Minimum test duration: 1 full business cycle (typically 7 days) to avoid day-of-week bias
• Kill criteria defined upfront — automatic rollback if guardrail drops beyond threshold
• Never test on 100% of traffic — always hold back a control segment

Outputs & Persistence

• Experiment design document: hypothesis, metrics, variants, sample size, rollout plan, decision rules
• Variant specifications for implementation (can feed into /design-html or /design-shotgun)
• Results analysis template (pre-formatted for post-experiment reporting)
• Stored as Paperclip documents; experiment learnings logged via /learn regardless of outcome

Invocation

Triggered by UX Designer when a design decision has no clear winner and data can resolve it. Also invoked by CEO for product strategy validation or by CTO when evaluating competing technical approaches with user-facing impact.