A/B Test Plan — Hypothesis, Sample Size and Stop Rules

You're a product experimentation lead who stops teams from peeking at results and calling winners too early. Plan my A/B test. The change and why we think it will help: {{change_and_hypothesis}} Primary metric, its current baseline, and its standard deviation if it's an average rather than a rate: {{primary_metric_baseline}} The smallest improvement worth detecting (minimum detectable effect); say whether it's relative, such as 10 percent (4.0 to 4.4 percent), or absolute, such as 0.4 percentage points: {{minimum_effect}} Eligible users or sessions per day: {{daily_traffic}} Split: {{{split: 50/50, 80/20, 90/10}}} **Deliver:** **Hypothesis:** "If we [change], then [metric] will [move], because [reason]". **Metrics:** the primary metric, 2 or 3 guardrail metrics, how each is defined, and the unit we randomize on (user, session or account). **Sample size and duration:** restate the effect as both relative and absolute, then estimate the units needed per variant for a two-sided test at 95 percent confidence and 80 percent power, with the formula and working shown. For an uneven split, size the smaller group and base the duration on its share of traffic. Convert to days at my traffic, rounded up to whole weeks, and note that returning users count once, so unique users grow more slowly than daily traffic times days. If it would take more than about 6 weeks, say so and suggest a larger effect, a more sensitive metric or a different test. **Stop rules:** when we'll look at results, what counts as a win, loss or no difference, and when we'd stop early for harm. **Risks:** things that could spoil the result, such as novelty effects, seasonality or users in both groups. **Decision plan:** what we'll do for each possible outcome, agreed before we start. Rules: Show the calculation and say it's an estimate; tell me to confirm it with my testing tool's calculator. Don't recommend stopping as soon as a result looks significant. Don't test dark patterns, misleading prices or anything that could harm users, and check with legal or privacy before testing prices or personal data use.