Media Puppies
A/B test sample size calculator
Decide the sample size before the test, not after. This tells you whether the difference you are hoping for is even detectable at your traffic.
Set from your location. The maths is the same in any currency — only the labels change.
Relative. A 20% lift on 3.2% means getting to 3.84%.
95% is the convention. 90% is defensible for low-risk changes.
The chance of spotting a real effect. 80% is standard.
Including the control.
Runs entirely in your browser. Nothing you type is sent to us or stored anywhere.
Why you decide this before starting
Significance fluctuates while a test runs. If you check it every morning and stop the moment it turns green, you will declare winners that are not real — the statistics assume you looked once, at a pre-decided sample size. Fixing the sample size in advance is what makes the p-value mean what it claims to mean.
The uncomfortable arithmetic
Sample size scales roughly with the inverse square of the effect you are trying to detect. Halving the lift you want to spot quadruples the traffic you need. That is why small sites cannot test small changes, and why “we tested the button colour and it was inconclusive” is the expected result rather than a surprise.
- Detecting a 5% lift needs roughly sixteen times the traffic of detecting a 20% lift.
- Adding a third variant does not just split traffic — it extends the whole test.
- A low base conversion rate needs far more traffic than a high one.
What to do when the answer is “months”
- Test something bigger. A different offer or headline, not a reworded subheading.
- Move up the funnel. Testing on a higher-volume step — clicks rather than purchases — reaches significance sooner, provided the step genuinely predicts the outcome.
- Accept 90% confidence for changes that are cheap to reverse.
- Stop testing and decide. Below a certain traffic level, judgement plus a clear rationale beats a test that can never conclude.
When the test finishes, put the numbers into the significance calculator to see whether the result holds.
Questions
What is minimum detectable effect?+
The smallest improvement you care about finding. Set it to what would actually change your decision — if a 5% lift would not change anything you do, do not size the test for it.
Should I run for full weeks?+
Yes. Traffic and intent differ across the week, so stopping mid-week biases the result even when the sample size is reached.
Why does power matter?+
Power is the chance of detecting a real effect if one exists. At 80% power you will miss one real winner in five. Raising it costs more traffic.
Can I add variants mid-test?+
No. It resets the maths and invalidates the comparison. Start over.
Embed this calculator
Free to use on your own site. Copy this into your page — the attribution line is the only condition.
The paragraph below the iframe is the part that matters: an iframe's contents are served from our domain, so a link inside it is not a link from your page. The paragraph is. The small script just resizes the frame to fit your column — drop it if you prefer, the fixed height still works.
Other free tools
Want someone to run the campaigns behind these numbers?
We run Meta and Google ads for Indian businesses where one customer is worth real money. You will get a straight answer on whether paid works for you — including “not yet”.
Get a free lead plan →