Media Puppies
A/B test significance calculator
Variant B is up 18% and you want to call it. This tells you whether that difference would survive being run again.
Runs entirely in your browser. Nothing you type is sent to us or stored anywhere.
What this actually calculates
A two-proportion z-test. It asks one question: if the two variants were genuinely identical, how likely is it that you would see a gap this large purely by chance? That likelihood is the p-value. Below 0.05 — a one-in-twenty chance — is the usual threshold for calling a winner.
The confidence figure is simply the other side of the same coin: a p-value of 0.03 is 97% confidence.
The mistakes that make tests lie
- Stopping the moment it goes green. Significance fluctuates during a test. If you check daily and stop at the first green reading, you will declare winners that are not real. Decide the sample size before you start, or at minimum run full weeks.
- Running less than seven days. Behaviour differs on weekdays and weekends. A three-day test measures Tuesday, not your audience.
- Testing something trivial. Button colour rarely produces a difference large enough to detect without enormous traffic. Test the offer, the headline, the form length.
- Running several tests at once on the same traffic. They contaminate each other and you cannot attribute the result.
- Ignoring what happens after the conversion. A variant that doubles leads and halves lead quality is a loss. Measure the outcome that matters, not the one that is easy.
When you simply do not have the traffic
Most small accounts cannot reach significance on a 10% difference in any reasonable timeframe. That is not a reason to skip testing — it is a reason to test bigger swings. A completely different angle can produce a difference large enough to detect in a fortnight; a reworded subheading cannot.
Questions
Is 95% confidence the right threshold?+
It is the convention, not a law. For a low-risk change, 90% is a reasonable bar. For anything expensive or hard to reverse, wait for 95% or higher.
How long should a test run?+
At least one full week, ideally two, and until you reach your pre-decided sample size. Full weeks matter because weekday and weekend behaviour differ.
What if neither variant wins?+
That is a genuine result. It means the change was not big enough to matter. Keep the simpler version and test something more substantial.
Can I use this for Meta ad tests?+
Yes, for comparing two ads on impressions and conversions. Note that Meta does not split traffic evenly by default, so its own A/B test tool is cleaner for a formal test.
Embed this calculator
Free to use on your own site. Copy this into your page — the attribution line is the only condition.
The paragraph below the iframe is the part that matters: an iframe's contents are served from our domain, so a link inside it is not a link from your page. The paragraph is. The small script just resizes the frame to fit your column — drop it if you prefer, the fixed height still works.
Other free tools
Want someone to run the campaigns behind these numbers?
We run Meta and Google ads for Indian businesses where one customer is worth real money. You will get a straight answer on whether paid works for you — including “not yet”.
Get a free lead plan →