An A/B test is conclusive when it has reached the sample size you decided before it started, has run for whole weeks, and shows a difference larger than the smallest effect you said you cared about. Most tests that get called "winners" meet none of those conditions. They were stopped the first time a dashboard showed a green number.
That habit produces a stream of changes that looked like wins and quietly made no difference, or made things worse. Here is how to run an A/B test whose result you can trust.
Decide the sample size before the A/B test starts
You need three inputs: your current conversion rate, the smallest improvement worth acting on, and the confidence you require, usually 95% significance with 80% power. Those determine how many visitors each variant needs. Small improvements on low conversion rates need large samples; there is no shortcut.
Our sample size calculator does the arithmetic. If the number it gives is more traffic than you will get in a month, test a bolder change, because a subtle one cannot be measured on your traffic.
Run whole weeks
Behaviour on weekdays and weekends differs, and so does traffic from different campaigns through the week. A test that runs from Tuesday to Friday measures Tuesday-to-Friday visitors. Run in multiples of seven days, at least two full weeks for most sites, even if the sample size arrives sooner.
The peeking trap
Significance tests assume you look once, at the end. If you check every day and stop the first time the result crosses 95%, the real chance of a false winner is far higher than 5%, because random fluctuations cross the line regularly early on. This is the single most common reason test results do not hold up after launch.
The fix is discipline: decide the sample and duration up front and do not act on interim numbers. If you must monitor, monitor for breakage, not for winners.
One primary metric
Decide in advance which metric decides the test: form submissions, purchases, qualified leads. Looking at ten metrics afterwards and picking the one that moved is another way to find a winner that is not there. Secondary metrics are for understanding, not deciding.
Reading the result
When the test reaches its planned size, check significance with a tool such as our significance calculator. A clear winner gets shipped. A flat result is also an answer: the change does not matter enough to measure, so choose on cost or simplicity and test something bolder next.
For lead generation pages, the changes most worth testing are usually the offer, the form length and the headline, not button colours. CRO for lead generation lists where the big effects usually are.
Questions people ask before hiring
How long should an A/B test run?
Until it reaches the sample size you calculated in advance, and for whole weeks, usually at least two. Stopping early because a dashboard shows significance produces false winners.
What sample size does an A/B test need?
It depends on your current conversion rate, the smallest improvement you care about and the confidence level. Low conversion rates and small effects need large samples, which a sample size calculator will show.
Why should I not stop a test as soon as it is significant?
Because checking repeatedly and stopping at the first significant result greatly increases the chance that the winner is random noise. Significance tests assume one look at the end.
What if the A/B test shows no difference?
That is a valid result. The change does not matter enough to measure on your traffic. Ship whichever is cheaper or simpler and test a bolder idea next.