The test said you won. The math says maybe not.
Quick answer: Most A/B tests that get called early, run on too few users, or get "peeked at" before completion produce results that look statistically significant but are not reliable. The three biggest causes are underpowered sample sizes, early stopping, and testing too many variants at once. Fixing all three requires calculating sample size before the test starts, not after.
A product manager on my old team once shipped a new checkout button because it "won" an A/B test after four days. Conversion was up 11%. Champagne emojis in Slack. Three weeks later, the lift had vanished. The button hadn't changed anything. The test had.
This happens constantly, and almost nobody catches it, because the test report looked completely normal. Green checkmark. Percentage lift. A p-value under 0.05. Everything a business stakeholder is trained to trust.
The problem was never the button. The problem was the math underneath the test, and it is one of the most common, least discussed failure points in modern product and marketing analytics.
What "Statistically Significant" Is Supposed to Mean in an A/B Test
An A/B test compares two groups (A and B) and asks a simple question: is the difference we see real, or could it plausibly be random noise?
To answer that honestly, the test needs enough data to detect a real difference if one exists. This required amount of data is called statistical power, and it depends on three things: how big a difference you actually care about detecting, how much natural variation exists in your metric, and how confident you want to be in the result.
Most teams skip this calculation entirely. They run the test until it "feels done," check the p-value, and move on. That is where things go wrong.
The Three Silent Sample Size Sins
1. Underpowered Tests
If your sample size is too small for the effect you're trying to detect, your test has almost no chance of reliably finding a real difference, even if one exists. Worse, when a small, underpowered test does show a "significant" result, that result is more likely to be an overestimate of the true effect, sometimes wildly so. This is why so many "huge wins" from small tests quietly shrink or disappear once the feature rolls out to everyone.
The fix is simple in concept and skipped in practice: calculate the required sample size before the test starts, based on the smallest effect size that would actually be worth acting on. Free sample size calculators exist for exactly this. Using one takes about two minutes and prevents most of this problem outright.
2. Early Stopping, Also Called Peeking
This is the checkout button story. Checking your test results every day and stopping as soon as you see a "significant" result dramatically increases your false positive rate. Random noise fluctuates. If you keep checking, you will eventually catch a moment where the noise happens to look like a win, purely by chance, and you'll stop right there and call it real.
Studies on this exact behavior have shown that repeatedly peeking at results and stopping at the first sign of significance can push your true false positive rate well above the 5% you think you're working with, sometimes several times higher. The fix is to decide your sample size and test duration in advance, and not look at the result as a verdict until you hit that number.
3. Testing Too Many Variants and Reporting Only the Winner
If you test five button colors at once and one of them "wins" at p = 0.04, that is a different situation than testing one button color and getting the same p-value. With five separate comparisons, the odds that at least one of them shows a false positive by chance are much higher than 5%. This is called the multiple comparisons problem, and it is one reason so many "winning" variants fail to replicate when retested on their own.
The fix involves either adjusting your significance threshold for the number of comparisons you're making, or simply being honest in your reporting about how many variants were tested before one "won."
A Simple Pre-Test Checklist
Before you launch your next A/B test, answer these four questions:
- What is the smallest lift that would actually be worth shipping this feature for?
- Based on that number and your current conversion rate, what sample size do you need? (Use a sample size calculator. Do not skip this step.)
- Will you commit to that sample size and duration before checking results, instead of peeking daily?
- How many variants are being tested at once, and does your significance threshold account for that?
If you can answer all four before you launch, your next test result will be dramatically more trustworthy than the last one.
Why This Matters More Than It Looks
None of this is about doing more math for its own sake. It's about the cost of being wrong. A team that ships features based on noise doesn't just waste engineering time once. It slowly trains itself to trust a testing process that quietly produces false positives, and that erodes decision quality across the entire product, one small "win" at a time.
The good news is that none of the fixes above require an advanced statistics degree. They require running one calculation before the test instead of after, and resisting the very human urge to check results early and celebrate the first green checkmark you see.
Frequently Asked Questions
How big should my A/B test sample size be?
It depends on your baseline conversion rate and the smallest lift you care about detecting. Use a sample size calculator and plug in both numbers before you launch the test, rather than guessing.
Why did my A/B test result not replicate after shipping?
The most common reasons are an underpowered original test, stopping the test early once it looked significant, or testing multiple variants and only reporting the one that won. All three inflate the apparent effect size beyond what is real.
Is checking A/B test results daily a problem?
Yes, if you stop the test as soon as you see significance. This practice, known as peeking, substantially increases your real false positive rate above the 5% most people assume they're working with.
What is statistical power in an A/B test?
Statistical power is the probability that your test will detect a real effect, if one truly exists, given your sample size and the size of the effect. Low power means real effects often go undetected, and any significant result that does appear is more likely to be exaggerated.
0 Comments