Understanding significance without a statistics lecture

Significance is the most frequently misquoted term in A/B testing. It does not mean “the variation is better”, and certainly not “the variation is 18% better”. It answers a narrower question, and that is still useful.

What 95% really says

A significant result at the 95% level means: if there were truly no difference, you would still see a measured difference this large in at most 5 out of 100 tests. It is a statement about the method, not about your specific variation.

That leads to something uncomfortable: evaluating twenty goals at once finds a “significant” winner by chance alone, on average. That is why you fix one primary goal before launch, the rest is context.

The interval says more than the p-value

“Significant” is a yes/no. A confidence interval, by contrast, tells you how large the difference plausibly is. A result of “+6 points, interval +3 to +9” is a solid improvement. “+6 points, interval −1 to +13” has the same average, and is practically worthless.

Rule of thumb: look at the interval first. If zero is inside it, you have no result, no matter how nice the average looks. If the interval is very wide, you need more data, not more interpretation.

Four signs of a false winner

• The test was stopped as soon as it looked good, not when the planned sample was reached.
• The traffic split deviates clearly from the planned one, then assignment is not working.
• The gain comes from a single day or a single segment.
• The primary goal was swapped after the test because another one looked better.

Dealing with “no difference”

About half of all cleanly run tests end without a clear winner. That is not a failed test but an answered question: this change moves nothing, so you do not have to build it, and you can test the next idea.

That only becomes valuable if you write it down. Otherwise somebody tests the same idea again in two years.