A/B Test Significance Calculator (Z-Test)
Enter visitors and conversions for versions A and B to see whether the gap is statistically significant or just noise. This is the A/B test calculator for the finished test; it assumes you fixed the sample size in advance and does not plan how many visitors you need.
Updated
Report a problem
Formula
Two-proportion z-test with pA = xA / nA, pB = xB / nB and pooled rate p = (xA + xB) / (nA + nB):
SE = sqrt(p × (1 − p) × (1/nA + 1/nB))z = (pB − pA) / SEp-value = 2 × (1 − Φ(|z|))(two-tailed, Φ the standard normal CDF)95% CI of B − A = (pB − pA) ± 1.96 × sqrt(pA(1−pA)/nA + pB(1−pB)/nB)Relative uplift = (pB − pA) / pA
The result is significant when the p-value is below your alpha.
Assumptions
- Visitors were randomly and independently assigned, and each counted once.
- The sample size was fixed in advance. Peeking at results and stopping when p dips below alpha inflates false positives.
- Counts are large enough for the normal approximation (about 10 or more conversions and non-conversions per group).
- One comparison. Testing many variants or metrics needs a multiple-comparison correction.
Worked example
A: 500 of 10,000 (5.00%). B: 570 of 10,000 (5.70%). Uplift is 14%. Pooled p = 1,070 ÷ 20,000 = 0.0535, SE = 0.003183, z = 0.007 ÷ 0.003183 = 2.20, two-tailed p-value 0.0278. At alpha 0.05 B is significantly better; at 0.01 it is not. The 95% interval for B − A is 0.08 to 1.32 percentage points.
More examples
- Clear win: A converts 200 of 4,000 (5.0%) and B converts 260 of 4,000 (6.5%). The uplift is 30%, z = 2.88, p = 0.0040, and the 95% interval for B − A is 0.48 to 2.52 points.
- Looks big, is not proven: 30 of 1,000 against 45 of 1,000 is a 50% uplift, yet z = 1.77 and p = 0.0775, so it is not significant at 0.05. A large relative lift on little traffic is often noise.
How it differs
This page judges a finished test. To plan traffic use the sample size calculator; for a bare p-value from a z-score use the p-value calculator; to compare two rates of the same funnel step use it after the conversion rate calculator.
Frequently asked questions
How does the A/B test significance calculation work?
It runs a two-proportion z-test. With 500 of 10,000 converting in A and 570 of 10,000 in B, the pooled rate is 5.35%, the standard error is 0.00318, z = 0.007 ÷ 0.00318 = 2.20 and the two-tailed p-value is 0.0278.
What does the p-value mean?
It is the probability of seeing a difference at least this large if A and B truly converted equally. A p-value of 0.0278 is below alpha 0.05, so the result is called significant at 5%, but it is not the probability that B is better.
What assumptions does the test make?
Visitors were randomly and independently assigned to A or B, each visitor counted once, the sample size was fixed before you started, and counts are large enough for the normal approximation (roughly at least 10 conversions and 10 non-conversions in each group). For small counts use an exact test.
Why is peeking at results a problem?
Checking repeatedly and stopping the first time p drops below alpha inflates false positives well above the nominal 5%, because each look is another chance for random noise to look significant. Decide the sample size up front, or use a sequential method designed for repeated looks.
What is the confidence interval for the difference?
It is the 95% range for B − A in percentage points, using the unpooled standard error. For 5.0% versus 5.7% it is 0.08 to 1.32 points, so the true gain plausibly lies anywhere in that range. The interval is always 95%, whichever alpha you pick for the verdict.
How many visitors do I need for an A/B test?
A common rule of thumb for 80% power at 5% significance is n = 16 × p × (1 − p) ÷ d² per variant, where d is the absolute lift you want to detect. For a 5% baseline and a 1 percentage point lift that is 16 × 0.05 × 0.95 ÷ 0.0001 = 7,600 visitors per variant. Smaller lifts need far more traffic.
Is 90% or 95% confidence enough for an A/B test?
95% (alpha 0.05) is the usual default, accepting about one false positive in 20 tests. 90% accepts one in 10, which can be fine for a cheap, reversible change, while pricing or checkout changes deserve 95% or 99%. The 5.0% versus 5.7% example (p = 0.0278) passes at 95% but fails at 99%.
How do I calculate A/B test significance in Excel?
Work out the pooled rate p = (xA + xB) ÷ (nA + nB), then SE = SQRT(p × (1 − p) × (1/nA + 1/nB)), z = (pB − pA) ÷ SE and p-value = 2 × (1 − NORM.S.DIST(ABS(z), TRUE)). For 200 of 4,000 against 260 of 4,000 this gives z = 2.88 and p = 0.0040, matching the calculator.