Free A/B testing tool
Statistical Power Calculator
Find out how likely your A/B test is to detect a real lift. Enter your baseline conversion rate, the lift you care about and the visitors you expect per variant. The calculator shows the test's power and how many visitors you need for 80% power.
Live calculator
Calculate the power of your A/B test
Statistical power
39.0%
Below 80%: a real lift of this size would often be missed.
- Variant rate you want to detect
- 3.60%
- Visitors per variant for 80% power
- 13,911
Two-sided z-test for two conversion rates, equal traffic split. Power is the chance the test shows a significant result if the lift is real.
What statistical power means for an A/B test
Every A/B test can go wrong in two ways. It can call a winner when there is no real difference (a false positive). The significance level controls that risk. Or it can miss a real difference (a false negative). Power controls that second risk: it is the chance the test finds a lift that really exists.
A test with low power is not just slow. When it ends without a winner, you cannot tell whether the change did nothing or the test was too small to see it.
How power is calculated
For two conversion rates p1 (control) and p2 (variant) with n visitors in each, power ≈ Φ(|p2 - p1| × √n / √(p1(1 - p1) + p2(1 - p2)) - z), where z is the two-sided critical value for your significance level (1.96 at 95%) and Φ is the standard normal distribution.
Example: a 3% baseline, a 20% relative lift (3% to 3.6%) and 5,000 visitors per variant give a power of about 39%. That test would miss a real 20% lift more often than it finds it. For 80% power you need about 13,911 visitors per variant.
This is the same formula the sample size calculator solves the other way round: there you choose the power and get the visitors. Use the test duration calculator to turn visitors into days.
What changes the power
- More visitors per variant: power goes up.
- A bigger lift to detect: power goes up. Small lifts need far more traffic.
- A lower baseline conversion rate: the same relative lift is harder to detect.
- A stricter significance level (99% instead of 95%): power goes down.
FAQ
What is statistical power?
Power is the probability that a test detects an effect when the effect is real. With 80% power, a real lift of the size you entered shows up as significant in about 8 out of 10 tests. In the other 2, the test misses it (a false negative, or type II error).
What is a good statistical power?
80% is the most common minimum. Some teams use 90% for important decisions. Higher power needs more visitors.
How do I increase the power of an A/B test?
Run the test on more visitors, test a bigger change (a larger lift is easier to detect), or accept a lower significance level. Splitting traffic evenly between two variants also gives the most power for a given total.
Plan it here, run it in PageDuel
PageDuel runs A/B tests with a visual editor and tracks conversions and revenue per variant, so you know when a test is really done.
14-day free trial · No credit card required