Minimum Detectable Effect in A/B Testing: The Number That Decides If Your Test Will Work

You launched an A/B test, waited three weeks, and got an inconclusive result. Sound familiar? In most cases, the test was doomed before it started — because nobody set the minimum detectable effect (MDE) correctly.

MDE is the smallest improvement your test is designed to reliably detect. Get it wrong and you'll either waste weeks chasing noise or miss real wins that could have grown your business. Yet most teams skip this step entirely, treating sample size calculators like slot machines instead of planning tools.

This guide explains what MDE actually means, how to calculate it, and how to pick the right number for your next experiment — whether you're running 500 visitors a day or 50,000.

What Is Minimum Detectable Effect?

Minimum detectable effect is the smallest true difference between your control and variant that your A/B test has enough statistical power to detect. It's not a prediction of what lift you'll get. It's a statement about what your test is capable of measuring.

Think of it like a scale's resolution. A bathroom scale can detect a 0.5 kg difference but not a 10-gram difference. Similarly, an A/B test with 1,000 visitors per variant can detect a 5% relative lift in conversion rate, but not a 0.5% lift. MDE is the resolution of your experiment.

When you use a free A/B testing tool like PageDuel and plug numbers into a sample size calculator, MDE is one of the three inputs (alongside your baseline conversion rate and desired statistical power) that determines how long your test needs to run.

Why MDE Matters More Than You Think

Setting MDE too high means you'll only detect large changes — and most real conversion improvements are modest. Research from Gelman and Carlin found that underpowered tests inflate observed effect sizes by up to 8x. Your "30% winner" might actually be a 5% improvement that got lucky.

Setting MDE too low means your test needs an impractical sample size. Halving your MDE roughly quadruples the required sample size. A test that could have finished in two weeks now takes two months, burning your testing velocity and blocking other experiments in the queue.

This is one of the most common A/B testing mistakes teams make: they don't plan MDE at all, run underpowered tests, and then either call inconclusive results "no winner" or peek at results early and declare a premature winner.

How to Calculate MDE

The core formula for MDE in a two-sample proportions test is:

MDE = (Z_α + Z_β) × √(2 × p(1−p) / n)

Where:

• Z_α = 1.96 for 95% confidence (two-tailed)
• Z_β = 0.84 for 80% power
• p = your baseline conversion rate
• n = sample size per variant

You don't need to memorize this. PageDuel's sample size calculator handles the math. But understanding the relationships helps you make smarter tradeoffs:

• More traffic → smaller MDE. Double your sample size and you can detect effects ~30% smaller.
• Higher baseline conversion → smaller MDE. A page converting at 10% is easier to optimize than one at 1%.
• Higher power → larger required sample. Going from 80% to 95% power adds ~60% more sample.

How to Choose the Right MDE for Your Test

Forget the generic "just use 5%" advice. Here's a practical framework that actually works:

Step 1: Start With Business Impact

Ask: "What's the smallest improvement worth acting on?" If your landing page gets 10,000 visitors per month and converts at 3%, a 10% relative lift (3.0% → 3.3%) means 30 extra conversions per month. Is that meaningful for your business? If each conversion is worth $100, that's $3,000/month — probably worth detecting. If each conversion is worth $2, maybe not.

Step 2: Check Your Traffic Budget

Use a sample size calculator to see how many visitors you need at your target MDE. If the answer is more traffic than you'll get in 4-6 weeks, your MDE is too ambitious. Increase it until the test fits a realistic timeline.

For sites with very low traffic, you may need an MDE of 20-30% — and that's fine. It means you're only looking for big, obvious improvements, which are exactly the kind of wins early-stage sites should chase first.

Step 3: Consider the Change You're Testing

Not all changes are created equal. A headline rewrite might produce a 5-15% lift. Changing a button color rarely moves the needle more than 2-3%. Match your MDE to the magnitude of change you're making:

• Major redesigns (new layout, new offer): MDE of 5-10%
• Copy and messaging changes: MDE of 10-20%
• Minor visual tweaks (color, spacing): MDE of 15-30% (or skip the test)

Step 4: Lock It In Before You Start

Write down your MDE, required sample size, and expected test duration before launching. This is your pre-test commitment — it prevents you from peeking at results early, stopping too soon, or moving the goalposts when the data doesn't look exciting.

Common MDE Mistakes and How to Avoid Them

Mistake 1: Picking MDE After Seeing Results

If you set your MDE based on what the test showed, you're not doing science — you're doing confirmation bias. Always define MDE before launch.

Mistake 2: Using the Same MDE for Every Test

A checkout page test and a blog sidebar test have wildly different traffic volumes and business impact. One size doesn't fit all.

Mistake 3: Ignoring Practical Significance

A statistically significant 0.1% lift is meaningless for most businesses. MDE should reflect practical significance — the smallest change worth the effort of implementing and maintaining the winning variant.

Mistake 4: Forgetting About Power

Most calculators default to 80% statistical power, which means a 20% chance of missing a real effect. For high-stakes tests (pricing page, checkout flow), consider bumping power to 90% — you'll need more sample but you'll catch more real winners.

MDE in Practice: A Quick Example

Say you're testing a new CTA on a landing page that converts at 4% with 2,000 daily visitors. You decide a 15% relative lift (4.0% → 4.6%) is the minimum worth pursuing. Plugging into a calculator:

• Baseline: 4%
• MDE: 15% relative (0.6 percentage points absolute)
• Power: 80%, Significance: 95%
• Required sample: ~8,700 per variant → ~17,400 total
• At 2,000 visitors/day with a 50/50 split: ~9 days

That's a realistic, well-scoped test. If you had demanded a 5% MDE instead, you'd need ~78,000 visitors per variant — nearly three months of traffic. Not worth it for a CTA test.

Getting Started With MDE-Driven Testing

PageDuel makes it easy to plan and run properly powered A/B tests. Use the built-in sample size calculator to define your MDE before launching, then let the platform handle traffic splitting, variant assignment, and statistical analysis. It's free to start — no credit card, no minimum traffic requirements.

The best A/B testing programs aren't the ones that run the most tests. They're the ones that run the right tests — properly powered, well-scoped experiments designed to detect the changes that matter. MDE is where that discipline starts.

Related Reading

• How to Run an A/B Test: A Complete Step-by-Step Guide
• A/B Testing Mistakes to Avoid: 9 Pitfalls That Kill Your Experiments
• A/B Testing With No Traffic: What to Do When Your Site Is Too Small
• Bayesian vs Frequentist A/B Testing: Which Method Should You Use?
• CUPED in A/B Testing: How to Get Faster Results Without More Traffic