March 20, 2026

How to Run an A/B Test: A Complete Step-by-Step Guide (2026)

A practical, no-fluff guide to running your first A/B test — from hypothesis to statistical significance — so you can stop guessing and start growing.

A/B testing sounds complicated until you've done it once. Then it feels like cheating — because instead of arguing about whether a red button or a green button converts better, you just find out. You split your traffic, collect data, and let the numbers tell you what to do.

The problem is most guides on how to run an A/B test either go way too deep into statistics or treat you like you need a PhD to get started. This guide cuts through both. You'll learn exactly how to run a real A/B test — from picking what to test to declaring a winner — using plain language and a free tool that doesn't require a credit card.

Step 1: Pick One Thing to Test

The number one mistake beginners make is testing too many things at once. Change the headline and the button color and the image? Now you have no idea what moved the needle. A/B testing only works when you isolate one variable.

Good starting points for your first test:

Headlines — The single biggest lever on most pages. A stronger value prop can double conversions on its own.
CTA button text — "Get Started" vs. "Try It Free" vs. "Start My Test" — small changes, big swings.
Hero image or video — Does showing your product convert better than a lifestyle shot?
Above-the-fold layout — More whitespace, fewer distractions, different visual hierarchy.
Social proof placement — Logos above the fold vs. below? Testimonials near the CTA?

Pick one. Just one. The goal of your first test isn't to optimize everything — it's to build the habit and learn the process.

Step 2: Form a Hypothesis

Before you build anything, write down what you think will happen and why. This is called a hypothesis, and it matters more than most people realize.

A good A/B test hypothesis has three parts:

What you're changing: "We're changing the CTA button text from 'Sign Up' to 'Start Free Trial'"
What you expect to happen: "We expect click-through rate on the button to increase"
Why you believe this: "Because 'Start Free Trial' is more specific about what happens next and reduces perceived commitment"

Writing the "why" forces you to think like a scientist, not a guesser. It also helps you learn faster — because even if you're wrong, you understand why the data disagreed with your assumption.

Step 3: Define Your Success Metric

What does "winning" look like? Be specific before the test starts, not after.

Your primary metric should be directly tied to the thing you're changing. If you're testing a headline, your metric might be time on page or scroll depth. If you're testing a CTA button, it's click-through rate or sign-up rate. If you're testing a pricing page layout, it's plan selection or checkout initiation.

Avoid the trap of switching metrics after the test runs. If you started measuring sign-up rate and your variant lost on sign-ups but "won" on time on page — that's not a win. Stick to what you defined upfront.

Step 4: Calculate Your Required Sample Size

This is where most people skip a step and regret it later. Running a test for two days and declaring a winner because you saw 12 conversions vs. 9 conversions is not A/B testing — it's wishful thinking.

Statistical significance is the measure of confidence that your results aren't due to random chance. The standard threshold is 95% confidence. To hit that, you need enough traffic in both variants to make the comparison meaningful.

As a rough guide:

If your page converts at 2% and you want to detect a 20% improvement (to 2.4%), you'll need roughly 15,000–20,000 visitors per variant
If your page converts at 5% and you want to detect the same lift, you need closer to 5,000–7,000 per variant
Higher baseline conversion rate = smaller sample size needed

For low-traffic sites, this math can feel brutal. But there are strategies for running tests without needing enterprise-level traffic — including focusing on higher-converting pages and testing bigger changes rather than incremental tweaks.

Step 5: Set Up Your Test With a Tool

You need an A/B testing tool to split traffic randomly between your variants. Random assignment is critical — if visitors who come on Mondays always see variant A and Tuesday visitors always see variant B, your data is contaminated by day-of-week effects.

Tools like PageDuel handle traffic splitting automatically and for free. You create your two versions, set your goal, and PageDuel randomly assigns each visitor to a variant — then tracks conversions on both. No coding required, no monthly fee.

Paid alternatives like Optimizely and VWO can cost thousands per month and are overkill for most teams just getting started. The tooling isn't what makes A/B testing work — the methodology is.

Step 6: Run the Test Long Enough

Two common mistakes kill otherwise good tests:

Stopping too early — You see variant B winning at 68% confidence after day 3 and call it done. But confidence at 68% means roughly 1-in-3 chance your "winner" is just noise. You need to hit your target sample size regardless of what the early numbers say.

Running too long — Tests that run for months can be contaminated by seasonal shifts, product changes, or external events. Two to four weeks is typically ideal. If you don't have enough traffic to reach significance in that window, consider testing bigger changes or running on higher-traffic pages first.

A good rule: set a runtime when you start the test (based on your sample size calculation and expected daily traffic), then don't touch it until that time is up.

Step 7: Read the Results Correctly

When your test ends, you're looking at three possible outcomes:

Variant wins (≥95% confidence): Deploy the winner. Document what you learned. Move on to the next test.
Control wins (≥95% confidence): The original was better. Document why your hypothesis was wrong — that's valuable. Move on.
Inconclusive (below 95% confidence): The difference between variants is too small to be sure it's real. This doesn't mean the test failed — it means this variable probably isn't a major lever. Find something bigger to test.

Most A/B tests are inconclusive. That's normal. A/B testing is a volume game — the teams with the most tests win, not the teams who nail it on the first try.

Step 8: Document and Iterate

Every test — winner, loser, or inconclusive — teaches you something about your users. Build a simple log with: what you tested, your hypothesis, the result, confidence level, and what you'll test next.

This log becomes the institutional knowledge of your optimization program. Six months of testing means you know which levers don't move your users, which ones do, and why. That's compounding knowledge that no competitor can copy from a blog post.

The best teams run 1–2 tests per week. PageDuel is built to support exactly that kind of continuous testing cadence — set up a test in minutes, let it run, review results, repeat. No developer required, no annual contract, and no paywall between you and your data.

Common A/B Testing Mistakes to Avoid

Testing without enough traffic — Statistical significance requires real numbers. Don't shortcut it.
Changing the test mid-run — Once the test starts, hands off. Changes contaminate results.
Only testing button colors — Headlines, offers, and layouts typically move the needle far more than cosmetic tweaks.
Ignoring mobile vs. desktop splits — Your winning variant on desktop might be losing on mobile. Always segment your results.
Not having a next test ready — The biggest mistake is stopping after one test. The compounding effect comes from continuous iteration.

If you're just getting started with A/B testing, the fastest path to results is a free tool, a focused hypothesis, and the patience to let data accumulate before you call a winner. Start your first test on PageDuel — it takes about five minutes to set up, and you'll learn something real about your users within two weeks.