A/B Testing

What is it?

A/B testing (also called split testing) is a controlled experiment in which two or more variants of a design element are shown to different segments of users simultaneously, and the variant that produces the better outcome on a defined metric is identified with statistical confidence. Unlike qualitative methods, A/B testing does not explain why one variant works better — it only reveals which one does. It is most powerful when combined with qualitative research that generated the hypothesis being tested.

Why it matters

Opinions are free and plentiful. A/B test results are evidence. In teams with strong opinions and stakeholders who override design decisions on instinct, A/B testing is the authoritative mechanism for resolving disagreements. It also prevents "local optimisation" — the trap of making a change that seems logical but actually reduces the metric that matters. Amazon, Netflix, and Booking.com run thousands of A/B tests simultaneously, making A/B testing not just a research method but a product development philosophy. Used correctly, it turns every release into a learning opportunity.

Best Practices

• Start with a hypothesis: "If we change X to Y, we expect to see Z because users currently experience [observed problem]." Never test without a directional hypothesis grounded in data.
• Test one variable at a time. Changing headline, colour, and CTA position simultaneously makes it impossible to know which change drove the result.
• Use multivariate testing only when you have very high traffic and need to test variable interactions — it requires significantly more traffic to reach significance.
• Calculate required sample size before starting, using a power analysis. Running a test and stopping when you see a result you like is p-hacking and produces false conclusions.
• Run tests for full business cycles (at minimum one week) to account for day-of-week variation in user behaviour.
• Define your primary metric before the test — the metric you will make your decision on. Secondary metrics can inform but should not override the primary.
• Set a significance threshold before testing (standard is 95% confidence / p < 0.05). Do not lower the bar when results are inconclusive.
• Watch for novelty effects: a new design often outperforms initially because of curiosity, then regresses. Run tests long enough to see stable behaviour.
• Document every test, hypothesis, result, and conclusion. A/B test history is one of the most valuable institutional knowledge assets a product team can build.
• Do not run too many tests on the same users simultaneously — overlapping tests corrupt results.

Common Mistakes

• Stopping a test early when results look good — this is the most common cause of false-positive A/B test conclusions.
• Testing without a hypothesis — "let's try a red button and see what happens" produces results you cannot learn from or apply.
• Insufficient traffic or sample size — declaring a winner with 50 users per variant is statistically meaningless.
• Optimising for proxy metrics (click-through rate) rather than business metrics (conversion, retention, revenue per user).
• Running A/B tests on low-impact elements (button border radius) when high-impact elements (headline, pricing, CTA copy) have never been tested.
• Treating inconclusive results as failures rather than information — a null result tells you the change doesn't matter, which is valuable.
• Ignoring segment-level results — a test may show no overall winner while hiding a strong positive result in a specific user segment.

Checklist

Research & Theory

Real-World Examples