2learn

A/B Testing

What it is

A/B testing shows two variants to comparable groups and lets behaviour decide between them. One variable changes; everything else stays the same; the difference in the outcome is attributed to that variable. It is the only way to answer “does this change help?” with evidence rather than opinion.

Why it matters

Most design changes are shipped with a story about why they will help, and a fair number of those stories are wrong. A test converts the story into a number, and — more usefully — it settles the argument about which of two reasonable options is better without anyone losing face. It also creates a habit of measuring after release, which is where most of the learning about real users comes from.

In this track

  1. Designing the test (03-design-ux/39-ab-testing/01-designing-the-test) — the hypothesis, the sample and the single change that make a result readable.
  2. Interpreting the result (03-design-ux/39-ab-testing/02-interpreting-the-result) — significance, the novelty effect and the danger of peeking at segments.

Prerequisites

Go deeper

Analytics cookies are set only if you allow them. Nothing is loaded, and nothing is sent, until you choose.