Sampling and reading the results — Examples

Example 1 — Who was actually in the number

A survey reports a satisfaction figure with no denominator. Reconstruct the three groups behind it, then write the sentence the result actually supports. The figures below are illustrative.

Reported: "90% of users are satisfied with checkout. (n = 40)"

Solution

Illustrative figures
Wanted:    all 4,120 people who started checkout in March
Reached:   1,880 who had an email and had not unsubscribed
Answered:  40  (2% of reached, 1% of wanted)
Missing:   anyone who abandoned before entering an email

Supported sentence: "Of 40 self-selected respondents, 90% of those who
completed checkout reported satisfaction; people who abandoned before
entering an email were never asked."

The headline reads as a statement about all users; the denominator shows it is a statement about forty volunteers, drawn in the direction most likely to be satisfied. The narrower sentence keeps the number honest. The usual mistake is quoting a percentage with no count or response rate, which lets 90% of forty wear the authority of a statistic.

Example 2 — Two segments, before the headline

The headline number below is quoted on a slide before anyone looks at who answered. Segment it by first-time and returning customers, then say which group is over-represented among respondents. The figures are illustrative.

Reported: "68% found reordering easy (n = 212, 11% response rate)."
Raw split: first-time 34 respondents, returning 178 respondents

Solution

Illustrative figures
First-time customers (34):   62% found it easy
Returning customers (178):   70% found it easy
Response mix: returning customers are over-represented (178 of 212)

Supported claim: the score mostly reflects returning customers, who
already know the flow; first-time customers answer in smaller numbers
and report a lower score.

Splitting before quoting the total shows that the confident answer comes largely from the group that already knows the flow, and from a response rate of 11%. Segmenting also pairs the claim with the mix of who replied. The usual mistake is quoting the total when the segments disagree, so an average of two different experiences is reported as one.