Treatment A has a higher success rate than B for mild cases, and for severe cases too. Combine the two groups, though, and B comes out ahead. It sounds impossible, but it really happens — that is Simpson's paradox.
The data below comes from a well-known kidney stone treatment study (UK, 1986). Split into mild and severe cases, treatment A (open surgery) always does better — but the two groups hold very different numbers of patients, so combining them flips the result. Pull the data apart yourself and see why.
Mild only, severe only, or both combined — switch between the three views and compare the success rates of treatments A and B.
Drag the slider to change what share of treatment A's patients are severe cases. Each group's success rate stays exactly the same — yet changing only the mix flips the overall ranking.
An overall success rate is not the plain average of the two group rates — it is a weighted average, weighted by how many patients are in each group. Treatment A took far more severe cases, where success rates are low (75% of its 263 patients), while treatment B took far more mild cases, where success rates are high (270 patients, 77%). So A's overall average gets dragged down toward the severe-case rate, and B's gets pulled up toward the mild-case rate.
In other words, what flipped the ranking was not any difference in how good A and B are, but a difference in who each one treated. Statisticians call this the effect of a confounding variable — here, the hidden confounder is how severe each patient's case was.
The same thing showed up in UC Berkeley's 1973 graduate admissions data. Looking only at the overall rates, men were admitted more often than women, which raised claims of bias. Broken down by department, though, women were admitted at a similar or higher rate in most departments. The reason: women applied in greater numbers to popular departments that were competitive and admitted fewer applicants overall.
Cases like these are why it pays to make a habit of asking, whenever you see an overall number, which subgroups it should be broken into before you trust it.