D. Statistical Analysis · Two-Variable Data Analysis and Validity

Hidden Variables, Simpson's Paradox and Misleading Summaries

D2.2 D2.5About 70 minutes

Learning goals

I can explain how a lurking variable can create, hide or reverse a relationship.
I can recognize and explain Simpson's paradox in tables and scatter plots.
I can explain why correlations based on averages or groups can overstate relationships for individuals.
I can identify two-variable summaries that misrepresent data and suggest a fairer summary.

Words to know

Simpson's paradox
A trend that appears in every group of data but disappears or reverses when the groups are combined.

aggregated data
Data combined into group totals or averages (for example, one average per province) rather than individual values.

ecological fallacy
The mistake of assuming that a relationship found for groups (averages) holds for the individuals in those groups.

stratify
To split data into subgroups (strata) by a third variable and analyse each subgroup separately.

The better goalie has the worse save percentage?

Two goalies on a junior team each faced 400 shots this season. Goalie A saved 75% of them and Goalie B saved only about 49%. Easy decision: A is better, right?

Now split the shots into easy (long-range) and hard (breakaways and rebounds). Goalie B has the higher save percentage for both kinds of shots. How can that be?

	Easy shots saved	Hard shots saved	All shots saved
Goalie A	270 of 300 (90%)	30 of 100 (30%)	300 of 400 (75%)
Goalie B	92 of 100 (92%)	105 of 300 (35%)	197 of 400 (49.25%)

Save records for two goalies (made-up data).

Goalie B wins in each type of shot, yet loses overall. The overall bars hide the fact that B faced three times as many hard shots.

Simpson's paradox

This reversal is called Simpson's paradox. It happens when:

a lurking variable (here, shot difficulty) strongly affects the response (save rate), and
the groups being compared have very different mixes of that variable (A faced mostly easy shots; B faced mostly hard ones).

The overall percentage is a weighted average of the group percentages. A's overall 75% is weighted toward the easy 90%; B's overall 49.25% is weighted toward the hard 35%.

The fix is to stratify: compare within groups that are alike on the lurking variable. Here the fair comparison is shot type by shot type, and Goalie B is better.

Worked example

Example 1: Checking the weighted averages

Show that Goalie B's overall save percentage is a weighted average of 92% and 35%.

B faced 100 easy shots and 300 hard shots, so the weights are 100/400 = 0.25 and 300/400 = 0.75.
Weighted average: 0.25(92%) + 0.75(35%) = 23% + 26.25% = 49.25% ✓.
For A: 0.75(90%) + 0.25(30%) = 67.5% + 7.5% = 75% ✓.

The overall rates depend as much on the mix of shots as on skill. That is why the comparison reverses.

The same paradox in a scatter plot

A board gives a tutoring app to students in two schools. School P, where marks were lower, was asked to use it a lot; School Q used it less. Within each school, more app time goes with higher marks. But put the schools together and the trend points down!

Within each school (solid lines) the relationship is strongly positive. Combined (dashed line), it looks negative, because the school with more app use started with lower marks.

Worked example

Example 2: Which conclusion is right?

For School P alone, r ≈ 0.94; for School Q alone, r ≈ 0.95; for both schools combined, r ≈ −0.51. A trustee says the app lowers marks. Respond.

The combined data mix two different groups. School is a lurking variable: it affects both app use (P was told to use it more) and marks (P had lower marks to begin with).
Within each school, more app use goes with higher marks.
The combined negative r reflects the difference between the schools, not the effect of the app.

The trustee's conclusion is not supported. Analysed school by school, app use is positively related to marks. (Even so, this is observational data; an experiment with random assignment would be needed to show the app causes improvement.)

Other ways two-variable summaries mislead

Correlations of averages. If you compute one average per province, the averages smooth away individual differences, so r for the 10 provincial averages is often much larger than r for the individual people. Concluding that the relationship is just as strong for individuals is the ecological fallacy.
Weak correlations, strong headlines. r = 0.2 from a big study may be real, but it means the line explains only 4% of the variation. A headline that says "X determines Y" overstates it.
Combining different groups (adults with children, two different years, two countries) can create a correlation that does not exist within any group, or hide one that does.
Missing groups. A summary of "all Canadians" can hide very different patterns for smaller groups, such as people in rural areas, newcomers, or First Nations, Métis and Inuit communities, if data for those groups are not collected or not reported.
Cherry-picked ranges. Showing only the years or the subgroup where the trend looks strongest.

Worked example

Example 3: An aggregate correlation

A report shows that for 10 provinces, average household income and average hours of sleep have r = 0.85. A columnist writes, "Richer people sleep much more." What is wrong?

The data are provincial averages, not individuals. Each point hides thousands of people with very different incomes and sleep times.
Correlations between averages are usually much stronger than between individuals, so r = 0.85 says little about how income and sleep are related for a person.
Provinces also differ in other ways (climate, cost of living, work patterns) that could be lurking variables.

This is the ecological fallacy. To make a claim about individuals, you would need data on individual people, and even then correlation would not prove cause.

Defending against hidden variables

Ask what third variables could affect both X and Y.
Stratify: analyse each subgroup separately and compare with the combined result.
Prefer rates to raw totals, and compare like with like.
Ask whether the data are for individuals or groups.
Ask who is missing from the data.

Just for fun

Riddle: two goalies walk into a statistics class. One has the better save percentage on easy shots AND on hard shots. Who has the better save percentage overall?

It depends on who faced the harder shots. That answer is so annoying it has its own name: Simpson's paradox.

Hands-on activity

Make your own paradox with coins or cards

You need: two cups labelled A and B, about 40 small objects in two colours (for example coins: pennies are no longer made in Canada, so use nickels and dimes, or two colours of LEGO bricks or buttons), and paper.

Think of each object as a shot: one colour means saved, the other missed.

Build Goalie A: put 9 saves and 1 miss in an Easy-A pile and 1 save and 3 misses in a Hard-A pile.
Build Goalie B: put 2 saves and 0 misses in an Easy-B pile and 4 saves and 6 misses in a Hard-B pile.
Find each goalie's save rate for easy shots, hard shots and all shots.
Check that B is better in each type of shot but worse overall.
Change the numbers to create your own paradox, and explain in one sentence why it happens.

What to look for: A: easy 9/10 = 90%, hard 1/4 = 25%, overall 10/14 ≈ 71%. B: easy 2/2 = 100%, hard 4/10 = 40%, overall 6/12 = 50%. B wins both types but loses overall, because B faced mostly hard shots. Keep small objects away from young children.

Your turn

Treatment X cured 80 of 100 mild cases and 20 of 100 severe cases. Treatment Y cured 9 of 10 mild cases and 75 of 300 severe cases. Find the cure rates for each type and overall.

Using the previous question, is there a Simpson's paradox? Which treatment looks better?

What two conditions are needed for Simpson's paradox?

What does it mean to stratify data?

A study finds countries with more chocolate eaten per person win more Nobel prizes per person. Name the fallacy and a possible lurking variable.

Why is a summary of "average Canadian income" possibly misleading for smaller groups?

More practice: warm-up

What is the name of a trend that reverses when groups are combined?

What is a lurking variable?

Are correlations of group averages usually stronger or weaker than correlations of individuals?

r = 0.2. What fraction of variation does the line explain?

What is the ecological fallacy?

More practice: on your own

Two hospitals: Hospital 1 had 900 routine surgeries (2% deaths) and 100 high-risk surgeries (20% deaths). Hospital 2 had 200 routine (1% deaths) and 800 high-risk (15% deaths). Find each hospital's overall death rate and explain which hospital is safer.

In a scatter plot of height vs reading level for 60 people aged 5 to 40, r = 0.6. Within the adults only, r ≈ 0. Explain.

A company reports r = 0.9 between monthly advertising and sales, using only the 6 best months of the year. Explain the problem.

Why might a relationship found in a national survey not apply equally to every community?

A city compares two neighbourhoods' average house price and average commute time, with only two points. What can be concluded about the relationship between price and commute for households?

Challenge

Create numbers for two basketball players so that Player B has a higher shooting percentage on both 2-point and 3-point shots, but a lower overall percentage.

Explain why, for Simpson's paradox to happen, the overall rate must be a weighted average with different weights for the two groups.

Describe how you would check a claim that "students who take more online courses have lower marks" for Simpson's paradox.

Check your understanding

Quiz

Simpson's paradox occurs when:

Goalie B has the higher save rate on easy and on hard shots but the lower overall rate. The most likely reason is:

To avoid being fooled by a lurking variable, you should:

A study finds r = 0.88 between average income and average life expectancy for 13 provinces and territories. Concluding the same strong relationship holds for individuals is:

Within each of two schools, more app time goes with higher marks, but combined the trend is negative. What is the lurking variable?

Hospital 1: 4% overall death rate. Hospital 2: 12% overall. Which statement is justified?

For parents and teachers

Big idea (D2.2, D2.5): a third variable can create, hide or even reverse a relationship. Combining groups with different mixes produces Simpson's paradox; using group averages leads to the ecological fallacy. The defence is to stratify and compare like with like.

How to help: The coin or button activity takes ten minutes and makes the paradox concrete. Ask your teen to explain the goalie example using the words "weighted average".

Common difficulties:

Believing the overall percentage is always the fair comparison.
Treating a correlation between averages as if it described individuals.
Forgetting to ask who is missing from the data.
Thinking stratified observational data prove cause (they reduce confounding but do not remove it).