E. Culminating Data Management Investigation · Culminating Data Management Investigation

Planning the Investigation: Population, Sample Size and Bias

E1.2About 70 minutes

Learning goals

I can write a complete plan for my investigation before collecting data.
I can choose a population, sampling frame and random sampling method that fit my question.
I can use the margin-of-error formula to choose a sample size, and balance precision against feasibility.
I can predict likely sources of bias and plan ways to reduce them.
I can match each hypothesis to the summaries, graphs and models I will use to analyse it.

Words to know

investigation plan
A written outline, made before collecting data, of the question, variables, population, sampling method, sample size, procedures, analysis, ethics and timeline.

sampling frame
The actual list or source from which the sample is drawn, such as a school's class list. Ideally it matches the population exactly.

systematic random sample
A sample made by choosing every k-th member of an ordered list, starting from a randomly chosen position between 1 and k.

margin of error
The amount added and subtracted from a sample estimate to form a confidence interval, for example 55% ± 14%. Larger samples give smaller margins of error.

non-response bias
Bias that occurs when the people who do not respond differ in important ways from those who do.

analysis plan
A list saying which numerical summaries, graphs and probability models will be used for each hypothesis.

Measure twice, cut once

Carpenters say "measure twice, cut once". In statistics, you plan before you collect, because mistakes in data collection cannot be fixed by clever analysis afterwards. A biased sample stays biased no matter how many graphs you draw.

The curriculum asks you to design a plan: identify the variables and the population, develop an ethical survey, establish the procedures for gathering, summarizing and analysing the data, and consider the sample size and possible sources of bias. This lesson covers everything except the survey itself, which has its own lesson next, Designing an Ethical Survey.

The parts of a plan

Question and hypotheses (from the last two lessons).
Variables with operational definitions and types (categorical or numerical, discrete or continuous).
Population and sampling frame. Are they the same? Who is missing from the frame?
Data source: primary (survey, experiment, simulation, playing a game) or secondary (a named database).
Sampling method and sample size, with reasons.
Procedure: exactly how the data will be collected and recorded, step by step.
Analysis plan: which summaries, graphs and models answer each hypothesis.
Bias: likely sources and how you will reduce them.
Ethics: consent, anonymity, permissions.
Timeline.

Choosing a sampling method

Recall the random sampling methods from Organization of Data for Analysis: simple random, systematic, stratified, cluster and multistage. Choose the one that fits your frame and your question. If your question compares groups, a stratified sample can guarantee enough members of each group. If you have an ordered list, a systematic sample with a random start is easy and fair, as long as the list has no repeating pattern that lines up with your interval.

Amira's sampling plan. The office list (the sampling frame) covers the whole population, and a random start makes every student equally likely to be chosen.

How big should the sample be?

In One-Variable Data Analysis you saw that the 95% margin of error for a sample proportion is about

ME = 1.96 × √( p(1 − p) / n ).

Solving for n gives the sample size needed for a chosen margin of error E:

n = p(1 − p) × (1.96 / E)².

If you have no estimate of p, use p = 0.5, which gives the largest (safest) n. Always round up.

Precision costs effort: halving the margin of error needs about four times the sample. A student project must balance precision against what is feasible, and then report the margin of error honestly. (When the sample is a large share of a small population, this formula slightly overstates the margin of error, so it is a safe, conservative choice.)

The 95% margin of error for a proportion when p = 0.5. The curve flattens: going from 48 to 97 people gains about 4 percentage points of precision, but reaching ±5% needs about 385.

Worked example

The model investigation, Stage 4: sample size

Amira would like a margin of error of ±10% for the proportion of students who meet the 8-hour guideline (95% confidence). (a) What sample size is needed? (b) The guidance office can distribute about 50 surveys. What margin of error will a sample of 48 give?

(a) With no earlier estimate, use p = 0.5: n = 0.5 × 0.5 × (1.96 / 0.10)² = 0.25 × 384.16 = 96.04. Round up: n = 97.
(b) ME = 1.96 × √(0.25 / 48) = 1.96 × √0.005208 = 1.96 × 0.07217 ≈ 0.141, about ±14 percentage points.
Decision: 97 is about 40% of the whole population of 240, which is not feasible for one student. She chooses n = 48 and will report the larger margin of error as a limitation.
Note: comparing two groups (job and no job) splits the sample, so each group's estimate is even less precise. She writes this down now so it is not a surprise later.

(a) 97 students for ±10%. (b) A sample of 48 gives about ±14%. Amira uses 48 and plans to discuss precision as a limitation.

Worked example

The model investigation, Stage 5: the full plan

Summarize the rest of Amira's plan: sampling, procedure, analysis and bias.

Population and frame: all 240 Grade 12 students; the office's alphabetical list of Grade 12 students (the frame matches the population).
Sampling: systematic, k = 240 ÷ 48 = 5. She rolls a die until it shows 1 to 5 and gets 3, so the sample is positions 3, 8, 13, ..., 238. An alphabetical list has no pattern related to sleep or work, so systematic sampling is fair here.
Procedure: office staff give each selected student an envelope with the anonymous survey and an information sheet; completed surveys go into a sealed box in the library within one week. Amira never sees who was chosen.
Analysis plan for Hypothesis 1: mean, median, standard deviation and five-number summary of sleep for each group; side-by-side box plots; histogram; a 2 × 2 table of job (yes/no) by meets guideline (yes/no); proportions with margins of error; a simulation to see whether a difference as large as the observed one could easily happen by chance.
Analysis plan for Hypothesis 2: scatter plot of sleep against work hours for students with jobs; correlation coefficient r; line of best fit by least squares; residual plot.
Bias to expect and reduce: non-response (busy working students may not return surveys; she will send one reminder through the office and report the response rate); response bias (people overstate sleep; the survey is anonymous and neutral in tone); measurement (recall of "usual" sleep is rough; she asks for the nearest quarter hour and defines the nights).
Timeline: week 1 plan and permission; week 2 pilot and revise survey; week 3 collect; week 4 organize and analyse; week 5 write report and presentation.

A complete plan names the frame, sampling method and reasons, procedure, analysis for each hypothesis, expected biases with fixes, and a timeline.

Worked example

Example 3: Planning a secondary-data investigation

Jonah asks: "Is there a relationship between the percentage of people who commute by public transit and population density, for Ontario's census divisions?" Plan his population, sampling and analysis.

Population: all census divisions in Ontario. With secondary data he can use every division, so no sampling is needed: this is a census of the population of divisions, and there is no sampling margin of error at that level (the census values themselves may have their own quality notes).
Data source: Statistics Canada Census of Population data tables: main mode of commuting, population and land area by census division. He records table numbers and the census year.
Variables: population density = population ÷ land area in km² (explanatory); percentage commuting by public transit (response). Both numerical, continuous.
Analysis: scatter plot, r, regression line, residuals; check for outliers such as Toronto, which may dominate the pattern; consider analysing with and without it.
Bias and limitations: aggregate data cannot tell us about individuals (a relationship across divisions does not prove that individuals in denser places choose transit); transit availability is a lurking variable.

Secondary data often allow a whole population of areas to be studied. The plan focuses on careful definitions, source records, outliers and the limits of aggregate data.

Plan the analysis now

For every hypothesis, write the exact summary or graph that will answer it before collecting data. If you cannot say how a survey question will be analysed, you probably do not need that question. If a hypothesis has no matching analysis, you need another question or another data source.

Just for fun

A student planned to survey 1000 people for a ±3% margin of error. Then they counted the people they knew: 74. The plan was revised to ±11%, and a nap.

Hands-on activity

Your investigation, step 3: write the plan

You need: your project log, a calculator, and a die or a random-number tool.

Write the 10 parts of a plan listed in this lesson for your own investigation.
Calculate the sample size for a margin of error of ±10% (or justify why you are using a census of a small population or a whole secondary data set). Then decide on a feasible size and calculate its margin of error.
If you are sampling, choose a random method. Practise selecting your sample: for example, number a list of 30 family contacts or 30 rows of a data set and use a die (or random numbers) to choose a systematic or simple random sample of 6.
Make a table with three columns: possible bias | why it could happen | how I will reduce it. Include at least three rows.
Show the plan to a family member and ask them to find one weak point. Revise.

What to look for: every hypothesis has a matching analysis; the sampling method is random; the margin of error is calculated; at least three biases are named with realistic fixes.

Your turn

Find the sample size needed for a margin of error of ±5% at 95% confidence when p is unknown.

Find the 95% margin of error for a sample of 120 when p = 0.5.

A list of 600 students is used to draw a systematic sample of 40. What is k, and what range should the random start come from?

Name the sampling frame in Amira's plan and say whether it matches the population.

Why does Amira not need to see who was selected?

Give one source of non-response bias in Amira's study and one way to reduce it.

More practice: warm-up

What p should you use in the sample-size formula if you have no estimate?

Should you round a sample size up or down?

To halve the margin of error, roughly how much larger must the sample be?

Name the five random sampling methods from this course.

Is a survey of every census division in Ontario a sample or a census of those divisions?

More practice: on your own

A previous survey suggests about 20% of students have a certain trait. Find n for a ±6% margin of error at 95% confidence.

Amira's study compares students with and without jobs. Explain how a stratified sample could help if only a few students had jobs.

A student plans to survey people leaving a gym about how often they exercise, to estimate the exercise habits of their town. Identify the problem and propose a better frame.

Write an analysis plan for the hypothesis "In my survey, the proportion of students who walk to school is greater for students living within 2 km than for students living farther away."

Why is it a problem to choose the sampling method after seeing which students answered?

Challenge

Amira's 48 surveys are split into two groups of roughly 22 and 18. If p = 0.5 in the job group of 22, what is the 95% margin of error for that group alone? What does this say about comparing groups?

A systematic sample uses every 10th house on a street where every 10th house is a corner lot (larger and more expensive). Explain the danger and fix it.

A student has data on all 82 games of a hockey team's season and wants a "95% confidence interval" for its home-win percentage this season. Explain whether a margin of error makes sense and what question it would actually answer.

Check your understanding

Quiz

What sample size gives a 95% margin of error of ±10% when p = 0.5?

Amira's sampling frame is:

Doubling the sample size makes the margin of error:

A systematic sample of 30 is drawn from a list of 450. The interval k is:

Which is an example of non-response bias?

Why is it best to write the analysis plan before collecting data?

For parents and teachers

Big idea (E1.2): a good plan decides the population, the random sampling method, the sample size and the analysis before any data are collected, and predicts likely sources of bias.

How to help: Ask your teen to explain how their sample will be chosen and why it is random. Check the margin-of-error arithmetic together. Help them be realistic: a smaller sample that is truly random and well documented is better than a big convenience sample of friends.

Common difficulties:

Using a convenience sample (friends, social media followers) and calling it random.
Forgetting to round the sample size up, or using p = 0.5 incorrectly when an estimate exists.
Not realizing that comparing groups splits the sample and reduces precision.
A plan with survey questions that are never analysed, or hypotheses with no matching analysis.

Permissions: any survey run at a school needs the school's permission; your teen should ask a teacher or principal well in advance.