C. Organization of Data for Analysis · Organization of Data for Analysis

Principles of Primary Data Collection: Control, Randomization and Replication

C2.1About 70 minutes

Learning goals

I can explain why a well-designed experiment can show cause and effect while an observational study usually cannot.
I can identify the treatment, experimental units, control group and response variable in an experiment.
I can explain the principles of control, randomization and replication and why each is needed.
I can identify confounding (lurking) variables and list the criteria for collecting reliable data.

Words to know

experimental units (subjects)
The individuals on which an experiment is done. When they are people, they are called subjects or participants.

treatment
A specific condition applied to the experimental units, such as a fertilizer amount or a study app.

factor and levels
A factor is an explanatory variable the researcher controls; its levels are the values used (for example, fertilizer at 0 g, 5 g and 10 g).

control group
A group that receives no treatment, a placebo or the usual treatment, to provide a baseline for comparison.

placebo
A fake treatment that looks like the real one but has no active ingredient, such as a sugar pill.

confounding (lurking) variable
A variable not included in the study that is related to both the explanatory and response variables, so its effect gets mixed up with the treatment's effect.

randomization
Using chance to assign units to treatments (in experiments) or to select a sample (in surveys).

replication
Using enough experimental units in each group, and repeating experiments, so that chance differences average out.

Did the fertilizer work, or was it the sunny window?

Your neighbour tried a new fertilizer on her tomatoes this summer and got a record crop. Was it the fertilizer? Maybe. Or maybe it was the hot, sunny summer, or the new garden bed in a sunnier spot, or the fact that she watered more because she was excited about the experiment.

To know whether something causes a change, you need a carefully designed experiment. Statisticians use three principles to design one: control, randomization and replication. The curriculum lists these as the principles of primary data collection.

Observational studies and experiments

Recall from the lesson on kinds of data:

In an observational study, the researcher observes or measures without changing anything.
In an experiment, the researcher deliberately imposes a treatment and measures the response.

Observational studies are often the only ethical or practical option (you cannot assign people to smoke). They can show associations, but a confounding (lurking) variable might be the real cause.

For example, among elementary students, children with larger shoe sizes tend to have higher reading scores. Big feet do not cause good reading! Age affects both: older children have bigger feet and read better. Age is a lurking variable.

A lurking variable (age) affects both shoe size and reading score, creating an association between them even though neither causes the other.

Experiment vocabulary

Experimental units or subjects: who or what the treatments are applied to (plants, students, batteries).
Factor: the explanatory variable the researcher controls. Levels: the values it takes.
Treatment: a specific condition applied to units (a level, or a combination of levels of several factors).
Response variable: what is measured to see the effect.
Control group: units that get no treatment, a placebo, or the current standard treatment, to give a baseline.

The three principles

1. Control. Compare the treatment group with a control group, and keep everything except the treatment the same for all groups: same soil, same water, same test, same time of day. Without a comparison group you cannot tell whether a change was caused by the treatment or by something else that happened at the same time (weather, the placebo effect, ordinary improvement over time).

2. Randomization. Use chance to decide which units get which treatment. Randomization balances out lurking variables, even ones you have not thought of, so the groups are alike at the start. If the researcher chooses ("I'll give the fertilizer to the healthiest-looking plants"), the choice itself can create bias.

Randomization matters in surveys too: a random sample avoids the selection bias you met in the last lesson. Random selection decides who is studied; random assignment decides who gets which treatment.

3. Replication. Use enough units in each group, and repeat the experiment. One plant per group tells you almost nothing: that single plant might be unusually strong. With many units per group, chance differences between individual units average out. Replicating a whole study (by other researchers, in other places) is how science builds confidence in a result.

A completely randomized design: volunteers are randomly assigned to the treatment or the control group, then the groups' responses are compared.

Criteria for reliable data

Whether you run an experiment or a survey, data are more reliable when:

the questions or measurements are appropriate: they actually measure the variable you care about;
sources of bias have been identified and reduced;
the sample size (or number of units per group) is large enough;
procedures are consistent and written down so someone else could repeat them;
data are recorded carefully at the time, with units, and checked for errors.

Worked example

Example 1: Designing a fertilizer experiment

You have 24 tomato seedlings and want to know whether the amount of a fertilizer affects how many tomatoes each plant produces. Design an experiment using the three principles.

Factor: fertilizer amount, with 3 levels: 0 g (control), 5 g and 10 g per week. Response: number of ripe tomatoes per plant by September 1.
Replication: 24 plants ÷ 3 treatments = 8 plants per treatment.
Randomization: number the plants 1 to 24. Use a random-number generator to choose 8 different numbers for 0 g, then 8 of the remaining for 5 g; the last 8 get 10 g.
Control: all plants get the same pot size, soil, water schedule and sunlight. Mix the pot positions on the deck randomly so no treatment gets all the sunny spots. The 0 g group is the control.
Analysis: compare the mean number of tomatoes per plant in the three groups.

A completely randomized design with 3 treatments × 8 plants, a 0 g control group and all other conditions kept the same.

Worked example

Example 2: Critiquing a study

A teacher tests a new review game. She uses it with her period 1 class and uses her usual review with her period 4 class. Period 1's mean test score is 6 marks higher. She concludes the game works. Identify the flaws and redesign the study.

No randomization: students were not randomly assigned; they were already in classes. The classes may differ in ability, interest or course selection.
Confounding: time of day is mixed up with the treatment. Morning students may be more (or less) alert. Any difference could be due to the class or the time, not the game.
Little replication: only one class per treatment. A single class's mean can differ by several marks by chance.
Redesign: randomly assign students within each class to the game or the usual review (if practical), or use many classes, randomly choosing which classes get the game; keep the same test and the same review time for everyone.

The 6-mark difference may be due to differences between the classes or the time of day. Random assignment, more groups and identical conditions would make the conclusion trustworthy.

Worked example

Example 3: Can this study show cause and effect?

Decide whether each is an experiment or an observational study and whether it can show cause and effect. (a) Researchers compare the heart health of people who say they drink coffee with people who say they do not. (b) Researchers randomly assign 200 volunteers to sleep either 6 or 8 hours for a week, then test their reaction times.

(a) Observational: people chose whether to drink coffee. Coffee drinkers may differ in other ways (sleep, stress, smoking, diet), which are possible lurking variables. It cannot establish cause and effect on its own.
(b) Experiment: researchers assigned the amount of sleep at random. Randomization balances lurking variables, so a clear difference in reaction time can be attributed to sleep.

(a) observational, association only; (b) randomized experiment, can show cause and effect.

Just for fun

Why did the statistician give a pep talk to half of her houseplants every morning and say nothing to the other half? Because without a control group, she would never know whether it was her encouragement or just the spring sunshine.

Hands-on activity

Paper airplane experiment

You need: 2 sheets of printer paper, a tape measure (at least 5 m), masking tape, a coin, and a clear indoor space such as a hallway or basement. Throw away from people, pets and breakable things.

Question: Does a dart-style airplane fly farther than a glider-style airplane?

Fold one dart (narrow, pointed) and one glider (wide wings). Mark a throwing line on the floor with tape.
Control: the same person throws both, from the same line, at the same height, using the same throwing motion.
Replication: plan 10 throws with each plane.
Randomization: before each throw, flip a coin: heads, throw the dart; tails, the glider (stop using a plane once it has had 10 throws). This prevents the thrower's arm getting tired, or getting better with practice, from always favouring one plane.
Measure the distance from the line to where each plane first touches the floor, to the nearest centimetre. Record it in a table with the throw number and plane.
Find the mean distance for each plane. Write a conclusion and list one limitation.

What to look for: a careful table, means for both planes, and a conclusion like "In 10 throws each, the glider's mean distance was 4.2 m and the dart's was 3.5 m." A good limitation: only one thrower, one pair of planes and one room, so the result might not hold for other planes or throwers.

Your turn

In an experiment, 30 students are randomly assigned to study with music or without music and then tested on vocabulary. Identify the experimental units, factor, treatments and response variable.

What is the purpose of a control group?

Why do researchers randomly assign subjects to treatments?

A study tests a new battery by putting it in one remote control. Which principle is missing?

Ice-cream sales and the number of people at beaches rise and fall together. Name a possible lurking variable.

Is a study in which people self-report hours of sleep and their marks an experiment? Can it show that sleep causes better marks?

More practice: warm-up

Name the three principles of experimental design.

What is a placebo?

In an experiment, who decides which treatment each unit gets?

What is the response variable in a study of how fertilizer affects plant height?

True or false: randomization is used only in experiments, never in surveys.

More practice: on your own

A company claims its new energy drink improves reaction time. It gives the drink to 50 volunteers and finds their reaction times improved from a pre-test to a post-test. Give two reasons this does not prove the drink works, and redesign the study.

Explain the difference between random selection and random assignment.

A study finds that students who take music lessons have higher marks. Suggest two lurking variables.

Why is it important to write down the procedure of an experiment?

A researcher has 60 mice and 3 diets. How many mice should be in each group for a balanced design, and how could she randomize?

Challenge

Explain why a large observational study can never fully rule out confounding, even after researchers adjust for many variables.

Researchers want to know whether a new reading program helps Grade 3 students. Design an experiment for 12 Grade 3 classes in 6 schools, explaining how you use control, randomization and replication. Why might it be wise to randomize classes within each school?

A study shows that cities with more police officers have more crime. A councillor says, "Police cause crime, so cut the police budget." Explain the flaw using the idea of a lurking variable or reversed cause.

Check your understanding

Quiz

Which principle of experimental design is mainly about balancing out lurking variables between groups?

An experiment tests a new pain medication. The control group should receive:

Using 50 plants in each group instead of 2 is an example of:

Students who eat breakfast have higher marks. Which is the best description of this result?

In a study of how hours of practice affect free-throw percentage, the response variable is:

Which is a criterion for reliable data?

For parents and teachers

Big idea (C2.1): good primary data come from careful design. In experiments, control (a comparison group and identical conditions), randomization (chance decides who gets what) and replication (enough units) let us conclude cause and effect. Observational studies show associations that may be caused by lurking variables.

How to help: The paper airplane activity is a fun way to practise all three principles together. When you hear a claim like "People who do X live longer," ask: "Was that an experiment? What else might be different about people who do X?"

Common difficulties:

Thinking any comparison of two groups is an experiment. It is only an experiment if the researcher assigns the treatment.
Confusing random selection (who is in the study) with random assignment (who gets which treatment).
Forgetting the control group, or not keeping other conditions the same.
Naming a lurking variable that affects only one of the two variables. A lurking variable must be related to both.