Research III: Evaluating Sources and Evidence Like a University Researcher

Peer-reviewed, published and wrong

In 1998 the respected medical journal The Lancet published a peer-reviewed paper by Andrew Wakefield and colleagues that suggested a link between the measles, mumps and rubella (MMR) vaccine and autism. Investigations later found serious problems, including misrepresented data and ethical violations. The Lancet fully retracted the paper in 2010, and large studies since then have found no such link. By then, the claim had spread around the world.

The lesson is not that peer review is useless. It is that no single label guarantees truth. At Grade 12, evaluating sources means asking harder questions: How strong is this kind of evidence? Who checked it, and how? What do the numbers really say? Do other independent sources agree? That is how researchers work, and it is how you will be expected to work in university.

What peer review does, and does not, do

In peer review, a journal editor sends a submitted article to experts who assess whether its methods are sound, its reasoning follows and its claims match its evidence. Reviewers often ask for changes; many articles are rejected.

But reviewers usually do not repeat the research, and they see only what the authors show them. So:

• Errors and fraud can get through. When they are discovered, the journal may publish a correction or a retraction. Before citing an older article, check the journal page for a retraction notice; a retracted article is usually stamped RETRACTED.
• Preprints are papers shared before peer review, often on public servers so that results circulate quickly. They can be valuable, but treat them as unreviewed and say so (a study not yet peer-reviewed found...).
• Predatory journals charge authors to publish and do little or no real review. Warning signs: very fast publication promises, spam invitations to submit, vague editorial boards and names that imitate well-known journals. Your librarian can tell you whether a journal is reputable.
• One study is one study. Even a good study can be an outlier. Look for whether later research replicated it.

Weighing kinds of evidence

Not all evidence is equally strong. For health and science claims, researchers often use a ranking like the pyramid below. A randomized controlled trial (RCT) assigns people to groups by chance, so the groups are alike except for the one thing being tested; that makes it much better than an observational study at showing cause. A systematic review gathers all the good studies on a question, which protects against relying on one outlier.

Literary research has its own version. The strongest evidence for a claim about a novel is the text itself, quoted accurately with a page number. Next comes published scholarship that analyses the text. Reputable reviews and interviews add context. Study-guide and summary websites are at the bottom: they are often unattributed, not reviewed and sometimes wrong, so do not cite them as evidence.

Reading numbers without being fooled

• Sample. How many, and who? 120 students at two private schools cannot speak for all Canadian teens. Ask whether the sample is representative.
• Margin of error. A poll of about 1000 randomly chosen people typically reports a margin of error of about ±3 percentage points, 19 times out of 20. If one option has 48% and another 46%, the poll cannot say which is really ahead.
• Relative vs absolute change. If a risk rises from 2 in 1000 to 3 in 1000, the relative increase is 50% (the risk is one and a half times as large), but the absolute increase is 0.1 percentage points: one more case per 1000 people. Headlines love relative numbers. Report both.
• Time frame and baseline. Crime up 20% since when, from what starting number? A cherry-picked start date can make any trend look dramatic.
• Graphs. Check axes that do not start at zero, uneven intervals and missing units (Grade 11 skills). Reading Informational and Graphic Texts takes data graphics further, with rates, margins of error and logarithmic scales.

Follow the money, check the citation

• Conflicts of interest. Scholarly articles usually include a funding statement and a competing interests or disclosures statement. Funding does not make a study wrong, but a study paid for by a company that sells the product needs independent confirmation.
• Verify that sources exist. AI chatbots and some low-quality websites produce citations that look real but are not: real-sounding journals, plausible authors, wrong years or invented titles. Before you cite anything, find it yourself: search the exact title in your library database or a scholarly search engine, open the article, and check that the author, journal, year and page numbers match. If a DOI (digital object identifier) is given, it should lead to the real article. Never cite a source you have not opened.
• Check quotations too. A quotation attributed to a famous writer online is often misattributed. Find it in the original work or in a reliable edition before using it.

Different knowledge systems

Completeness includes recognizing that knowledge is produced in more than one way. Indigenous knowledge, held and passed on by First Nations, Métis and Inuit Elders, Knowledge Keepers and communities, is grounded in long observation of the land, in relationships and in oral tradition. It has its own methods of verification and its own protocols about who may share what, and when.

• Oral history is evidence. In Delgamuukw v. British Columbia (1997), the Supreme Court of Canada recognized that oral histories must be given due weight as evidence, alongside written documents.
• Two-Eyed Seeing (Etuaptmumk in Mi'kmaw) is a guiding principle brought forward by Mi'kmaw Elder Albert Marshall: learning to see with the strengths of Indigenous knowledges with one eye and the strengths of Western knowledges with the other, and using both eyes together. Many Canadian researchers in health and environmental science now design studies with this principle.
• Respect the protocols you learned in Grade 11 (asking permission, OCAP® for First Nations data, citing Elders and Knowledge Keepers as named sources).

This is not about choosing one system over another. It is about evidence being complete: a study of a northern river that ignores the community that has lived beside it for generations is missing evidence.