Check References as You Read · Method

Purpose

Every source you read for a project stands on other sources. When one of them cites a study for a claim you plan to rely on, the citation is a promise: “this is where it comes from.” Checking references as you read means opening the cited source, at least for the claims that carry weight, and seeing whether it keeps that promise.

This is the habit Umberto Eco teaches in How to Write a Thesis: work from the original rather than someone’s account of it, and quote so that a reader can go back and check. It’s the same move as Follow the Citation Chain, which traces one claim back to its origin. Here it’s built into reading for a project: dozens of sources, hundreds of references, and a quick triage for which ones to open.

It matters because citations are wrong more often than most readers assume. A 2015 meta-analysis by Hannah Jergas and Christopher Baethge pooled 28 studies of quotation accuracy in medical journals. About one in four citations (25.4%) contained an error. About one in eight (11.9%) had a major error: the cited source contradicted the claim, didn’t support it, or had nothing to do with it.

When you need it

A claim you plan to rely on rests on a single citation.
A number, a strong claim (“proves,” “most,” “rarely”) or a surprising finding comes with a reference.
A source cites a review or a textbook for a specific result, rather than the study that produced it.
You notice the same reference attached to the same claim in paper after paper.
The citing text is vague: “research shows,” “studies suggest,” with several references bundled together.

How to do it

1Mark the claims that carry weight

As you read, flag the cited claims your own work will depend on: the ones you’d repeat, build an argument on, or quote. Most references are background and don’t need checking. A few hold up your whole project.

2Triage

Check first the references behind claims that are central to your work, the ones with exact numbers or strong wording, the ones that seem too neat, and the ones cited secondhand (a study cited through a review). Also check any reference that the same claim leans on everywhere you look, since a widely repeated citation is one that people may be copying rather than reading.

3Open the source and find the passage

Get the cited work itself, not its abstract or a summary. Search the full text for the key terms and find the passage the citation depends on. Note the page or section. If the reference points to a whole book or a long report with no page number, that’s worth noting too.

4Compare what it says with what it’s cited for

Ask whether the source says the same thing, about the same people or cases, with the same numbers and the same certainty. Problems come in a few kinds:

Quotation errors: the source says something different, says something weaker, or doesn’t address the point at all.
Citation drift: the claim grows as it’s passed along. Conditions drop off, a small sample becomes “patients,” a “may” becomes “does.”
Citation laundering: a hypothesis or a weak report gains authority simply by being cited many times, until later authors treat it as established. Steven Greenberg’s 2009 study of one claim in muscle disease research documented this: claims were expanded by papers with no data of their own, and a hypothesis was turned into an accepted fact through citation alone.

5Record what you found

In your notes, label each checked reference: supports the claim, partly supports it, doesn’t support it, or couldn’t check. Note the page. If you rely on the claim yourself, cite the original you actually read. If you couldn’t get it, say “cited in” and name the source you saw it in.

6Adjust your trust in the citing source

One bad citation doesn’t sink a source, but it tells you to check its other key references more carefully. A source whose citations keep holding up earns more trust as you go.

Try it

Your task

For decades, articles cited a 1980 letter in the New England Journal of Medicine by Jane Porter and Hershel Jick as evidence that patients treated with opioid painkillers rarely become addicted. Find the letter and read it (it’s one short paragraph). What patients did it cover, how many, and in what setting? Then find the 2017 analysis of how the letter was cited, and note what it found.

Show a hint

The letter’s title is “Addiction Rare in Patients Treated with Narcotics.” The 2017 analysis was published in the same journal, also as a letter.

Show the answer and the path

The letter is Porter and Jick, “Addiction Rare in Patients Treated with Narcotics,” New England Journal of Medicine 302(2): 123, 10 January 1980. It runs five sentences, about a hundred words.

It drew on the files of a hospital drug-monitoring program (the Boston Collaborative Drug Surveillance Program): 39,946 hospitalized medical patients monitored consecutively.
Of those, 11,882 received at least one narcotic. The authors found four cases of reasonably well-documented addiction among patients with no history of addiction.
It concluded that, despite wide use of narcotics in hospitals, addiction is rare in medical patients with no history of addiction. It gave no methods detail, and it said nothing about long-term use or about patients outside the hospital.

The 2017 analysis is Leung, Macdonald, Stanbrook, Dhalla and Juurlink, “A 1980 Letter on the Risk of Opioid Addiction,” New England Journal of Medicine 376(22): 2194–2195. They found 608 citations of the letter up to March 2017. In 439 of them (72.2%), the citing authors used it as evidence that addiction was rare in patients treated with opioids, and most did not mention that the patients were in hospital. The journal then added a note to the online letter warning readers that it had been “heavily and uncritically cited.”

What drifted: a short report about hospital inpatients became a general claim about opioid treatment. Anyone who opened the letter could see the mismatch in under a minute.

Common mistakes

Checking that the source exists, not what it says. A real reference attached to the wrong claim is the most common major error.
Reading only the abstract. The limits of a finding are usually in the methods and results.
Copying references from another paper’s list. It’s how errors spread. Studying how typos in references get copied from paper to paper, Mikhail Simkin and Vwani Roychowdhury estimated that only about 20% of citers had read the original.
Taking a heavily cited claim as well supported. Many citations can mean many people copying one weak source.
Trying to check everything. You’ll run out of time. Check the references your work depends on.

Limits

Checking takes time, and some sources are hard to get: paywalls, out-of-print books, unpublished data, other languages. When you can’t check a reference, say so rather than pretending you did.

A citation that matches its source only tells you the source says it. It doesn’t tell you the source is right; judging the evidence itself is a different job, covered on ThinkingCritically.net. And a broken citation doesn’t make the claim false. The claim may still be true, just not supported by that reference.

The error rates above come mostly from medical journals, and the studies they pooled used different methods and definitions, so the rate in any given field or source could be higher or lower. The point isn’t the exact number. It’s that errors are common enough that the references you rely on are worth a look.