My View
One Study Is Enough To Be Dangerous
BY NORA E. GORDON/School Administrator, January 2021

WHEN SOMEONE ASKS, “But what’s your evidence for that?” the easiest way to satisfy them is to cite a research paper — and just one paper will get the job done.

While that may be the easiest way to defend a decision you’ve already made, it’s not the best way to come to an informed decision that will best serve students. And, to be honest, if you really just need a citation and not information, you probably don’t need my help.

Prevailing Beliefs

If you go looking for a study showing your idea is a good one, you will find one. You might find studies showing it’s a bad one too, but your subconscious is on the job, looking for reasons to dismiss those findings.

Perhaps the study took place in an urban district, and you’re working in a rural one. Maybe there were only 20 kids in the study or the research predates the curricular standards in your state. Or the study focused on 8th graders and you are thinking about 6th graders. At the same time, the study that supports your idea probably has a similar set of issues, but our cognitive biases make it easy to dismiss such concerns when we are predisposed to agree with the study.

Even if a study was conducted in a context similar to your own, you still need to judge how convincing it is. If the study shows students in an after-school program had higher math scores, how do you know it was the math program itself that raised their test scores? Maybe teachers selected students they thought would benefit most. Maybe the families that were able to take advantage of the program were better equipped to help support their students’ learning in other ways.

Randomized control trials are a great solution to these challenges, but most studies in education do not involve random assignment to treatment or control groups. In other words, if you limit yourself to the studies with the strongest research designs for distinguishing cause and effect, you will leave a lot of questions unanswered.

Finally, the findings in published papers often do not stand up when other researchers try to replicate them. The social science research community is engaged in vigorous debate about why this “replication crisis” exists — and the role of researchers’ own cognitive biases. Regardless of why this crisis exists, its presence suggests consumers of research should not put all their eggs in the basket of a single study.

Look to the Forest

Nonetheless, there’s hope for using research to learn and improve. Even though it is usually hard to find a study conducted in a con-text just like your own, the accumulation of evidence over a range of contexts is informative. Many studies make convincing comparisons to help prove cause and effect — even if they do not rely on a true experiment with random assignment. And new research teams do often successfully replicate existing findings. To take advantage of all the good research out there, while avoiding these dangers, focus on the lay of the land rather than one study at a time.

Two practical ways to do this:

»When reading writeups of research in education reporting, try to figure out whether the author is describing one new study or reviewing the body of knowledge; and

»When looking for the big picture, look for systematic summaries of the research literature, rather than individual studies. Two good sources: the What Works Clearinghouse’s practice guides, and the Campbell Collaboration.

In short, let someone else do the hard work of sorting through all the research, so you can benefit from the big picture.

NORA GORDON is an associate professor at Georgetown University’s McCourt School of Public Policy in Washington, D.C., and co-author of Common-Sense Evidence: The Education Leader’s Guide to Using Data and Research. Twitter: @NoraEGordon