In a recent segment of his show “Last Week Tonight,” comedian John Oliver took on the issue of research and scientific studies. He satirized the outlandish reporting of dubious research, such as news stories alleging that a glass of wine has the same health benefits as an hour at the gym.
While these claims are blown out of proportion by sensationalist media outlets, they can also stem from poor research methods in scientific studies. Journals are more likely to publish new, interesting, and statistically significant work, incentivizing researchers to unfortunately neglect sound research practices or disregard principles of transparency and replicability. Other researchers – often without ill intentions – fall into the dangerous trap of “p-hacking,” or data fishing for patterns that are statistically significant but practically meaningless.
Here’s an example of “p-hacking” from xkcd.
These poor research methods can create evidence about nearly anything, reducing faith in scientific discoveries. A reporter on the Today show summed up the problem of scientific studies, saying “the way to live your life is to find the study that sounds best to you, and you go with that.”
J-PAL North America promotes proper transparency and reproducibility as well as sound research design. When conducted with proper care, randomized evaluations can be used to reliably quantify the impact of social programs. J-PAL North America also works to help policymakers and donors interpret social science research correctly and synthesize lessons across different studies.
Watch the full John Oliver segment below: