A Meta-Post about Meta-Research

… or, How to Love the Null Hypothesis

First, bear with me for an important definition.  “Meta” used as a prefix, has come to mean self-reference, or “an X about X”.  For example, meta-analysis is an analysis of other analyses.  If you are arguing with your boyfriend about how unfairly he argues, you’re having a meta-argument, an argument about other arguments.  You with me?

I usually write about a current study in the medical literature, but this week I’m writing about a researcher, John Ioannidis, whose field of study is other medical research.  He researches the biases, flaws and errors that shape medical research.  He is a meta-researcher, and is a leading expert on the credibility of medical research.  Next month’s issue of The Atlantic has a fascinating article about his work (link below).  If you have any interest in medicine or science I urge you to read it.

The Atlantic article highlights the many problems in published peer-reviewed medical studies and the reasons that many of the studies’ conclusions are false.  So that even doctors who are doing their best to stay current with the latest findings are frequently following misleading information.  Once the medical studies are distorted and exaggerated in the general press, it’s no wonder that much of the public believes that medical science is no better than reading tealeaves.  How many times have we heard one year that some medicine is a panacea, and the next year that it’s poison?  How can we argue against unproven alternative medicine when traditional medicine is constantly self-contradicting?

One way for medicine to reclaim its credibility is for doctors to more clearly explain why we know what we think we know, and to be honest about the many things that we don’t know.  Another way for the public to regain trust in medical science is for the general media and the public to become more sophisticated consumers of scientific findings, which is to say to build a more scientifically literate public.

Which brings me to the purpose of these weekly posts.  Obviously the main purpose is to highlight some new and hopefully interesting finding.  But a longer-term purpose is to teach all of you how to separate important findings from hype, speculation from evidence, correlation from cause.  Obviously none of us are going to become statisticians or experts in evaluating scientific experiments, but here are three suggestions that will go a long way to helping you ignore bad studies.

Ignore all non-randomized studies. John Ioannidis in the Atlantic article also agrees with that strategy.  Not sure what the difference is between an observational study and a randomized study?  See the link my post on epidemiology, below.  This simple step will allow you to blissfully forget about 90% of the health related hype you hear.

If possible, read the original scientific article. If you read a scintillating newspaper article about an exciting new finding, try to find the original scientific publication.  The news story will frequently cite the scientific journal, and you’ll usually be able to at least find the abstract online.  Read it.  You may not understand all the statistics, but you’ll have a good sense of what the researchers did and the conclusions they drew, which may be much more modest than what the newspaper is suggesting.

Practice skepticism. The null hypothesis is the assertion that two things have nothing to do with each other.  For example, if I’m trying to prove that eating carrots cures insomnia, the null hypothesis is “eating carrots has no effect on insomnia.”  The null hypothesis is what scientists have to disprove to show a connection between any two things.  People love making connections.  Our brain has evolved to believe in the interrelatedness of things.  We unfortunately instinctively reject the null hypothesis.  If I remember once that I ate carrots with dinner and slept well, then I may have an overwhelmingly compelling belief that the carrots were the cause of my uninterrupted sleep, even though the two are likely coincidental.  We should remember that most pairs of things have nothing to do with each other, which is to say, the null hypothesis is usually right.  Even for pairs of things that are tightly linked in the popular imagination (like cell phones and cancer, or pesticides and health problems) the scientific link is frequently nonexistent.  We should assume that things are not interconnected unless we are presented with repeated large randomized trials showing that one thing is linked to the other.

The problem is that skepticism is hard to do, since our brain is wired for belief, not for skepticism.  That means that we all believe things that are false, but we don’t know which of our beliefs these are.  And since science is a human endeavor it progresses slowly and takes wrong turns and frequently disproves what it earlier believed.  But we don’t have a better tool for getting at the truth than science.  So with all its contradictions, evidence-based medicine is still the best defense against quackery.

So I have to live with the fact that many of the conclusions of my posts are probably wrong, but it’ll take years to figure out which.  But I hope the conclusion of this post is true.  It’s a meta-conclusion.

(Thanks to Timo K. for pointing me to the Atlantic Article.)

Learn more:

The Atlantic article:  “>Ignore Epidemiology, Maybe It’ll Go Away

Important legal mumbo jumbo:
Anything you read on the web should be used to supplement, not replace, your doctor’s advice.  Anything that I write is no exception.  I’m a doctor, but I’m not your doctor despite the fact that you read or comment on my posts.  Leaving a comment on a post is a wonderful way to enter into a discussion with other readers, but I will not respond to comments (just because of time constraints).