Are published research findings mostly false?
Recent talk, old paper: Concerns raised that suggest research findings are false in at least half the peer-reviewed and published scientific papers.
At last week’s Victorian Science Teacher Conference, (STAVCON held in Melbourne, Australia), I heard Professor David Vaux present a disconcerting keynote address that suggests research findings are false in at least half the peer-reviewed and published scientific papers. Many of these papers are in some of the more prestigious journals such as Nature, Science and PNAS.
There are many reasons for this and only a small percentage (about 2-3 percent) can be attributed to misconduct. Reasons given for this include the obvious financial interests and accompanying pressure to publish, and the problems of bias. What surprised me and shook my faith in the peer-review process was that a good chunk of the dodgy results happen because scientists are using the wrong or inappropriate statistical methods, or the statistical results are missing information that is vital to making any sort of valid conclusion. And David had no shortage of examples to show us, many that came to him for review, and many more that made it through the peer-review process and are still available in many of the prestigious journals. According to David they shouldn’t be there, but apparently getting a paper retracted is difficult.
I am the first to admit that my use of statistics ended with my honours research nearly 20 years ago and then I only learnt what I had to. So my knowledge of stats is poor, at best. But then I am not a scientist trying to get a paper published in a peer-reviewed journal. What shook my faith, if I understood Prof Vaux correctly, was the inadequate understanding scientists have of statistical methods, which suggests the dodgy papers are largely not that way deliberately
Nobody denies there is uncertainty in science and that a certain percentage of results will turn out to be false. It is the nature of the scientific method. And 50 percent and my context-lacking post make it sound a bigger deal that it possibly is, but I wonder, however, if it is naïve of me to think we should be concerned if even 10 percent of published papers were statistically dodgy.
To get the context and wider perspective on this, PLOS Medicine published a paper and accompanying editorial and commentary in 2005
The fact it is a 2005 paper means I am finding out about all this late, but from the questions David got after the talk, I reckon I am not the only ignorant one.
Prof David Vaux runs a lab (Mechanisms of Cell Death, Aging and Homeostasis) in La Trobe University’s Dept of Biochemistry.
Jason