Most Published Scientific Research Is Wrong

[roster]

No, SnoreDog, science will never catch up with what you are saying.

Publish and be wrong
Oct 9th 2008
From The Economist print edition

One group of researchers thinks headline-grabbing scientific reports are the most likely to turn out to be wrong
Adrian Johnson
IN ECONOMIC theory the winner’s curse refers to the idea that someone who places the winning bid in an auction may have paid too much. Consider, for example, bids to develop an oil field. Most of the offers are likely to cluster around the true value of the resource, so the highest bidder probably paid too much.

The same thing may be happening in scientific publishing, according to a new analysis. With so many scientific papers chasing so few pages in the most prestigious journals, the winners could be the ones most likely to oversell themselves—to trumpet dramatic or important results that later turn out to be false. This would produce a distorted picture of scientific knowledge, with less dramatic (but more accurate) results either relegated to obscure journals or left unpublished.

In Public Library of Science (PloS) Medicine, an online journal, John Ioannidis, an epidemiologist at Ioannina School of Medicine, Greece, and his colleagues, suggest that a variety of economic conditions, such as oligopolies, artificial scarcities and the winner’s curse, may have analogies in scientific publishing.

Dr Ioannidis made a splash three years ago by arguing, quite convincingly, that most published scientific research is wrong. Now, along with Neal Young of the National Institutes of Health in Maryland and Omar Al-Ubaydli, an economist at George Mason University in Fairfax, Virginia, he suggests why.

It starts with the nuts and bolts of scientific publishing. Hundreds of thousands of scientific researchers are hired, promoted and funded according not only to how much work they produce, but also to where it gets published. For many, the ultimate accolade is to appear in a journal like Nature or Science. Such publications boast that they are very selective, turning down the vast majority of papers that are submitted to them.

Picking winners
The assumption is that, as a result, such journals publish only the best scientific work. But Dr Ioannidis and his colleagues argue that the reputations of the journals are pumped up by an artificial scarcity of the kind that keeps diamonds expensive. And such a scarcity, they suggest, can make it more likely that the leading journals will publish dramatic, but what may ultimately turn out to be incorrect, research.

Dr Ioannidis based his earlier argument about incorrect research partly on a study of 49 papers in leading journals that had been cited by more than 1,000 other scientists. They were, in other words, well-regarded research. But he found that, within only a few years, almost a third of the papers had been refuted by other studies. For the idea of the winner’s curse to hold, papers published in less-well-known journals should be more reliable; but that has not yet been established.

The group’s more general argument is that scientific research is so difficult—the sample sizes must be big and the analysis rigorous—that most research may end up being wrong. And the “hotter” the field, the greater the competition is and the more likely it is that published research in top journals could be wrong.

There also seems to be a bias towards publishing positive results. For instance, a study earlier this year found that among the studies submitted to America’s Food and Drug Administration about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results were. But negative results are potentially just as informative as positive results, if not as exciting.

The researchers are not suggesting fraud, just that the way scientific publishing works makes it more likely that incorrect findings end up in print. They suggest that, as the marginal cost of publishing a lot more material is minimal on the internet, all research that meets a certain quality threshold should be published online. Preference might even be given to studies that show negative results or those with the highest quality of study methods and interpretation, regardless of the results.

It seems likely that the danger of a winner’s curse does exist in scientific publishing. Yet it may also be that editors and referees are aware of this risk, and succeed in counteracting it. Even if they do not, with a world awash in new science the prestigious journals provide an informed filter. The question for Dr Ioannidis is that now his latest work has been accepted by a journal, is that reason to doubt it?

http://www.economist.com/science/displa ... d=12376658

[jnk]

Does that mean that the people who did the research concluding that people who do research and then get published are wrong are also wrong themselves, since they did research and got published? So is the point really the opposite, since they are most likely wrong about published research being wrong, or am I wrong? Wait, I must be less wrong than them because I haven't been published!

[Goofproof]

It means that most of the money spent on research is given to people who couldn't find their hat in a box, in the meantime funding for worthwhile projects is all used up. It's a self serving system, not overseen by anyone with real knowledge of facts. A closed Elite Club so to say. Jim

[-SWS]

http://medicine.plosjournals.org/perlse ... 50201&ct=1

http://www.pubmedcentral.nih.gov/articl ... id=1182327

[rested gal]

Cool. Reminds me of something you wrote, -SWS....

"How much bad methodology has shown up in scientific publications throughout the years? Plenty."

... in this old thread:
viewtopic.php?p=140080#p140080

[Snoredog]

Hey, I coulda told ya that

[Velbor]

Does the statement "Most Published Scientific Research Is Wrong"
include reference to opinions published in this thread?

[jnk]

Does the statement "Most Published Scientific Research Is Wrong"
include reference to opinions published in this thread?

Probably.

Which is why I will now give my opinion:

Most scientific research sets forth a hypothesis, presents a method for exploring that assumption, then gives conclusions based on 'what we think may have occurred when we tried to do what we set out to do.' The hypothesis is an opinion. The method of exploration is another opinion. And the conclusion is yet another opinion based on the preceding two opinions.

But in order to entertain any question, you have to draw lines around a part of the larger problem to frame it, so all questions and all answers are flawed to some extent in one form or another, scientifically speaking. But there is still value in asking questions and looking for some consensus on where to look for the answers.

So who decided in the study of other studies which studies were "right" and which studies were "wrong"? What percentage of judged wrongness equaled "wrong"? That too, in and of itself, is an opinion based on how researchers arbitrarily decided to frame their own question, exploration, and conclusion.

Most research that is published is published because it is interesting, controversial, and addresses a larger issue that there is little consensus on--or else why publish it at all? And honest, humble scientists (both of them ) present their little studies as really just a glorified way of saying 'Hey, fellas, we did this to see what would happen--whaddya think? Are we headed in the right direction or not? Wanna do something silmilar to see if you get the same results?' So how is there any "rightness" or "wrongness" in that? At least someone is trying to figure out what may or may not be the answer. And at least someone is spreading the word.

The public (of which I am a proud member) needs to be reminded on a regular basis that a study is just a study and nothing to build your life around. Sure. But calling studies "wrong" or "right" or trying to summarize what "most" of them are is a question and answer of opinion on no higher a level than the studies they are attempting to put a label of opinion on.

The limits of the scientific method are what they are. But the solution isn't to ignore the questions or the opinions on what may or may not turn out to be the answer.

Now that I've "published" that opinion, it surely must be wrong. But I feel better.

[-SWS]

Does the statement "Most Published Scientific Research Is Wrong"
include reference to opinions published in this thread?

Well, the 2005 John P. A. Ioannidis essay entitled "Why Most Published Research Findings Are False" serves as the basis for the school of thought in that article cited by Rooster. When the loannidis essay was first published it made great waves in the general scientific community---inspiring plenty of debate and further consideration.

If we accept the premise that substantiated scientific research is very often wrong, then I think we can also logically assume that less-substantiated interpretive opinion is going to be wrong very often as well. Additionally, Critical Thinking as a traditional school of thought will embrace the fact that we inevitably maintain ample biases, blind spots, and incorrect assumptions---virtually guaranteeing that our interpretive opinions and even highly methodical analytic conclusions will be wrong on a very consistent basis.

Given the above critical reasoning criteria, it is irrational to conclude with absolute certainty that we are right or wrong, while formulating any given interpretive opinion that cannot be exhaustively substantiated as veritable fact. Additionally, it is irrational to conclude all that we believe to be verified fact is, indeed, properly substantiated fact.

Please bear in mind the above is only my interpretive opinion---be it right or wrong.

[jnk]

"I have not failed. I've just found 10,000 ways that won't work."
- Thomas Alva Edison

"Each failure brings you one step closer to success."
- Zig Ziglar

"For you to profit from your mistakes, you have to get out and make some."
- Anonymous

[Velbor]

"Provability is weaker notion than truth."
(Hofstadter, "Godel, Escher, Bach")

[jnk]

"Provability is weaker notion than truth."
(Hofstadter, "Godel, Escher, Bach")

That is true. But I can't prove it.

[-SWS]

"Provability is weaker notion than truth."
(Hofstadter, "Godel, Escher, Bach")

Truthability is a stronger notion when proven.

Indeed, we can know truths that we cannot prove. However, the fact that we cannot prove all truths that we happen to know, does not imply that we can know all truths or falsehoods.

[jnk]

"Provability is weaker notion than truth."
(Hofstadter, "Godel, Escher, Bach")

Truthability is a stronger notion when proven.

Indeed, we can know truths that we cannot prove. However, the fact that we cannot prove all truths that we happen to know, does not imply that we can know all truths or falsehoods.

"What is truth?"
--Pontius Pilate.

[-SWS]

Interesting that you should mention that, Jeff.

I was just thinking how science, religion, and philosophy are attempts to know the same truths... and how often those three have historically been at odds with one another. Those three very often embrace the perception of knowing versus proving differently. As a result their conclusions of truth frequently differ as well.