For Science's Gatekeepers, a Credibility Gap

By Aussiegirl

An interesting and in-depth article on the problems with the famed peer-review system.

For Science's Gatekeepers, a Credibility Gap - New York Times

For Science's Gatekeepers, a Credibility Gap
By Lawrence K. Alttman, M.D.
Published: May 2, 2006

Recent disclosures of fraudulent or flawed studies in medical and scientific journals have called into question as never before the merits of their peer-review system.

The system is based on journals inviting independent experts to critique submitted manuscripts. The stated aim is to weed out sloppy and bad research, ensuring the integrity of the what it has published.

Because findings published in peer-reviewed journals affect patient care, public policy and the authors' academic promotions, journal editors contend that new scientific information should be published in a peer-reviewed journal before it is presented to doctors and the public.

That message, however, has created a widespread misimpression that passing peer review is the scientific equivalent of the Good Housekeeping seal of approval.

Virtually every major scientific and medical journal has been humbled recently by publishing findings that are later discredited. The flurry of episodes has led many people to ask why authors, editors and independent expert reviewers all failed to detect the problems before publication.

The publication process is complex. Many factors can allow error, even fraud, to slip through. They include economic pressures for journals to avoid investigating suspected errors; the desire to avoid displeasing the authors and the experts who review manuscripts; and the fear that angry scientists will withhold the manuscripts that are the lifeline of the journals, putting them out of business.By promoting the sanctity of peer review and using it to justify a number of their actions in recent years, journals have added to their enormous power. [....]

However, even the system's most ardent supporters acknowledge that peer review does not eliminate mediocre and inferior papers and has never passed the very test for which it is used. Studies have found that journals publish findings based on sloppy statistics. If peer review were a drug, it would never be marketed, say critics, including journal editors. [....]

Journals have rejected calls to make the process scientific by conducting random audits like those used to monitor quality control in medicine. The costs and the potential for creating distrust are the most commonly cited reasons for not auditing.

In defending themselves, journal editors often shift blame to the authors and excuse themselves and their peer reviewers.

Journals seldom investigate frauds that they have published, contending that they are not investigative bodies and that they could not afford the costs. Instead, the journals say that the investigations are up to the accused authors' employers and agencies that financed the research.

Editors also insist that science corrects its errors. But corrections often require whistle-blowers or prodding by lawyers. Editors at The New England Journal of Medicine said they would not have learned about a problem that led them to publish two letters of concern about omission of data concerning the arthritis drug Vioxx unless lawyers for the drug's manufacturer, Merck, had asked them questions in depositions. Fraud has also slipped through in part because editors have long been loath to question the authors. [....]

Despite its flaws, scientists favor the system in part because they need to publish or perish. The institutions where the scientists work and the private and government agencies that pay for their grants seek publicity in their eagerness to show financial backers results for their efforts. [....]