People who've grown sceptical of the hyping of the
latest medical breakthrough got some validation this week:
A new review found one out of every three highly cited studies
published in influential medical journals is either refuted
or seriously weakened by subsequent research.
Some notable examples: the initial finding that hormone-replacement
therapy reduced older women's heart disease risk (a
larger trial later found that HRT actually raised cardiovascular
and cancer risk); the promise of vitamin E in preventing
heart attack (it doesn't); and the news that vitamin
A supplements cut breast cancer risk (again, a larger, randomised
trial showed no effect).
None of this means the initial, promising trial shouldn't
have been published or publicised, stressed Dr John Ioannidis,
author of the review appearing in the July 13 issue of the
Journal of the American Medical Association.
'This is largely the scientific process at work'
"This is largely the scientific process at work," said Ioannidis,
an associate professor of medicine at the Institute for
Clinical Research and Health Policy Studies at Tufts-New
England Medical Centre, in Boston. "There is nothing wrong
with publishing promising evidence."
Ioannidis, who is also chairman of the department of hygiene
and epidemiology at the University of Ioannina School of
Medicine in Greece, said the message for the public is "to
not lose your trust [in science], but don't expect that
everything you hear about will be true forever."
How the analysis was done
In his analysis, Ioannidis examined the long-term validity
of 49 of the most widely-cited studies published between
1990 and 2003 in three of the world's most prestigious
"generalist" medical journals - the Journal of the American
Medical Association, The Lancet, and the New England
Journal of Medicine - as well as leading "specialty"
journals such as Circulation, Diabetes and the
Journal of the National Cancer Institute.
He then tracked whether or not the main findings from
the 49 studies held true over time. Thirty-nine of the studies
were randomised - meaning that participants were randomly
assigned to one of two or more treatment arms in a clinical
trial. Large, randomised trials conducted prospectively
(i.e., unfolding over time as opposed to a review of past
data) are considered the gold standard of clinical research.
According to Ioannidis, "five out of six non-randomised
studies and nine out of 39 randomised studies were contradicted
or found to have stronger effects compared with subsequent
studies on the same topic." Smaller studies were more likely
to be refuted than larger ones, and findings from non-randomised
trials were often overturned by the results of a larger,
32 percent of findings didn't hold
In total, 16 percent of the trial studies ended up being
contradicted by later research, and findings from another
16 percent were considerably weakened by later, more reliable
Ioannidis said his goal was to reach an "objective estimate"
of the pace of change in current medical theory. "I think
it is healthy thinking to acknowledge that scientific knowledge
is not immutable and final, but is likely to change over
time," he said.
He added that none of his findings cast doubt on the integrity
of most medical journals.
"These journals have high scientific standards, robust
editorial processes, and they are independent," Ioannidis
said. "We just have to accept that contradictions will occur,
and that we should not consider that science is cut in stone."
'A single study is not the final word'
Editors at the New England Journal of Medicine
agreed. In a group statement, they said the conclusion of
Ioannidis' review is "widely appreciated: while many
published studies are subsequently confirmed, some are not.
This is how medical and scientific research progresses.
A single study is not the final word, and that is an important
But does the media - aided by drug companies eager to
capitalize on positive findings - too often hype and simplify
the results of new medical research?
"This may be a problem occasionally," Ioannidis said.
"The general public should be sensitised to the fact that
scientific knowledge has limitations, it is never final
and things may change down the road."
Jeff Trewhitt, a spokesman for the Pharmaceutical Research
and Manufacturers of America, representing the drug industry,
agreed that "there's always new information as the product
is introduced to a larger patient population." But he also
believes "the evidence is clear that in most situations,
new data tends to confirm the initial findings."
Trewhitt also noted that, besides having to pass through
the journal editorial process prior to publication, clinical
trials are also designed "to meet the tough testing standards
at the FDA."
"We believe we are not routinely seeing data that would
change the indicated use of the product as approved by the
FDA," he added.
Too much focus on positive findings?
Ioannidis did have one criticism for major journals, however:
their tendency to publish "positive" findings (where a therapy
was proven to be effective) over "negative" ones (where
a therapy's effectiveness was cast in doubt).
"There's nothing wrong with publishing promising evidence,"
he said. "What is wrong is not publishing less promising
evidence and studies with 'negative' results when
[the studies] are equally well-designed and conducted as
those that find impressive effects."
Still, Ioannidis seemed reassured by the findings. "Scientific
advances do happen, medical progress is moving at a fast
speed," he said. In interpreting medical news that could
potentially impact health, Ioannidis advises people to "make
decisions about it, try to find out what the limitations
and caveats might be, and try and get a sense from your
physician or physicians about the surrounding uncertainty.
Critical thinking is always useful."