To The Editor:
We have been encouraged by the explicit emphasis on evidence-based orthopaedics
in The Journal, and we read with great interest "Meta-Analyses
in Orthopaedic Surgery. A Systematic Review of Their Methodologies" (83-A:
15-24, Jan. 2001), by Bhandari et al. This article raised some important
points, one being that adherence to strict scientific methodology can
limit bias and improve the validity of meta-analyses. However, we
would like to comment on the methodology and consequent findings
of their study.
The first issue is their omission of relevant studies published
in the Cochrane Database of Systematic Reviews. As Bhandari et al.
stated, they consulted this database in their search for meta-analyses,
yet some thirty to fifty systematic reviews available in this database
in 1999 were not included, which we find puzzling. Omission of these reviews,
which, if included, would have more than doubled the number of reviews
of fracture treatment, results in a distorted picture of the information that
is available to the orthopaedic specialist. Moreover, as acknowledged
by Bhandari et al., there is evidence that these reviews have higher
scientific quality than those published elsewhere1.
Unfortunately, the article failed to provide the full list of
studies included, although it may be that spatial limitations precluded
this. The electronic search risked missing studies from journals
not listed in Medline but available in other readily accessible
databases, such as EMBASE. Inference from Table I indicates that
at least two additional relevant studies2,3 were
missed.
A third issue is partly one of definition. The emphasis by Bhandari
et al. on meta-analyses is understandable but is likely to have
resulted in a misleading picture. We believe that the correct emphasis
should be on systematic reviews, with the option, when appropriate,
of pooling data from comparable trials with use of appropriate statistical techniques.
M. Bhandari, A.V. Kulkarni, and P. Tornetta III reply:
We thank Dr. Parker and colleagues from the Cochrane Collaboration
Musculoskeletal Injuries Collaborative Review Group for their letter.
We are glad to see that they agree with us in stressing the importance
of using proper methodology in conducting meta-analyses.
Dr. Parker and colleagues are correct in pointing out that we
chose not to include meta-analyses from the Cochrane Database of
Systematic Reviews unless they also were published in a journal.
Therefore, it is not at all puzzling why many of the meta-analyses
that they refer to were not included in our paper. These meta-analyses
simply did not meet the very transparent eligibility criteria that
we outlined in the Materials and Methods section, one of which was
that all meta-analyses "had to have been published or accepted
for publication." We stated in the same section of the
article: "The Cochrane Database for Systematic Reviews
was also searched to identify any additional studies that may have been
published in the orthopaedic literature." We understand
that Parker et al. may not agree with our decision to limit ourselves
to just the meta-analyses that had been published in journals. However,
it is important to understand that, from the outset of the study,
our specific goal was to identify the meta-analyses in the field
of orthopaedics that we believed were most likely to be read by orthopaedic
surgeons. Judging from the responses of many of our colleagues, we
felt that a fair representation of these would be from published
journals only. We did not intend for this decision, in any way,
to diminish the importance of other high-quality meta-analyses,
such as those found in the Cochrane Database. In fact, as noted
by Parker et al., we took pains to state very clearly in the Discussion
section that meta-analyses in the Cochrane Database of Systematic
Reviews "have been shown to be higher in scientific quality than
were meta-analyses published in other sources."
In regard to our search strategy, we utilized the Medline database,
searched the proceedings of major meetings of orthopaedic societies,
consulted textbooks and the opinions of content experts, and conducted
labor-intensive hand searches of several journals. The criticism
of Parker et al. concerning our omission of an EMBASE database search
is, however, a valid one. Regardless, we do believe that our sample
of meta-analyses is still representative of the existing body of
such meta-analyses. In fact, we took the liberty of retrieving the
two meta-analyses identified by Dr. Parker and colleagues and found
that their quality scores (as judged by two reviewers) were well within
the range of those for the studies in our original analysis (mean
and standard deviation for the overall quality score of all forty
meta-analyses = 4.2 ± 1.78). It
should also be noted that our original manuscript submission did include
a list of all of the meta-analyses that we included. However, efforts
to limit the size of the manuscript during the editorial process
led to the omission of that list from the final, published version
of the article.
Finally, we agree with Dr. Parker and colleagues that, in conducting
a systematic review, statistical pooling should only be performed
when appropriate. We thank them for emphasizing this point in their
letter. We believe that the comparability of studies, as Dr. Parker and
colleagues point out, is related to many factors, including similarity
of the point estimates of treatment effects, widely overlapping
confidence intervals between studies, nonsignificant results of
tests of heterogeneity, and clinical relevance. Statistical pooling may
be indicated if clinicians can expect more or less the same treatment effect
across the range of populations, interventions, outcomes, and methodologies.