Extract
In their meta-analysis of the effectiveness of navigated total knee
replacement, Bauwens et al. found that navigation was associated with
favorable results in terms of several radiographic parameters. The data were
insufficient to evaluate effects on complication rates or functional outcomes.
The article stimulated the above letter from Mason et al. and a letter from
Gregori and Holt13,
which prompted additional letters of clarification from Bauwens et al.Caught in the crossfire, readers might well ask why a meta-analysis led to
such editorial dueling. Of note, controversy over meta-analysis is
long-standing14.
The debates stem in part from the methodological complexity of meta-analysis,
a powerful but challenging analytic technique that permits pooling of
estimates across studies. We will discuss a few of the many methodological
complexities of meta-analysis to put the correspondence about navigated total
knee replacement in perspective.
In their meta-analysis of the effectiveness of navigated total knee
replacement, Bauwens et al. found that navigation was associated with
favorable results in terms of several radiographic parameters. The data were
insufficient to evaluate effects on complication rates or functional outcomes.
The article stimulated the above letter from Mason et al. and a letter from
Gregori and Holt13,
which prompted additional letters of clarification from Bauwens et al.
Caught in the crossfire, readers might well ask why a meta-analysis led to
such editorial dueling. Of note, controversy over meta-analysis is
long-standing14.
The debates stem in part from the methodological complexity of meta-analysis,
a powerful but challenging analytic technique that permits pooling of
estimates across studies. We will discuss a few of the many methodological
complexities of meta-analysis to put the correspondence about navigated total
knee replacement in perspective.
If pooling raises so many questions, why bother to pool estimates
quantitatively across studies? In many reviews, the authors simply array the
findings of separate studies in evidence tables without attempting to
synthesize them quantitatively into single estimates of effect. A key
rationale for pooling is that the available evidence may consist of small
studies that show positive (or negative) effects but lack power to establish
the associations with significance. Pooling these smaller studies may avoid
false-negative results due to Type-II error.
A useful example of this application of meta-analysis was provided by
Felson and Anderson in a meta-analysis of the effect of cytotoxic therapy and
corticosteroids compared with that of corticosteroids alone for patients with
lupus nephritis15.
Prior small studies had suggested a beneficial effect of cytotoxic therapy.
The meta-analysis overcame the small sample sizes of the component studies and
illustrated the beneficial effect of cytotoxic therapy across studies.
Pooling also permits the investigator to examine whether particular study
characteristics are associated with the principal outcome. This technique is
termed metaregression. The investigator develops a regression model
in which each study serves as a single observation, contributing a single
estimate of outcome and of each covariate. The investigator can weight studies
differentially in order to give greater importance in the regression to those
that have larger sample sizes or that are of higher methodological quality.
Metaregression can yield insights about sources of variability in outcome
measures across studies. For example, it may be that trial designs are
associated with larger effects and nonrandomized designs, with smaller
effects, or vice versa.
Pooling the results of separate studies into single estimates of effect
involves several assumptions that frequently are not satisfied by the
literature under review. Clearly, the outcome variable must be consistent
across studies. This constraint poses no problem when the outcome is
unambiguously defined, such as thirty-day all-cause mortality following hip
replacement. However, when studies measure satisfaction, pain relief,
functional status, and other such complex outcome variables, the task becomes
more complicated. These domains are often measured with different tools in
different studies, or different cutoffs are used to define success. For
example, the authors of some studies of the outcome of total knee replacement
might use the WOMAC (Western Ontario and McMaster Universities Osteoarthritis
Index) as the principal outcome measure whereas others might use the SF-36
(Short Form-36) or the Knee Society Scale. Attempting to synthesize results in
these circumstances involves essentially combining apples and oranges and is
not advisable. Standardization of outcome assessment and reporting in specific
fields would assist investigators who wish to perform meta-analysis.
In addition, the underlying statistical methodology of meta-analysis
assumes that each of the studies to be synthesized represents one observation
from a single distribution of studies. This assumption is validated with tests
of homogeneity of the odds ratios (or other effect estimates) across studies.
If the group of studies to be synthesized appears to emanate from a single
distribution, the homogeneity criterion is met and the studies may be
synthesized in a meta-analysis. If, on the other hand, the assumption of
homogeneity is not met, and the studies appear to be heterogeneous, then the
investigators should be cautious about pooling. The investigators could simply
choose not to pool the studies quantitatively. Alternatively, the
investigators might wish to perform a metaregression to identify sources of
heterogeneity. For example, it may be that higher-quality studies or a
particular study design (e.g., trials) are associated with higher effect
estimates.
A meta-analysis is essentially an observational study of individual
studies16. As with
all observational studies, the results are influenced by the selection
criteria that dictate which studies are included in the meta-analysis and
which are excluded. An issue that arises frequently, and was a major focus of
contention about the paper by Bauwens et al., is whether to include
unpublished studies. Excluding unpublished studies risks publication bias, a
form of selection bias in meta-analyses that arises because positive studies
are, on the average, more likely to be published than negative studies.
However, including unpublished studies that have not passed peer review risks
the inclusion of studies with results that may not be credible.
Another important decision is whether to restrict the analysis to
randomized controlled trials or to include observational designs. The
advantage of restricting the analysis to randomized controlled trials is that
randomization greatly reduces the risk of selection bias in each component
study of the meta-analysis. Including observational studies permits the
meta-analysis to simply propagate the biases inherent in the component
studies. The disadvantage of restricting the sample to randomized controlled
trials is that for many clinical problems, including navigated total knee
replacement, there are few randomized controlled trials and most of the
relevant literature includes observational designs.
Bauwens et al. handled most of the above-mentioned issues with
sophistication. They decided to pool because they were concerned that multiple
underpowered studies would fail to establish an effect that might become
apparent in a pooled analysis. They included nonrandomized trials because they
were not comfortable restricting the analysis to randomized controlled trials.
(An alternative approach would be to use metaregression to examine whether the
magnitude of effect differed between randomized and observational studies; if
it did, the meta-analysis could be done in subgroups.) The authors weighted
the studies according to sample size and quality. They used appropriate
analytic techniques to look for publication bias and, finding no evidence of
such a bias, they restricted the analysis to published studies. In addition to
stating the results of these analyses of publication bias, displaying the
graphical evidence would have been helpful to readers.
Bauwens et al. concluded that the studies that they wished to synthesize
were heterogeneous. Having established heterogeneity, the authors could have
simply decided not to pool the studies at all. Alternatively, they could have
developed a metaregression model, which would have been useful in identifying
and ultimately controlling for sources of heterogeneity. They could have
stratified according to such characteristics and tested whether the stratified
meta-analysis would have yielded less heterogeneity. The authors did indeed
perform a metaregression, but they did not use it to identify strata in which
studies were more homogeneous, as discussed here. By documenting heterogeneity
and not doing anything about it, the authors in a sense made a diagnosis
without offering a remedy.
Synthesizing the results of various studies is ultimately a collaborative
activity. The investigator will often wish to contact other scientists who
have access to original trial data or who themselves have attempted a data
synthesis. These collaborations can help move the field forward. In fact, the
National Institutes of Health (NIH) and other research sponsors have developed
specific provisions for facilitating data sharing in order to best leverage
the precious data garnered in NIH-funded studies. In this regard, we were
particularly impressed by the willingness of Bauwens et al. to share their
data and we were disappointed that Mason et al. chose to communicate their
observations in a letter to The Journal without discussing the
findings with the original authors. Readers, and ultimately patients, were not
served well by this failure to behave collaboratively.
The meta-analysis by Bauwens et al. prompted questions about selection of
studies, choice of common outcome measures across studies, assessment and
management of heterogeneity, interpretation of results, and approaches to
collaboration. The lessons learned from these studies of navigated total knee
replacement are that investigators should make individual studies as
definitive as possible by using the most rigorous designs feasible, powering
studies adequately, and using standardized measures of outcome. Pooling is a
powerful method for aggregating information across studies, but it is
ultimately a collaborative effort. Leaders in the field should designate
standard measures of outcome to facilitate pooling, and investigators should
work collaboratively with one another so that data syntheses move the field
forward, bringing quality and value to patients.
Bäthis H, Perlick L, Tingart M,
Lüring C, Zurakowski D, Grifka J. Alignment in total knee arthroplasty. A
comparison of computer-assisted surgery with the conventional technique.
J Bone Joint Surg Br.
2004;86:
682-7.86682
2004
[PubMed][CrossRef]
Perlick L, Bäthis H, Lerch K,
Lüring C, Tingart M, Grifka J. [Navigated implantation of total knee
endoprostheses in secondary knee osteoarthritis of rheumatoid arthritis
patients as compared with conventional technique]. Z Rheumatol.
2004;63: 140-6.
German.63140
2004
[PubMed][CrossRef]
Saragaglia D, Picard F, Chaussard C,
Montbarbon E, Leitner F, Cinquin P. [Computer-assisted knee arthroplasty:
comparison with a conventional procedure. Results of 50 cases in a prospective
randomized study]. Rev Chir Orthop Reparatrice Appar Mot.
2001;87: 18-28.
French.8718
2001
[PubMed]
Sparmann M, Wolke B, Czupalla H, Banzer
D, Zink A. Positioning of total knee arthroplasty with and without navigation
support. A prospective, randomised study. J Bone Joint Surg Br.
2003;85:
830-5.85830
2003
[PubMed]
Chauhan SK, Scott RG, Breidahl W, Beaver
RJ. Computer-assisted knee arthroplasty versus a conventional jig-based
technique. A randomised, prospective trial. J Bone Joint Surg
Br. 2004;86:
372-7.86372
2004
[CrossRef]
Confalonieri N, Manzotti A, Pullen C,
Ragone V. Computer-assisted technique versus intramedullary and extramedullary
alignment systems in total knee replacement: a radiological comparison.
Acta Orthop Belg. 2005;71:
703-9.71703
2005
[PubMed]
Kim SJ, MacDonald M, Hernandez J, Wixson
RL. Computer assisted navigation in total knee arthroplasty: improved coronal
alignment. J Arthroplasty.
2005;20(7 Suppl 3):
123-31.20123
2005
[PubMed][CrossRef]
Perlick L, Bäthis H, Tingart M,
Perlick C, Grifka J. Navigation in total-knee arthroplasty: CT based
implantation compared with the conventional technique. Acta Orthop
Scand. 2004;75:
464-70.75464
2004
[CrossRef]
Bolognesi M, Hofmann A. Computer
navigation versus standard instrumentation for TKA: a single-surgeon
experience. Clin Orthop Relat Res.
2005;440:
162-9.440162
2005
[PubMed][CrossRef]
Mielke RK, Clemens U, Jens JH, Kershally
S. [Navigation in knee endoprosthesis implantation—preliminary
experiences and prospective comparative study with conventional implantation
technique]. Z Orthop Ihre Grenzgeb.
2001:139: 109-16.
German.139109
2001
[PubMed][CrossRef]
Anderson KC, Buehler KC, Markel DC.
Computer assisted navigation in total knee arthroplasty: comparison with
conventional methods. J Arthroplasty.
2005;20(7 Suppl 3):
132-8.20132
2005
[PubMed][CrossRef]
Haaker RG, Stockheim M, Kamp M, Proff G,
Breitenfelder J, Ottersbach A. Computer-assisted navigation increases
precision of component placement in total knee arthroplasty. Clin
Orthop Relat Res. 2005;433:
152-9.433152
2005
[CrossRef]
Goodman SN. Have you ever meta-analysis
you didn't like? Ann Intern Med.
1991;114:
244-6.114244
1991
[PubMed]
Felson DT, Anderson J. Evidence for the
superiority of immunosuppressive drugs and prednisone over prednisone alone in
lupus nephritis. Results of a pooled analysis. New Engl J Med.
1984;311:
1528-33.3111528
1984
[PubMed][CrossRef]
Kaizar EE. Metaanalyses are
observational studies: how lack of randomization impacts analysis. Am J
Gastroenterol. 2005;100:
1233-6.1001233
2005
[CrossRef]