To The Editor:

In their meta-analysis of the effectiveness of navigated total knee replacement, Bauwens et al.(1) found that navigation was associated with favorable results in terms of several radiographic parameters. The data were insufficient to evaluate effects on complication rates or functional outcomes. The article stimulated the above letter from Mason et al.(2) and a letter from Gregori and Holt(3), which prompted additional letters of clarification from Bauwens et al.(1).

Caught in the crossfire, readers might well ask why a meta-analysis led to such editorial dueling. Of note, controversy over meta-analysis is long-standing(4). The debates stem in part from the methodological complexity of meta-analysis, a powerful but challenging analytic technique that permits pooling of estimates across studies. We will discuss a few of the many methodological complexities of meta-analysis to put the correspondence about navigated total knee replacement in perspective.

Why Pool? Meta-Analysis Compared with Traditional Literature Review

If pooling raises so many questions, why bother to pool estimates quantitatively across studies? In many reviews, the authors simply array the findings of separate studies in evidence tables without attempting to synthesize them quantitatively into single estimates of effect. A key rationale for pooling is that the available evidence may consist of small studies that show positive (or negative) effects but lack power to establish the associations with significance. Pooling these smaller studies may avoid false-negative results due to Type-II error.

A useful example of this application of meta-analysis was provided by Felson and Anderson in a meta-analysis of the effect of cytotoxic therapy and corticosteroids compared with that of corticosteroids alone for patients with lupus nephritis(5). Prior small studies had suggested a beneficial effect of cytotoxic therapy. The meta-analysis overcame the small sample sizes of the component studies and illustrated the beneficial effect of cytotoxic therapy across studies.

Pooling also permits the investigator to examine whether particular study characteristics are associated with the principal outcome. This technique is termed metaregression. The investigator develops a regression model in which each study serves as a single observation, contributing a single estimate of outcome and of each covariate. The investigator can weight studies differentially in order to give greater importance in the regression to those that have larger sample sizes or that are of higher methodological quality. Metaregression can yield insights about sources of variability in outcome measures across studies. For example, it may be that trial designs are associated with larger effects and nonrandomized designs, with smaller effects, or vice versa.

Why Not Pool?

Pooling the results of separate studies into single estimates of effect involves several assumptions that frequently are not satisfied by the literature under review. Clearly, the outcome variable must be consistent across studies. This constraint poses no problem when the outcome is unambiguously defined, such as thirty-day all-cause mortality following hip replacement. However, when studies measure satisfaction, pain relief, functional status, and other such complex outcome variables, the task becomes more complicated. These domains are often measured with different tools in different studies, or different cutoffs are used to define success. For example, the authors of some studies of the outcome of total knee replacement might use the WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index) as the principal outcome measure whereas others might use the SF-36 (Short Form-36) or the Knee Society Scale. Attempting to synthesize results in these circumstances involves essentially combining apples and oranges and is not advisable. Standardization of outcome assessment and reporting in specific fields would assist investigators who wish to perform meta-analysis.

In addition, the underlying statistical methodology of meta-analysis assumes that each of the studies to be synthesized represents one observation from a single distribution of studies. This assumption is validated with tests of homogeneity of the odds ratios (or other effect estimates) across studies. If the group of studies to be synthesized appears to emanate from a single distribution, the homogeneity criterion is met and the studies may be synthesized in a meta-analysis. If, on the other hand, the assumption of homogeneity is not met, and the studies appear to be heterogeneous, then the investigators should be cautious about pooling. The investigators could simply choose not to pool the studies quantitatively. Alternatively, the investigators might wish to perform a metaregression to identify sources of heterogeneity. For example, it may be that higher-quality studies or a particular study design (e.g., trials) are associated with higher effect estimates.

What to Pool?

A meta-analysis is essentially an observational study of individual studies(6). As with all observational studies, the results are influenced by the selection criteria that dictate which studies are included in the meta -analysis and which are excluded. An issue that arises frequently, and was a major focus of contention about the paper by Bauwens et al.(1), is whether to include unpublished studies. Excluding unpublished studies risks publication bias, a form of selection bias in meta-analyses that arises because positive studies are, on the average, more likely to be published than negative studies. However, including unpublished studies that have not passed peer review risks the inclusion of studies with results that may not be credible.

Another important decision is whether to restrict the analysis to randomized controlled trials or to include observational designs. The advantage of restricting the analysis to randomized controlled trials is that randomization greatly reduces the risk of selection bias in each component study of the meta-analysis. Including observational studies permits the meta-analysis to simply propagate the biases inherent in the component studies. The disadvantage of restricting the sample to randomized controlled trials is that for many clinical problems, including navigated total knee replacement, there are few randomized controlled trials and most of the relevant literature includes observational designs.

Returning to Navigated Total Knee Replacement

Bauwens et al.(1) handled most of the above-mentioned issues with sophistication. They decided to pool because they were concerned that multiple underpowered studies would fail to establish an effect that might become apparent in a pooled analysis. They included nonrandomized trials because they were not comfortable restricting the analysis to randomized controlled trials. (An alternative approach would be to use metaregression to examine whether the magnitude of effect differed between randomized and observational studies; if it did, the meta-analysis could be done in subgroups.) The authors weighted the studies according to sample size and quality. They used appropriate analytic techniques to look for publication bias and, finding no evidence of such a bias, they restricted the analysis to published studies. In addition to stating the results of these analyses of publication bias, displaying the graphical evidence would have been helpful to readers.

Bauwens et al.(1) concluded that the studies that they wished to synthesize were heterogeneous. Having established heterogeneity, the authors could have simply decided not to pool the studies at all. Alternatively, they could have developed a metaregression model, which would have been useful in identifying and ultimately controlling for sources of heterogeneity. They could have stratified according to such characteristics and tested whether the stratified meta-analysis would have yielded less heterogeneity. The authors did indeed perform a metaregression, but they did not use it to identify strata in which studies were more homogeneous, as discussed here. By documenting heterogeneity and not doing anything about it, the authors in a sense, made a diagnosis without offering a remedy.

Data Sharing

Synthesizing the results of various studies is ultimately a collaborative activity. The investigator will often wish to contact other scientists who have access to original trial data or who themselves have attempted a data synthesis. These collaborations can help move the field forward. In fact, the National Institutes of Health (NIH) and other research sponsors have developed specific provisions for facilitating data sharing in order to best leverage the precious data garnered in NIH-funded studies. In this regard, we were particularly impressed by the willingness of Bauwens et al.(1) to share their data and we were disappointed that Mason et al.(2) chose to communicate their observations in a letter to The Journal without discussing the findings with the original authors. Readers, and ultimately patients, were not served well by this failure to behave collaboratively.

Concluding Remarks

The meta-analysis by Bauwens et al.(1) prompted questions about selection of studies, choice of common outcome measures across studies, assessment and management of heterogeneity, interpretation of results, and approaches to collaboration. The lessons learned from these studies of navigated total knee replacement are that investigators should make individual studies as definitive as possible by using the most rigorous designs feasible, powering studies adequately, and using standardized measures of outcome. Pooling is a powerful method for aggregating information across studies, but it is ultimately a collaborative effort. Leaders in the field should designate standard measures of outcome to facilitate pooling, and investigators should work collaboratively with one another so that data syntheses move the field forward, bringing quality and value to patients.

The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the authors, or a member of their immediate families, are affiliated or associated.

References:

1. Bauwens K, Matthes G, Wich M, Gebhard F, Hanson B, Ekkernhamp A, Stengel D. Navigated total knee replacement. A Meta-Analysis. J Bone Joint Surg Am 2007;89:261-269.

2. Bauwens K, Matthes G, Wich M, Gebhard F, Hanson B, Ekkernhamp A, Stengel D. Navigated total knee replacement. A Meta-Analysis. J Bone Joint Surg Am 2007;89:261-269. [Letter to The Editor] J Bone Joint Surg Am. epub 25 Jul 2007. http://www.ejbjs.org/cgi/eletters/89/2/261.

3. Bauwens K, Matthes G, Wich M, Gebhard F, Hanson B, Ekkernhamp A, Stengel D. Navigated total knee replacement. A Meta-Analysis. J Bone Joint Surg Am 2007;89:261-269. [Letter to The Editor] J Bone Joint Surg Am. epub 27 Mar 2007. http://www.ejbjs.org/cgi/eletters/89/2/261.

4. Goodman SN. Have you ever meta-analysis you didn't like? Ann Intern Med. 1991;114:244-6.

5. Felson DT, Anderson J. Evidence for the superiority of immunosuppressive drugs and prednisone over prednisone alone in lupus nephritis. Results of a pooled analysis. New Engl J Med. 1984;311:1528-33.

6. Kaizar EE. Metaanalyses are observational studies: how lack of randomization impacts analysis. Am J Gastroenterol. 2005;100:1233-6.

We read with great interest the letter from Dr. Mason and colleagues. Since they raised substantial concerns about the validity of our findings, we carefully reviewed the dataset that formed the basis for all analyses and figures presented in the Journal.

We reviewed our references 2-5 and found that there was no data shift between the conventional and navigated groups. This was unlikely, since the forest plots consistently showed an advantage for the navigated cohort.

Mason et al. also claimed that they found additional errors of data extraction from our references 6 to 9, but unless they are more specific in their criticisms, we cannot respond properly.

We would refer the Dr. Mason et al. to the Methods Section of our paper, where we stressed that the numbers of patients were extracted from histograms whenever possible. This may explain most differences eventually noted between their and our dataset. Additional differences might be related to different handling of the unit of interest, that is, the patient or the knee. Indeed, Bolognesi and Hofmann(1) reported the alignment of the femoral and the tibial component rather than the mechanical axis. However, if navigation improves both femoral and tibial component alignment, it is very likely that the resulting mechanical axis will be optimized as well. Since the observed effects were consistent with others, we decided to include the study in our analysis. We definitely identified and excluded some kinship studies, but could not retrieve a dual publication published by Mielke and colleagues(2).

When posing a null-hypothesis it is important to define the accepted standard of care. Risk ratios and other relative measures are asymmetric. This was the reason why we also provided risk differences, that can be used for calculating the number needed to treat. Currently, navigation is an experimental add-on, and may either decrease the risk of malalignment, or increase the chance of alignment. It is, however, not justified to argue that conventional surgery would increase the relative risk of malalignment over navigated component placement. With regard to health policy decisions, this is a dangerous statement, since it would imply that all patients who are not operated on with computer assistance are at a higher risk of malalignment when compared to those who undergo conventional TKA by an experienced surgeon.

Importantly, our analyses and plots showed a significant advantage of navigated over conventional knee replacement in radiological surrogates, so we are in complete agreement with Dr. Mason. Yet, unless these advantages are consistent with improved outcomes, we feel that our conclusion "Navigated knee replacement provides few advantages over conventional surgery on the basis of radiographic end points" is valid.

Finally, we regret that Dr. Mason, after receiving our dataset (which shows our openness and willingness to engage in scientific debate), did not contact us again to compare both datasets, and to discuss, explore and resolve any possible differences jointly before submitting a Letter to the Editor challenging our scientific reputation. We are sorry that Dr. Mason's group could not publish their paper, but we are deeply disappointed in their behavior.

References:

1. Bolognesi M, Hofmann A. Computer navigation versus standard instrumentation for TKA: a single-surgeon experience. Clin Orthop Relat Res. 2005;440:162-169.

2. Mielke RK, Clemens U, Jens JH, Kershally S. Navigation in knee endoprosthesis implantation-preliminary experiences and prospective comparative study with conventional implantatioin technique. Z Orthop Ihre Grenzgeb. 2001;139:109-116.

To The Editor:

We read with interest and concern the article, Navigated Total Knee Replacement: A Meta Analysis by Bauwens et al.(1). We submitted a similar meta-analysis to the Journal of Bone Surgery over one year ago, which was appropriately rejected for publication due to the inclusion of abstracts and uncontrolled case series data. The reviewers and editors also expressed concern that our finding of an advantage for navigated total knee arthroplasty (TKA) versus conventional TKA based on radiographic alignment endpoints needed to be balanced against the lack of evidence comparing the two procedures on cost-effectiveness, complication rates, and long term outcomes.

We were in the process of updating our meta-analysis in light of more recent publications (excluding abstract and uncontrolled case series data), when the study by Bauwens et al.(1) was published. Having reviewed essentially the same database, we were perplexed by the authors' conclusions that “navigated knee replacement provided few advantages over conventional surgery on the basis of radiographic endpoints”, as our own meta-analysis revealed a significant improvement in radiographic endpoints with computer-assisted navigation.

Our concerns about the discrepancies between our findings and those of Bauwens et al. prompted us to investigate their source data. We contacted them, and they graciously provided us with the raw data for all studies included in their meta-analysis. Upon further review, we discovered multiple inaccuracies of data extraction and/or data entry in their analysis:

In four of the studies reviewed in the Bauwens article(2-5) the data for conventional techniques was entered into the navigated data set for analysis while the data for the navigated set was entered under conventional techniques.

In four additional studies(6-9) we were able to determine errors of data extraction, data entry, patient count or patient group assignment.

One paper(10) was included and counted as reporting mechanical axis data when this was not reported in the study.

A kinship study (i.e., a study sharing overlapping data with an already included study) was included that should have been excluded(11).

There were two additional studies (12,13) in which the numbers we extracted were slightly different from those in Bauwens et al; we note these only as discrepancies (not errors) in extraction.

Our further review of their paper also suggests that their labeling and description of results was misleading. Specifically, they describe their meta-analyses as those of “relative risk of malalignment”, and label their figures accordingly. Yet, in the discussion, they state that “the available data suggest that navigation reduces the relative risk of 3 degrees of malalignment by 25%”. This statement is in error, because their meta-analysis was not of the relative risk of malalignment, but rather the relative risk of alignment, (i.e., the chance that a patient has alignment after the procedure). It would, therefore, have been accurate for them to state that conventional total knee arthroplasty decreases the relative chance of alignment by 25%. When misfit is the outcome of choice, instead of fit, the results are quite different from those reported by Bauwens et al. Correctly stated, the risk of malalignment is approximately three times that with conventional replacement relative to CAS.

In conclusion, our findings of data extraction and entry errors cause us to challenge the conclusions in the article regarding the meta-analysis of radiographic endpoints in conventional versus navigated knee replacement surgery. A correct data analysis demonstrates overwhelming evidence of a much lower error rate with navigation. Reversal of some of the extracted data and misreporting relative risks for fit as risks of malalignment is partially responsible for the muted difference that Bauwens described between navigated and conventional total knee arthroplasty. These errors, however, do not obviate Bauwens’ other discussion points regarding methodological limits of the available trials, including a dearth of evidence on long term outcomes, quality of life, and costs.

While we recognize and understand the challenges inherent in performing meta-analyses, our intent is to bring these errors to the attention of the readers of the Journal to correct any erroneous impression this work may have left with the readership.

In support of their research for or preparation of this work, one or more of the authors received, in any one year, outside funding or grants in excess of $10,000 from Depuy, and Johnson & Johnson. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the authors, or a member of their immediate families, are affiliated or associated.

REFERENCES:

1. Kai Bauwens, Gerrit Matthes, Michael Wich, Florian Gebhard, Beate Hanson, Axel Ekkernkamp, and Dirk Stengel Navigated Total Knee Replacement. A Meta-Analysis J Bone Joint Surg Am 2007; 89: 261-269

2. Bathis H, Perlick L, Tingart M, Luring C, Zurakowski D, Grifka J. Alignment in total knee arthroplasty. A comparison of computer- assisted surgery with the conventional technique. J Bone Joint Surg Br 2004; 86: 682-7.

3. Perlick L, Bathis H, Lerch K, Tingart M, Grifka J. [Navigated implantation of total knee endoprosthesis in secondary knee osteoarthritis of rheumatoid arthritis patients as compared to conventional technique. Z Rheumatol 2004; 63: 140-6.

4. Saragaglia D, Picard F, Chaussard C, Montbarbon E, Leitner F, Cinquin P. [Computer-assisted knee arthroplasty: comparison with a conventional procedure. Results of 50 cases in a prospective randomized study]. Rev Chir ORthop Reparatrice Appar Mot 2001; 87: 18-28.

5. Sparmann M, Wolke B, Czupalla H, Banzer D, Zink A. Positioning of total knee arthroplasty with and without navigation support. A prospective, randomised study. J Bone Joint Surg Br 2003; 85(6): 830-5.

6. Chauhan SK, Scott RG, Breidahl W, Beaver RJ. Computer-assisted knee arthroplasty versus a conventional jig-based technique. A randomized, prospective trial. J Bone Joint Surg Br 2004; 86(3): 372-7.

7. Confalonieri N, Manzotti A, Pullen C, Ragone V. Computer- assisted technique versus intramedullary and extramedullary alignment systems in total knee replacement: a radiological comparison. Acet Orthop Belg 2005; 71: 703-9.

8. Kim SJ, Macdonald M, Hernandex J, Wixson RL. Computer assisted navigation in total knee arthroplasty: improved coronal alignment. J Arthroplasty 2005; 20: 123-31.

9. Perlick L, Bathis H, Tingart M, Perlick C, Grifka J. Navigation in total-knee arthroplasty: CT-based implantation compared with the conventional technique. Acta Orthop Scand 2004; 75: 464-70.

10. Bolognesi M, Hofmann A. Computer navigation versus standard instrumentation for TKA: a single-surgeon experience. Clin Orthop Relat Res 2005; 440: 162-9.

11. Mielke RK, Clemens U, Jens JH, Kershally S. [Navigation in knee endoprosthesis implantation—preliminary experiences and prospective comparative study with conventional implantation technique]. Z Orthop Ihre Grenzgeb 2001: 139: 109-16.

12. Anderson KC, Buehler KC, Markel DC. Computer assisted navigation in total knee arthroplasty: comparison with conventional methods. J Arthroplasty. 2005; 20(7 suppl 3): 132-8.

13. Haaker RG, Stockheim M, Kamp M, Proff G, Breitenfelder J, Ottersbach A. Computer-assisted navigation increases precision of component placement in total knee arthroplasty. Clin Orthop Relat Res. 2005; 27: 152-9.

We read with great interest the comments of Alberto Gregori and Graeme Holt on our meta-analysis. We believe all the issues they raise were clearly addressed in the printed article and the electronic appendix, but we will be happy to respond to their letter in a point-to-point fashion.

1. We do not agree that the conclusion of the abstract conflicts with current best evidence. Most trials focused on alignment, not function, quality of life, or cost. We feel that all would agree that higher precision in restoring the physiological limb axis is an advantage of navigated over conventional total knee replacement, but patient-centered and health-economic values have more weight in clinical and political decision making. In the Discussion, we stressed the need for high-quality trials aiming at investigating clinically relevant outcomes.

2. Meta-analyses (especially in orthopedics) are often criticized for including only RCT, thereby limiting the external validity of the results. We are very much aware of the discrepancy between methodological and clinical demands. In the methods section, we clearly pointed out that we conducted a meta-regression analysis to account for different study designs. There was no meaningful difference in effect estimates between RCT and other study settings.

All key features of our search strategy were mentioned in the methods section. Specifically, we (i) reported all databases searched, (ii) tried diligently to avoid a tower of Babel bias by including studies of all languages, (iii) did a manual search, (iv) reported the study selection in a QUOROM flow-chart, (v) assessed methodological features by two or more independent raters, (vi) tested for publication bias and statistical heterogeneity. If we had missed any important quality criterion of a valid meta-analysis (or a relevant paper that contradicts our findings), we would be pleased to be informed by Drs.Gregori and Holt.

4. In the Discussion, we admitted the limits of the chosen endpoints- however, as indicated in their letter, this was not a shortcoming of the quantitative summary, but the lack of reporting of other endpoints in the original manuscripts.

Dr. Gregori and Dr. Holt conclude that navigated total knee arthroplasty improves implant alignment, but consequent improved implant survival remains unproven. We are happy about this conclusion, since it perfectly agrees with the findings of our meta-analysis.

To The Editor:

In their recent meta-analysis(1), Bauwens et al. concluded that “navigated knee replacement provides few advantages……on the basis of radiographic end points”. However, our analysis of this paper suggests that this conclusion is invalid.

While meta-analysis of randomised controlled trials represents the gold standard in validation of surgical interventions, overcoming the reduced statistical power of small sample sizes, it cannot compensate for poor scientific methodology in the analyzed papers. The authors (1) included not only randomised, but also quasi-randomized controlled trials, non- randomized cohort studies, studies with historical cohorts, and studies investigating the outcome of CT or image-free navigation systems for both unicompartmental and total knee arthroplasty.

A meta-analysis must use a predefined, documented search strategy allowing assessment of its completeness; this was not reported. “Mean straightness of mechanical axes” is an inappropriate outcome measure. The mean mechanical axis says nothing about the distribution of values that it represents without reporting standard deviations and range, though 95% confidence intervals were stated. However, two groups may have significantly different distributions of alignment values centered about similar mean values.

Navigation reduces the number of implants with a predetermined variance from the true mechanical axis, commonly defined as ±3o. The authors estimate a risk ratio of a deviation of >3° with navigated versus conventional knee arthroplasty at 0.79 and 0.76 for a threshold of 2o. Navigation reduced the relative risk of >3° malalignment by 25% thus avoiding one additional patient with unfavorable component positioning in any five patients managed with computer-assisted instead of jig-based methods.

The authors conclude that “the benefits of navigation diminished rapidly with increasing thresholds of proper implant positioning”. If we were to accept a deviation of up to 6 degrees from the true mechanical axis then both conventional jig and navigation based arthroplasty are almost equally efficacious; however, this degree of error is greater than most arthroplasty surgeons would accept.

Navigated total knee arthroplasty improves implant alignment, but consequent improved implant survival remains unproven. We are concerned that this meta-analysis(1) will be regarded by many as definitive evidence even though its methodological shortcomings and interpretation of results do not justify the conclusions reached.

The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. A commercial entity (Biomet & BBraun) paid or directed in any one year, or agreed to pay or direct, benefits in excess of $10,000 to a research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the authors, or a member of their immediate families, are affiliated or associated.

Reference:

1. Bauwens K, Matthes G, Wich M, Gebhard F, Hanson B, Ekkernkamp A, Stengel D. Navigated total knee replacement - A meta-analysis. J Bone Joint Surg Am. 2007;89(2):261-9.