To the Editor:

We are responding to the letter by Pietrobon et al regarding the introduction of Levels of Evidence to the Journal. We appreciate their attention and interest. Before responding to their specific comments, it is important to reiterate the rationale for introducing Levels of Evidence to the Journal. Surgeons are inundated with newly published literature. The literature, however, varies in quality. Better quality research should be more likely to change clinical practice. Most surgeons have neither the time nor inclination to critically appraise all published articles.

Levels of Evidence ratings, albeit far from perfect, provide a filter whereby practicing surgeons can quickly evaluate published studies. Levels of Evidence also permits authors and readers to become more familiar with research design, to appreciate designs other than case series, and to allow the Journal to monitor trends in article quality.

Pietroban questions whether randomized clinical trials should be at the top of the hierarchy of Levels of Evidence. His first concern is that the results from randomized trials, by including a select group of consenting subjects, may not be relevant to those outside the trial. Research ethics restricts study to consenting participants, who are almost certainly different from those that don't consent, and this will always be an issue of concern. However, this same problem applies to both randomized controlled trials and to well-designed cohort studies (which also have explicit inclusion and exclusion criteria and require consent to participate) and thereby differ from RCTs mainly by lack of randomization.

Second, Dr Pietroban argues that well performed cohort or case control studies may provide results comparable to RCTs. This is a controversial topic and has not been observed in all comparisons. In any case,he raises a legitimate concern and we agree that RCTs may not always be the highest level evidence. Indeed, RCTs in the JBJS Levels of Evidence ratings may be either Level I or II, dependent on quality.

Despite these concerns, Levels of Evidence provides a generally accepted means of evaluating the medical and surgical literature. Although it does not replace a detailed critical appraisal of individual articles, we believe Levels of Evidence is an important and useful analytic tool for readers of the Journal. We look forward to continued discussions on the Evidence-Based initiatives of the Journal.

Yours truly,

James G. Wright, MD, MPH, FRCSC

To the Editor,

The recent decision of The Journal to add a "Levels of Evidence" classification to each clinical paper focusing on therapeutic evaluation is a laudable effort in the direction of having surgeons making therapeutic decisions based on evidence [Wright, 2003] . However, after more than ten years since the first proposal for Evidence-Based Medicine [ Evidence-based medicine working group, 1992 ] and numerous demonstrations that randomized controlled trials are not the only source of "evidence based medicine", we would agree with Wright et al., that using "Levels of Evidence" should not be a substitute for an "in-depth assessment" of each individual study.

In this letter, we wish to point to an additional feature that, although not specifically mentioned by Wright et al., is crucial to the proper evaluation of the quality of any clinical research study. Because the effect of the inclusion of a "Levels of Evidence" classification in The Journal is now starting to be reflected in journal clubs and surgeon decisions throughout the U.S. and the world, our letter is submitted in the spirit of constantly discussing how the best evidence should be evaluated.

When well conduted, randomized trials are able to reduce the confounding effect of extraneous variables on the association between treatment and outcome. Because treatments are randomly allocated, the difference in patient outcomes can be causally attributed to the different treatment modalities. But a less often recognized aspect of randomized studies is that in many situations investigators must exclude large groups of patients who have the condition of interest to make the study sample "homogenous." As a result, participants in a randomized study may not be representative of the patient population of interest to the clinician, and the study results may not be applicable. Such problems have been demonstrated in numerous studies [Kennedy, 2003] [Cunningham, 1995] [Fayers, 1997] [Susser, 1995]. Therefore, one might argue that the gold standard should be a well conducted non-randomized trial with a representative population.

But then a question could be raised of whether we can really trust the results of nonrandomized trials. Several studies have demonstrated that the differences in results between well-conducted randomized and non-randomized studies may not be as large as one would initially hypothesize. For example, Benson and Hartz [Benson, 2000] compared the results of observational studies and randomized controlled trials evaluating the treatment efficacy of the same diseases. After identifying observational studies by electronically searching MEDLINE and the Cochrane Library, the magnitudes of the treatment effects in the various observational studies were combined by the Mantel Haenszel or weighted analysis-of-variance procedure and compared with the combined magnitude of the treatment effects in the randomized controlled trials that evaluated the same treatment. The authors final sample involved 136 reports about 19 surgical and nonsurgical treatments, inlcuding calcium channel blocker therapy for coronary artery disease, appendectomy, and interventions for subfertility. In most cases, the estimates of the treatment effects from observational studies and randomized controlled trials were similar. In only 2 of the 19 analyses of treatment effects did the combined magnitude of the effect in observational studies lie outside the 95% confidence interval for the combined magnitude in the randomized controlled trials. The authors concluded that there was little evidence that estimates of treatment effects in observational studies either consistently larger than or qualitatively different from those obtained in randomized controlled trials. In a second example, Concato et al. ([Concato, 2000]) used published meta-analyses to identify randomized clinical trials and observational studies that examined the same clinical topics and then compared the results of the original reports according to the type of research design. For each of five topics chosen by the authors, summary estimates and 95% confidence intervals were calculated on the basis of data from the individual randomized controlled trials and the individual observational studies.The results indicated that for the 5 clinical topics and 99 reports evaluated, the average results of the observational studies were remarkably similar to those of the randomized controlled trials. The authors concluded that the results of well-designed observational studies do not systematically overestimate the magnitude of the effects of treatment as compared with randomized controlled trials on the same topic.

In conclusion, in spite of the "Levels of Evidence" classification being a nice proxy for studies with high internal validity, internal validity may not be the most important factor to be evaluated in a study. In addition, well-conducted non-randomized studies are not necessarily inferior when compared to randomized trials for the evaluation of surgical treatments and, under the circumstances where randomized designs may inherently distort results by investigating "the wrong" patient population, well-designed non-randomized studies may indeed represent the highest available "Level of Evidence" to guide therapeutic decisions. In other words, although a collection of static criteria may initially seem to be authoritative, it is important that these criteria be constantly discussed and reevaluated to avoid that applications without judicious judgment ultimately lead to misinterpretations and, consequently, inappropriate therapeutic and health policy decisions.

Bibliography

Wright JG. Swiontkowski MF. Heckman JD. Introducing levels of evidence to the journal. [Editorial] Journal of Bone & Joint Surgery - American Volume. 85-A(1):1-3, 2003 Jan.

Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA. 268 (1992), pp. 2420-2425.

Cunningham WE, Bozzette SA, Hays RD, Kanouse DE, Shapiro MF. Related Articles, Links Abstract Comparison of health-related quality of life in clinical trial and nonclinical trial human immunodeficiency virus-infected cohorts. Med Care. 1995 Apr;33(4 Suppl): AS15-25.

Fayers PM, Hand DJ. Related Articles, Links No abstract Generalisation from phase In clinical trials: survival, quality of life, and health economics. Lancet. 1997 Oct 4;350 (9083):1025-7.

Susser M. Related Articles, Links No abstract The tribulations of trials - intervention in communities. Am J Public Health. 1995 Feb;85(2):156-8

Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research design. N Engl J Med. 2000;342:1887-1892

Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342:1878-1886.

Ricardo Pietrobon, MD* Ulrich Guller, MD MHSc§ Laurence D. Higgins, MD* James A. Nunley 11, MD*

*Duke University Medical Center Division of Orthopaedic Surgery

§Basel University Basel, Switzerland

Corresponding Author:

Ricardo Pietrobon, MD Assistant Professor of Orthopaedic Surgery Duke University Medical Center Durham, NC 27710