Dr. Pedowitz has described some of the issues of the AAOS evidence-based clinical practice guidelines (CPGs) that have concerned a number of Academy members and deserve some clarification.
The first issue is the selection of evidence for the AAOS CPGs. In the AAOS guidelines process, there is no restriction of evidence to Level-I studies. All AAOS clinical practice guidelines start with preliminary questions developed by a Work Group composed of clinical expert volunteers from the AAOS and from other medical and surgical professional societies so that all of the interests of the patient may be included. The experts develop the questions, and a complete systematic literature review is performed to address them. To prevent bias when members of the Work Group do not like the answers, questions cannot be changed once they are finalized. The CPG process is guided by the questions posed by the Work Group. Some guidelines have asked more clinically relevant and practice-relevant questions than others. The AAOS is developing a broader and more inclusive method to allow more members and other stakeholders to be involved in the formulation of questions for the guideline process.
When answering the questions, the best available evidence is used to find the answers. When high-quality studies are available, lower-quality studies are not used. When higher-quality studies are not available, lower-quality studies are used (with the exception of retrospective noncomparative case series, which are not used in the guideline process). Even prospective, Level-IV case series are included as evidence despite their inherent weaknesses. Even the experts are often surprised that the evidence for some of our cherished treatments is weak or nonexistent. A number of AAOS members have been upset that their favored or popular treatments have not withstood scrutiny of the literature.
Retrospective case series are common in orthopaedics. Unfortunately, this study design has substantial limitations in terms of the quality and reliability of the data. Typically, we cannot be certain how patients were selected for one treatment versus another treatment (selection bias), whether the data were reliably found in the charts, or what the patients’ actual functional results were. By their design, case series cannot compare one treatment with another and, because of differences in patient populations, and it is usually impossible to compare patients or outcomes across various series. Case series are also prone to the problem of “regression to the mean.”1 Because patients typically seek medical care when their symptoms are most severe, they also typically enroll in studies at that time. However, in many conditions, symptoms wax and wane over time. Because patients see physicians when symptoms are at their worst, they are highly likely to improve simply because their symptoms regress toward the mean.
In noncomparative studies, it is not possible to determine whether one treatment is better than another except in unusual circumstances when the results of one treatment are clearly disastrous. Trying to compare various case series is also difficult or even impossible because different series may represent different populations, may use different outcomes, may include patients who were entered into the study at different points in the disease, may have different lengths of follow-up, and may define important variables such as malunion or nonunion differently. Regression to the mean prevents us from knowing whether the treatment presented in a case series even improves on the natural history unless the natural history is extremely well defined, which is not the case for most orthopaedic disorders. While these studies can be used to generate hypotheses for higher-quality studies, their substantial limitations cause them to provide very little evidence for treatment decisions unless substantial complications render the treatment potentially dangerous.
Second, Dr. Pedowitz indicates the importance of expert opinion in orthopaedic practice. Expert opinion needs to be placed in proper perspective. Many of our experts may have undisclosed conflicts of interest, may see a different group of patients than we do, or may value certain outcomes more than our patients do, which may bias their perspectives. While their clinical advice may be helpful, it cannot constitute evidence. It is used infrequently in the AAOS CPG process when a patient's life or limb may be at stake and the information is unlikely to be found in clinical studies. Expert consensus opinion also can be applied to low-cost diagnostic or therapeutic interventions.
Third, and most importantly for many of the concerns about the AAOS CPGs, is the problem of lack of evidence. The AAOS guidelines constitute a thorough review of the literature on a topic from the questions generated by the guidelines Work Group. The guidelines simply state the evidence. Expert opinion and retrospective case series cannot be included for the reasons mentioned above. However, lack of evidence for a treatment does not necessarily mean that the treatment does not work. Ultimately, only evidence that a treatment does not work means that it does not work. The inconclusive recommendations cry for high-quality research. The problem of only very poor evidence being found during the development of a guideline was recently taken up following the CPG on Osteochondritis Dissecans of the Knee when the evidence was found to be poor and the Work Group subsequently took the initiative to develop a multi-institutional group to enact a detailed research agenda.
Finally, Dr. Pedowitz addresses the ethics of RCTs with the concern that patients may be made to suffer if not offered surgery that we are convinced is successful. He is correct that RCTs cannot be performed without clinical equipoise and that certain studies are therefore unethical. Patients with hip fractures have a high mortality rate, and there is not much debate about whether to surgically treat these fractures. This will not be subject to an RCT. Many of our orthopaedic disorders do not have such well-defined outcomes and require comparative studies, although perhaps not RCTs. Remember, all comparative studies are included in the AAOS CPG process. Patients want to know that we will make them better, and only better evidence will provide us the assurance that we can do so.