The Orthopaedic In-Training Examination (OITE), produced by the American Academy of Orthopaedic Surgeons (AAOS), is administered yearly to orthopaedic residents across the United States. First administered in 1963, the OITE is the oldest existing examination of its kind created for specialty training. This examination serves multiple purposes. As initially set forth by the AAOS, the examination serves to (1) allow residents to assess their level of orthopaedic knowledge on the basis of a national average, (2) set minimal standards for orthopaedic residents, and (3) quantify the quality of education at various orthopaedic programs1. The OITE has served not only as an annual evaluation of orthopaedic resident training in the U.S., but also as a learning tool for residents, as they are subsequently given the preferred answers to questions and pertinent citations after examination scores are tallied.
We analyzed the shoulder and elbow content of the OITE to assist trainees and residency training program directors in preparing for the OITE and, ultimately, the American Board of Orthopaedic Surgery (ABOS) examination. Research has shown OITE scores to be a predictor of success on the ABOS examination, and effective preparation for the OITE may serve trainees well in preparing for the examination at the culmination of their residency training2. Many residency training programs have formal OITE preparatory curricula, and the analysis and classification undertaken in this study can guide the creation and organization of a syllabus for this purpose. Preparing for the OITE can be an opportunity to focus on the highly tested topics in orthopaedics. Identification of commonly tested concepts may also help improve the efficiency of resident preparation by directing a methodical approach to a large volume of information. It is important to remember that the OITE has never been validated to measure the overall quality of an orthopaedic resident or how he or she might perform in clinical practice.
Our goal was to analyze the shoulder and elbow subcategory of questions on the OITE to help guide resident preparation. We hope analysis of the shoulder and elbow portion of the OITE will also serve to maintain the clinical relevance of these questions on future examinations.
We reviewed the OITE questions for the examinations given in the years 2002 through 2007. We initially quantified the shoulder and elbow content and determined the percentage of shoulder and elbow questions on each examination. Each question was then classified into one of several topic categories: trauma, anatomy/biomechanics, degenerative joint disease/stiffness/arthroplasty, instability/athletic injury, rotator cuff, and miscellaneous. We then calculated the percentage of questions represented by each subcategory of shoulder and elbow questions. Additionally, we categorized the questions on the basis of their taxonomy: level 1, recognition and recall of isolated orthopaedic information; level 2, interpretation of information or imaging to identify an answer; and level 3, problem-solving or application of knowledge to a specific scenario to identify a solution3,4. We further classified questions by any associated imaging, such as radiographs, computed tomography, magnetic resonance imaging, and arthroscopic or clinical images. Recall questions (taxonomy level 1) required the examinees to draw from their fundamental knowledge of the shoulder and elbow. Interpretation questions (taxonomy level 2) most often consisted of a short history paired with radiographic, computed tomography, magnetic resonance, arthroscopic, or clinical images, and required the examinee to determine the diagnosis. The third and highest level of questions—the problem-solving questions (taxonomy level 3)—presented the examinee with a series of images and asked for the best treatment plan on the basis of the compilation of patient history and pertinent images. We also analyzed the citations for each of the shoulder and elbow questions and categorized them on the basis of type (journal or book) and a specifically named source. We then calculated the most commonly cited sources.
Statistical Analysis
The data were descriptively analyzed.
Source of Funding
There was no funding for this study.
Earlier analyses of OITE questions have focused on pathology content3, foot and ankle content6, hand content7, level of evidence of clinical treatment questions8, and sports content9. To our knowledge, our study is the first to analyze the shoulder and elbow content of the OITE.
Although the shoulder and elbow content of the OITE composes a small portion of the entire examination (an average of fifteen, or 5.5%, of the 275 OITE questions each year), the shoulder and elbow questions that do appear focus on degenerative disease and instability and heavily test the ability of the examinee to interpret imaging. Our analysis of the shoulder and elbow content may allow focused institutional instruction and resident preparation for future OITEs. In addition, evaluation of previous OITE questions may help OITE examiners assess trends of the commonly tested concepts and determine whether this content should be changed on the basis of what they consider to be the most appropriate material.
From 2002 to 2007, shoulder and elbow content ranked as the third smallest domain by percentage of the overall test (5.5%), lower than hip and knee reconstruction (8.9%), sports medicine (7.5%), hand (6.9%), spine (6.8%), and foot and ankle (5.8%)10. Shoulder and elbow as a subspecialty, however, has become increasingly popular over the past twenty years as the number of fellowships continues to rise and the number of surgical procedures involving the shoulder continues to increase at a greater rate than that of other orthopaedic procedures11,12. Of the top twenty-five procedures reported by Board-eligible candidates for Part II of the ABOS Practice Certification Examination, four relate to shoulder surgery11. By comparison, only two procedures relate to foot and ankle surgery11. It may be appropriate to increase the relative percentage of shoulder and elbow content on the OITE on the basis of the increased understanding of shoulder pathology in recent years and the increased numbers of shoulder surgical procedures being performed. This determination is ultimately charged to the Evaluation Committee of the AAOS. Such decisions are inherently challenging because quantifying the relative knowledge volume and the relevant clinical practice frequency of the various disciplines and their disorders is difficult.
OITE questions are created and classified with use of taxonomies described by Buckwalter et al.13. Level-3 taxonomy incorporates basic knowledge recall with interpretation and problem-solving, ultimately requiring higher cognitive function involving integration of multiple facets of patient presentation4. We found that only 4.4% of shoulder and elbow questions were taxonomy level 3 as compared with foot and ankle (41.9%)6, sports (25.5%)9, and pathology (32.3%)3 questions. It is possible this variation may simply be the result of the different interpretations of the taxonomy classification scheme by the different study authors. Nevertheless, enhancing the focus on problem-solving skills within the shoulder and elbow domain of the OITE is a worthwhile goal. Although the body of shoulder and elbow knowledge will continue to change and evolve with research and innovation, problem-solving skills will always remain a crucial skill for a clinical practitioner.
Our data also suggest an emphasis on degenerative disease and instability with a relative lack of focus on rotator cuff pathology. Data from a review of statistics from Part II of the ABOS Practice Certification Examination reveal that repair of the rotator cuff moved from the 37th to the 14th most commonly performed procedure from 1999 to 2003 by Board-eligible candidates11. Shoulder arthroplasty and instability procedures did not rank in the top twenty-five procedures reported during this same time period11. Our subclassification data again provide an opportunity for OITE examiners to evaluate these trends and determine if changes should be made regarding the relative weight of tested concepts. With only a few questions available with which to determine an examinee's overall shoulder and elbow knowledge, several considerations beyond what a practitioner commonly treats must be balanced, such as whether it is appropriate to test disorders that are rare in clinical practice (as in the evaluation and management of posterior instability of the shoulder) to emphasize the importance of recognizing these conditions so that the patient may receive quality care.
The significant focus on image interpretation within the shoulder and elbow content is appropriate because orthopaedic surgeons are routinely required to evaluate various imaging modalities in the clinical setting. However, considering the increased frequency of shoulder arthroscopy11, an increased focus on the testing of arthroscopic anatomy seems prudent. Only three arthroscopic images pertaining to the shoulder and elbow content domain were found in our entire study period. A greater emphasis on anatomy, biomechanics, and general principles may also fit with the examination's goal of testing "core knowledge" as fundamental knowledge that is unlikely to change with time. Although only 15.6% of questions were classified as anatomy or biomechanics, many of the other questions undoubtedly required an understanding of anatomy for image interpretation.
It is well known that successful OITE performance has been correlated to frequent review of current orthopaedic journals, specifically The Journal of Bone and Joint Surgery (American volume) and Clinical Orthopaedics and Related Research5. We found that an average of 2.02 citations (182 citations per ninety questions) were used for each shoulder and elbow question during the study period and that the most frequently referenced sources were the Journal of Shoulder and Elbow Surgery and The Journal of Bone and Joint Surgery (American volume). In addition, Srinivasan et al.9 recently analyzed the sports content of the OITE and found that, of the primary literature citations, 60% were sources that had been published within five years of the test administration date. We found that no citation referenced journal articles published within a year before the OITE, only 38.7% of the sources cited for shoulder and elbow questions had been published between one and five years before the OITE, 40.5% had been published between six and ten years before the examination administration, and 20.8% had been published more than ten years before the given OITE. This finding again represents an opportunity for future examination preparation.
Resident performance on the OITE is also correlated with subsequent performance on the ABOS examination. Specifically, mean percentile performance on the OITE for the second through the fourth year of training and OITE performance in the last year of training are factors that have been shown to predict success and failure on the ABOS I and II2. Analysis of ABOS content is impossible because of the confidential nature of the certification examination. The OITE is considered to be of such great importance by many residency program directors that a portion of the resident curriculum in orthopaedic residency training programs in the U.S. is often directed toward OITE preparation. With the restrictions in work hours, time for formalized resident teaching and test preparation has become increasingly limited. We believe that our study can help residency training program directors prepare an efficient curriculum for the testing of shoulder and elbow topics. Limited lecture hours can be focused on the more commonly tested topics, leaving less commonly tested material for independent study. In addition, residents would be well served to improve their image interpretation skills, particularly of shoulder magnetic resonance images and radiographs. When reviewing primary literature, a focus on literature published from one to ten years before the examination in the Journal of Shoulder and Elbow Surgery and The Journal of Bone and Joint Surgery (American volume) represents a high-yield strategy. We strongly caution against the use of our findings to define the curriculum for the education of orthopaedic surgery residents. The OITE is limited in its ability to assess overall knowledge, and the topics tested correspond only to a sampling of a comprehensive knowledge base.
Finally, the trends presented in our study can be evaluated by the OITE examiners, affording them an opportunity when constructing the examination to determine if the goals of the questions and testing in general are being met. The AAOS Committee on Evaluations could then consider the relative importance of different orthopaedic concepts and disorders encountered in clinical practice as compared with the topics that appear on the OITE examination.