In the last decade, there has been an effort to refocus on the efficacy of resident education practices. Much of this effort was summarized by a statement released in 1999 by the Accreditation Council for Graduate Medical Education, which defined medical knowledge as one of six clinical care domains in which residents must receive instruction and show competency. Since then, there have been a number of subsequent reformations to residency training, such as the limitation of resident work hours to no more than eighty hours per week. Within the orthopaedic community, these changes have resulted in a tremendous interest in optimizing resident education. This interest is reflected in the literature by a near doubling in the number of studies concerning orthopaedic resident education and/or training in the past five years from 2004 to 2008 (n = 75) as compared with the preceding five years from 1999 to 2003 (n = 46) (Fig. 1). This interest served as one of the justifications for the present study as we attempted to understand the role of using the recent medical literature for the education and training of orthopaedic residents.
One of the tools that the orthopaedic community has established to evaluate resident training is the Orthopaedic In-Training Examination (OITE)1. The examination focuses on material supported by current scientific literature. The purpose of the present study was to evaluate what literature resources might be most beneficial for orthopaedic residents who are preparing for the OITE. The primary questions for the present study were: (1) What are the recommended references for the questions for each of the subject areas tested on the examination? (2) Do the subject areas tested correlate with the current literature? (3) What are the views of current residents on the usefulness of literature as an educational tool? (4) What educational resources did practicing orthopaedic surgeons use and find helpful during their previously completed residency training?
The American Academy of Orthopaedic Surgeons distributes a score key for each OITE. This key identifies the content domain for each question and provides one or more recommended references for the material tested. Twelve content domains are tested on the examination: (1) foot and ankle, (2) hand, (3) hip and knee reconstruction, (4) medically related issues, (5) musculoskeletal trauma, (6) orthopaedic diseases, (7) orthopaedic science, (8) pediatric orthopaedics, (9) rehabilitation, (10) shoulder and elbow, (11) spine, and (12) sports medicine (Fig. 2). For the present study, the OITE score key was reviewed for the five examinations between 2002 and 2006. The most frequently referenced journals and textbooks were identified. In addition, the most commonly recommended literature resources were stratified according to content domain and year. The percentages that each reference represented of the overall total and of the stratifications by content domain were compared.
In order to assess whether the subject areas that are tested on the OITE correlate with the current literature, separate studies have been or are currently being conducted to evaluate various subject areas and each of the content domains. The present study evaluated the results of two of these reports, including one report that assessed recent hand literature2 and one report that evaluated current literature regarding six specific subject areas, including osteonecrosis, idiopathic scoliosis, chondrosarcoma, lumbar fusion, femoral neck fracture, and hip resurfacing3. The detailed descriptions of these assessments comparing the literature and the OITE have been described previously2,3. These assessments reviewed five years of the OITE (2002 to 2006) and identified all hand-related questions and those associated with the six specific subject areas. The hand-related questions were further stratified according to anatomical location, diagnosis, and treatment. Similarly, the osteonecrosis-related questions were stratified according to whether they were focused on etiology, pathophysiology, diagnosis, or treatment. The osteonecrosis treatment-focused questions were further stratified according to treatment type (nonoperative, core decompression, nonvascularized grafting, vascularized grafting, osteotomy, or arthroplasty surgery). Next, the MEDLINE bibliographic database was utilized to conduct a systematic review of recent literature for the five years (2001 to 2005) preceding each test, and all hand-related articles and articles associated with the six specific subject areas were identified. The hand-related and osteonecrosis-related articles were stratified according to the same categories as the OITE questions. Finally, the proportions of the questions were compared with the percentage of articles for the various categories.
In order to assess the resources that current orthopaedic residents found most useful in their preparation for the most recent OITE, a request was distributed to orthopaedic residency program directors, listed by the Accreditation Council for Graduate Medical Education, asking them to allow their residents to participate in a web-based survey regarding their preparation for the OITE. The survey was blinded, and no identifiers were collected for the participants. The results of the first 150 participants were assessed at the time of the present study. One hundred and twenty-nine residents indicated that they were in an academic program, and twenty-one residents indicated that they were in a community program. With regard to the distribution according to post-graduate year (PGY), nine residents were PGY-1, twenty-six were PGY-2, thirty-eight were PGY-3, forty-two were PGY-4, and thirty-five were PGY-5.
The questionnaire consisted of fifteen questions. Two questions focused on resident use of textbook and journal resources. For these questions, the residents rated the value of the journal and textbook resources that they utilized on a 0 to 5-point scale, with 0 indicating that they did not use the resource and 5 indicating that they considered the resource to be "very helpful." More specifically, the questions asked were (1) How helpful were The Journal of Bone and Joint Surgery (American Volume), the Journal of the American Academy of Orthopaedic Surgeons, Clinical Orthopaedics and Related Research, and The Journal of Bone and Joint Surgery (British Volume) in preparing for the OITE? and (2) How helpful were the American Academy of Orthopaedic Surgeons self-assessment examination questions, First Aid for the Orthopaedic Boards, Campbell's Operative Orthopaedics, Review of Orthopaedics (Miller), AAOS Comprehensive Orthopaedic Review (Lieberman), Orthopaedic Knowledge Update, Orthopaedic Key Review Concepts (Chin), and the Orthopaedic Surgery Essentials series in preparing for the OITE? The resources most highly recommended were identified, and the proportions of residents ranking each resource at each score on the 0 to 5-point scale were compared.
Similar to the web-based survey distributed to the orthopaedic residents, a questionnaire was created to assess the study resources that practicing orthopaedic surgeons had found useful during their previous residency training. This questionnaire was distributed at the 2009 Annual Meeting of the American Academy of Orthopaedic Surgeons (held in Las Vegas, Nevada) as part of the scientific exhibit associated with the present study. The questionnaire was also available online for surgeons to participate. Among the surgeons who participated, eight had been in practice for one to two years; eighteen, for three to five years; seven, for six to ten years; eleven, for eleven to fifteen years; and twelve, for sixteen years or more. The specialties were listed as general (four surgeons, 7%), hand (nine surgeons, 16%), joint reconstruction (thirteen surgeons, 23%), spine (two surgeons, 4%), pediatrics (fourteen surgeons, 25%), sports (nine surgeons, 16%), trauma (one surgeon, 2%), oncology (two surgeons, 4%), and foot and ankle (two surgeons, 4%). The majority of the participants (forty-seven surgeons, 84%) had worked in an academic center during the past year, with the remaining surgeons having worked in private practice (six surgeons, 11%) or as hospital employees (three surgeons, 5%). Almost all of the participants reported that they had trained in an academic setting (fifty surgeons, 89%) as opposed to a community program (six surgeons, 11%).
The questionnaire for the practicing surgeons asked the respondents to rate the usefulness of various journals and textbooks in a similar manner as the resident survey with the surgeons indicating how helpful the resources were in preparing them for their current practice. An additional question assessed whether the surgeons thought the OITE helped them prepare for the American Board of Orthopaedic Surgery Certification Examination. The resources most highly rated were identified and the proportions of surgeons ranking each resource at each score on the 0 to 5-point scale were compared.
The differences in proportions were assessed with use of a chi-square analysis with Yates correction. A p value of <0.05 was considered significant.
Source of Funding
There was no external funding source for the present study.
OITE Recommended References
For the five years of the OITE that were reviewed, the three most frequently recommended journal references were all general orthopaedic journals: The Journal of Bone and Joint Surgery (American Volume) (number of recommended references [n] = 346, 20%), Clinical Orthopaedics and Related Research (n = 177, 10%), and the Journal of the American Academy of Orthopaedic Surgeons (n = 124, 7%) (Fig. 3-A). When the OITE recommended references were stratified according to the year of the examination, the same three general orthopaedic journals were the most frequently referenced resources, except during 2002, when the Journal of Pediatric Orthopaedics was the third-most-frequently referenced journal for that year.
Similar to the overall assessment of OITE questions, general orthopaedic journals were frequently recommended for the questions in each content domain. However, the frequency with which the general orthopaedic journals were referenced varied depending on the specific content domain. For example, >51% of the hip and knee reconstruction journal references on the OITE from 2002 to 2006 were from The Journal of Bone and Joint Surgery (American Volume) and Clinical Orthopaedics and Related Research, compared with only 15% for The American Journal of Sports Medicine (p < 0.001). In addition, the top three recommended references for each of the content domains typically included a specialty journal related to that domain. For example, the Journal of Pediatric Orthopaedics was the most common journal recommended for pediatric orthopaedic-related questions (Table I).
Textbooks represented 40% of the recommended references listed in the OITE answer key between 2002 and 2006. The three most frequently recommended textbook references for the OITE for these years were Orthopaedic Knowledge Update (number of recommended references [n] = 248, 22%), Orthopaedic Basic Science: Biology and Biomechanics of the Musculoskeletal System (n = 91, 8%), and Instructional Course Lectures (n = 73, 6%) (Fig. 3-B). The Orthopaedic Knowledge Update was also the most referenced textbook for each year from 2002 to 2006 on the OITE.
Similar to the journal references, relevant specialty-specific textbooks were frequently given as a recommended reference for their associated content domain. The only textbook that was one of the top three most frequently recommended for all of the content domains was the Orthopaedic Knowledge Update (Table II).
Comparison of Content of OITE and Current Literature
The recent studies that compared the OITE with the current literature showed that there were similar proportions of articles and OITE questions for a number of subject categories (Table III). There were some differences between the OITE and the current literature with regard to the proportions of articles and questions. For example, the percentage of osteonecrosis-related OITE questions (0.58%) was significantly lower than the corresponding proportion of articles that were related to osteonecrosis (2.01%) (p < 0.001). Another example of the difference between the OITE and the reviewed literature was the higher total proportion of articles as opposed to questions related to idiopathic scoliosis (2.57% compared with 0.65%; p < 0.001). However, additional analysis showed that 7% of the studies published in the subspecialty journal were related to idiopathic scoliosis for the five-year period reviewed. When this subspecialty journal was not included in the statistical analysis, there was no longer a significant difference between the content of the OITE and the content of the literature.
Resident OITE Recommended References
The journal that was rated highest by the residents was the Journal of the American Academy of Orthopaedic Surgeons (mean, 1.9 points [2.8 points when excluding the 0 points assigned in the cases in which residents reported that they did not use the journal]). Chi-square analysis showed significant variation in the ratings of the journals (p < 0.001). Additional analysis showed that the Journal of the American Academy of Orthopaedic Surgeons was rated as being "very useful" by a higher percentage of residents (twenty-three residents, 15%) than all other journals (p < 0.001). The Journal of Bone and Joint Surgery (American Volume) was the most frequently used journal (number of students = 85, 57%) and also had the second-highest rating (mean, 1.3 points).
The highest rated textbook or review book was Miller's Review of Orthopaedics (mean, 2.6 points), with 85% of residents reporting that they used this book. When the residents who did not use each book were factored out, the American Academy of Orthopaedic Surgeons self-examination questions had a similar mean score of 3.3 points, compared with 3.1 points for Miller's Review of Orthopaedics. Additionally, the percentage of residents ranking the American Academy of Orthopaedic Surgeons self-examination questions and Miller's Review of Orthopaedics as "very helpful" were similar (28% and 26%, respectively; p = 0.795). The mean score for the Orthopaedic Knowledge Update, the most frequently referenced textbook in the OITE answer key, was 1.4 points, with 55% of the residents reporting that they used the book during their preparation for the examination.
Practicing Surgeon OITE Recommended References
The Journal of Bone and Joint Surgery (American Volume) was the only journal that all practicing surgeons reported using as a resource during the time of their residency. The proportions of surgeons reporting a score of "very helpful" for The Journal of Bone and Joint Surgery (American Volume), the Journal of the American Academy of Orthopaedic Surgeons, and the specialty journal related to their current practice were similar (38%, 38%, and 43%, respectively). Campbell's Operative Orthopaedics was the most frequently used reference book (forty-nine surgeons, 88%). A similar percentage of surgeons ranked Campbell's Operative Orthopaedics (thirty-one surgeons, 55%) and the Orthopaedic Knowledge Update (twenty-five surgeons, 45%) as being either "useful" or "very useful" (p = 0.345). Forty-two surgeons (75%) reported that the OITE helped to prepare them for the American Board of Orthopaedic Surgery Certification Examination.
The present study characterized the content of the OITE over a five-year period in terms of high-level content domains. Three of these domains (pediatrics, trauma, and orthopaedic science) make up >40% of the material tested. Other recent studies have been conducted to further analyze the content and type of questions for various content domains and subject areas. In one of these studies, Frassica et al. reported that approximately 10% of the questions tested resident knowledge of orthopaedic pathology, with the majority of these questions requiring residents to interpret imaging studies and/or histological material4. Additional studies in other subject areas have been or are currently being conducted to further define study content for residents who are preparing for the examination2,3. Compared with the test preparation material available for other standardized tests such as the United States Medical Licensing Examination or even the American Board of Orthopaedic Surgery Certification Examination, there are limited resources for residents who are preparing for the OITE. One of the purposes of the present study was to characterize the content tested on the OITE in an effort to increase the awareness of the residents who are seeking the most effective resources for examination preparation.
The findings of our review of the recommended references for the OITE as well as the results of the resident survey suggest that residents may benefit the most in their preparation for the examination by reading general orthopaedic journals. The three journals that are recommended on the basis of the present study are The Journal of Bone and Joint Surgery (American Volume), Clinical Orthopaedics and Related Research, and the Journal of the American Academy of Orthopaedic Surgeons. Because of the large number of content-specific articles published in specialty journals, articles in specialty journals are less likely to be a recommended reference for the OITE. For example, hand-specific journals and textbooks were frequently recommended as references for the OITE questions. However, an assessment of the recommended references on the OITE indicated that the proportion of hand-related material referenced was higher for general orthopaedic journals such as The Journal of Bone and Joint Surgery (American Volume) and Clinical Orthopaedics and Related Research2.
This recommendation for using general orthopaedic journals in preparing for the OITE is further strengthened by the results of a recent study by Miyamoto et al.5. Those authors reported that there is a direct correlation between performance on the OITE and the use of the general orthopaedic literature. They found that daily orthopaedic reading (r = 0.34, p = 0.03) as well as use of both The Journal of Bone and Joint Surgery (American Volume) (r = 0.6, p < 0.01) and the Journal of the American Academy of Orthopaedic Surgeons (r = 0.36, p = 0.02) had a positive correlation with OITE performance.
There were some discrepancies between the journals and the OITE3. The example of osteonecrosis suggests that procedures for which there is not a consensus regarding an optimum treatment modality frequently are reported in the current literature but are not likely to be tested on the OITE. The results of the present study also suggest that subspecialty journals may focus more on some topics as compared with both the OITE and the general orthopaedic journals. For example, it was shown that the subspecialty journal that was reviewed for idiopathic scoliosis had a greater proportion of articles related to this subject in comparison with both the OITE and the general orthopaedic journals that were reviewed. Residents who are using the current literature as a study tool should account for these potential differences as they read the recent studies and consider that material published in general orthopaedic journals may more accurately reflect the content of the questions tested on the OITE.
Although the present study suggests that general orthopaedic journals should be recommended as a study tool for residents who are preparing for the OITE, this finding conflicts with the results of the survey of practicing orthopaedic surgeons, which suggested that the use of specialty journals during residency is equally important for surgeons in preparation for their future practice in a given orthopaedic specialty. This finding suggests that residents who have decided to pursue a given specialty should utilize both general and specialty orthopaedic journals as they prepare for their future practice.
The present study will allow residents to better customize their preparation for the OITE, and orthopaedic residency directors can use these findings to develop appropriate educational programs and journal clubs for their residents in training. Reading the recent literature as well as textbooks and review books represents only one aspect of resident education and training. Additional studies are being conducted by us to more fully assess the role of the type and number of surgical cases in optimizing resident training. 