The effectiveness of didactic courses with lecture presentations in a PowerPoint format (Microsoft Office; Microsoft, Redmond, Washington) in increasing medical knowledge and improving health-care outcomes has been questioned1-3. Lecture-based courses are common in many surgical disciplines including orthopaedic surgery, but the degree to which course participants acquire and retain new medical knowledge is hard to measure and remains largely unknown.
Residents in training must acquire vast amounts of medical knowledge to practice competently, and the Accreditation Council for Graduate Medical Education has defined medical knowledge as one of the six core competencies4. With the advent of work-hour guidelines, residency programs must optimize the use of formal educational hours. However, few studies have assessed or compared different methods through which residents increase their medical knowledge.
Medical knowledge in orthopaedic surgery is often evaluated by the residents’ overall scores and their scores in various content domains on the annual Orthopaedic In-Training Examination (OITE). The OITE provides an objective method to compare medical knowledge between residents and can be used to assess the effect of various educational interventions to increase resident knowledge.
The Orthopaedic Trauma Association (OTA) offers a three-day predominantly didactic trauma course for residents, which consists primarily of lectures given by orthopaedic trauma experts combined with some smaller-group surgical skills laboratories and/or demonstrations. This course takes place, on the average, four weeks prior to the yearly November administration of the OITE. The purpose of the present study was to determine whether residents who participated in this course improved their scores on the trauma domain of the OITE taken shortly after the course. A secondary goal was to compare the effectiveness of this course and that of a focused clinical rotation in orthopaedic trauma in improving these scores.
After institutional review board approval was obtained, a retrospective analysis of the trauma domain OITE scores of forty-seven orthopaedic surgery residents in postgraduate year (PGY) 2 at a large academic teaching hospital was performed. The data were collected from the academic years 2002 through 2009. Two primary groups were established: twenty-nine PGY-2 residents who took the OTA trauma course approximately four weeks before the November administration of the OITE and eighteen residents who did not take the course before the OITE. The second group of residents took a trauma course later in the year after the November administration of the OITE. The two groups were assigned by the Program Coordinator on the basis of the resident clinical responsibilities at the time of the two courses.
The OTA resident trauma course is held in conjunction with the OTA annual meeting in October. The course is taught by approximately seventy faculty who are OTA members attending the annual meeting. This large faculty, who are experts in their content area, has contributed to the OTA course's rise to prominence as a premier trauma educational venue for junior residents. The course accommodates approximately 125 residents annually. A review of the course syllabus from 2003 demonstrated that approximately two-thirds of the three-day course consisted of large group lectures and one-third involved smaller group sessions on surgical skills, usually with Sawbones models (Pacific Research Laboratories, Vashon, Washington). It was designed to cover fracture problems throughout the spine and extremities. The 2009 OTA resident trauma course syllabus revealed a similar distribution of educational formats with the addition of a few case-based discussion sessions and a slight reduction in the number of lectures.
A secondary analysis of the same forty-seven residents examined the effects of having an orthopaedic trauma rotation before the OITE (fifteen residents) compared with having an orthopaedic trauma rotation after the examination (thirty residents). The scores of two residents, who had their trauma rotation during the annual OITE, were excluded from this portion of the analysis.
The orthopaedic trauma rotation in PGY-2 is eight weeks and focuses nearly exclusively on musculoskeletal trauma. The clinical work includes covering trauma consultations in the emergency room, assisting on orthopaedic trauma cases, participating in trauma clinics with patients seen for the first time and for follow-up evaluation, and a weekly small-group case-based educational trauma conference. The trauma rotation takes place at a level-I trauma center that treats trauma patients with a wide range of injury severity including isolated limb trauma and patients with polytrauma. PGY-2 residents who have not been on this rotation are exposed to trauma only through night call and departmentwide conferences, which focus on orthopaedic trauma with three forty-five-minute lectures per month. In PGY 1, the clinical rotations and conference exposures are the same for all residents. The combined effect of the OTA course and a trauma rotation was assessed by an analysis of residents who had taken the course and had a trauma rotation before the OITE and those who had taken the course and had the trauma rotation after the OITE.
The OITE, which is developed by the Evaluation Committee of the American Academy of Orthopaedic Surgeons, is a 275-item multiple choice examination that covers the spectrum of orthopaedic surgery. The trauma domain is the largest domain of the examination and has an average of fifty-five questions. These questions are submitted by orthopaedic traumatologists and are chosen for the test after editing and peer review by three or four other members of the committee who are also orthopaedic traumatologists.
The percentage correct score on the trauma domain of the OITE for these forty-seven residents in PGY 2 was the main outcome measure. Baseline orthopaedic knowledge between groups was compared with use of the overall OITE percentage scores. Groups within these forty-seven residents were established on the basis of when they took the OTA course or had a trauma rotation in relation to when they took the OITE. Differences in the mean percentage scores were assessed with a t test, and the level of significance was set at p < 0.05.
This study compared two common methods through which orthopaedic residents acquire medical knowledge: a lecture-based course and a focused clinical rotation. Although the number of residents available for review was relatively small, these results suggest that a didactic trauma course alone did not lead to improved resident scores on the trauma domain of the OITE taken four weeks after the course. These scores improved significantly only when residents had a trauma rotation prior to the examination and, as might be expected, to a greater extent when both a rotation and a course were completed before the examination. These improvements were relatively small and could be accounted for by an average of 4.8 additional questions answered correctly out of a total of fifty-five questions.
Although we were surprised that a focused and intense three-day course in orthopaedic trauma did not significantly improve scores in the trauma domain of the OITE taken a few weeks later, there are several possible reasons for this finding. First, the OTA course is not designed as a review course for the trauma section of the OITE. Questions are not reviewed as part of the course format, and most lecturers have little knowledge of what material is to be covered on the OITE. Second, the study habits of the residents in the two groups are unknown. Residents who did not take the course might have been motivated to spend more time in focused review of orthopaedic trauma including question review. Residents who took the course scored better in other areas of the examination, perhaps reflecting more focused study in these areas.
Didactic courses have previously been shown to be poor methods to acquire and retain medical knowledge at least as assessed on standardized tests. For instance, the efficacy of several commercially available review courses was analyzed to determine their ability to improve scores on the United States Medical Licensing Examination Step 15. The scores of the students who took a comprehensive review course prior to the examination were compared with those who had not. The groups were similar on the basis of their Comprehensive Basic Science Examination scores. No difference was noted between those who completed a review course and those who elected self-study. The positive effect of specific preparation courses for the Scholastic Aptitude Test and Medical College Admission Test has also been found to be extremely small or nonexistent6. If there is a benefit to such courses, it may not relate to gaining new knowledge but rather to test coaching, including more efficient identification of weaknesses, test-taking practice, maximizing motivation, controlling test anxiety, and instruction in test-taking skills7. Considering these data, it is less surprising that the OTA course, which was not designed as a specific preparation for the OITE, did not significantly improve the trauma domain scores.
The result of this study raises the question of whether there are better methods for courses and residency curricula to help residents in training to obtain and retain medical knowledge rather than relying predominantly on lectures. Alternate approaches include hands-on skills sessions, problem-based learning, case conferences, audience participation, small group teaching, needs assessment, learner-driven curricula, and testing before and after a course to reflect what has been learned. Current trends in medical educational curricula have emphasized an interactive problem-based approach over lecture-based courses. In medical education, as well as in continuing medical education, it has been recognized that maximum knowledge is acquired and retained through interactive sessions that enhance participant activity and offer the opportunity to practice technical skills. In one study, students taught basic science in a problem-based learning curriculum and those who had a lecture-based learning curriculum were compared with regard to their academic performance on a standardized oral comprehensive examination. The science and medical knowledge component score was significantly better for the problem-based learning group8. In another study, Davis et al.1 suggested that adult education should be learner-centered and an active rather than a passive activity that is relevant to the learner's needs. They concluded that, when performance change is the immediate goal, the exclusively didactic modality has little or no role to play.
The degree to which changes in educational styles, particularly with respect to formal courses, would improve performance on standardized tests such as the OITE remains unknown. However, they are more likely to engage residents in active learning processes and may offer a better chance for them to acquire medical knowledge from primary references or from experienced clinicians. The end result of these alternative, non-lecture-based teaching styles may be more retained knowledge not only to improve test scores but for better decision making and improved patient care.
The OTA resident trauma course is likely to improve the knowledge base, decision making, and skills of orthopaedic residents in ways not captured by performance on the OITE. Operative skills and patient care are examples of competencies critical to a resident learning the principles of orthopaedic trauma care that are not assessed by the OITE. In addition, an educational venue such as the OTA resident course may improve the ability of residents to acquire knowledge to improve OITE or board certification scores in future years.
A clinical rotation actively engages the resident in the subject matter, and a rotation of several months’ duration might be expected to be a very effective way for residents to increase medical knowledge. Residents on a trauma rotation have the opportunity to observe clinical cases, engage in discussion with faculty and more senior residents, and read about the clinical cases. There is, however, little information available in the literature to demonstrate how effective a clinical rotation is in increasing the scores of residents on standardized tests. In this study, we found that residents who had a clinical rotation in orthopaedic trauma had significantly improved scores in the trauma domain of the OITE, albeit to a relatively small extent, compared with residents who had not done this clinical rotation. This small improvement was also less dramatic than we had expected. In a time when graduate medical educational programs are under increased pressure through work-hour restrictions, and with the explosion of medical knowledge that residents must acquire, the traditional model of a clinical rotation needs to be continually modified. Valuable resident time must be spent in learning activities of maximum yield, and utilizing resident time to support clinical service must be minimized. Clinical rotations and other educational programs during residency training must identify optimal methods to provide residents important medical knowledge and maximize the exposure to these learning activities.
Orthopaedic residents improve their knowledge as they progress through residency as measured on the OITE. Our residents improve their raw score on the trauma domain of the OITE by an average of approximately 3.2 correctly answered questions per year of training. In addition, although the numbers are small, our data suggest that the combination of a rotation and a didactic course is a reasonably effective method to improve the trauma domain scores on the OITE. The experience of a trauma rotation may better prepare residents to absorb and retain the volume of information from the didactic course, which may be too much to absorb for the less prepared residents who have not had such a rotation. It seems reasonable to conclude that orthopaedic knowledge is acquired gradually during the course of the residency and that the degree to which performance on a standardized test improves over short durations of time through a focused course is small to the point at which it is hard to measure on a standardized test.
One of the strengths of this study was the use of the OITE as an outcome measure. This is a well-recognized and widely accepted test of medical knowledge for orthopaedic residents in training. However, given the mostly negative results of this study, it is reasonable to ask whether the trauma domain of the OITE is a good measure of orthopaedic resident trauma knowledge. In the absence of another so-called gold standard for comparison, it is hard to answer this question. Multiple choice examinations, such as the OITE, are utilized and accepted in most medical specialties to assess knowledge prior to graduate medical training, during residency training, and for certification examinations after training. Buckwalter et al. compared the results of the OITE with faculty assessment of resident performance and found a close but not perfect correlation between the two9. The OITE has been compared with the results of the Part-I certifying examination of the American Board of Orthopaedic Surgery and was found to have a close but not perfect correlation9,10.
Similar to other multiple choice examinations, the score ranges on the OITE for a given year in training are relatively narrow. However, the reliability (KR-20) of the 275-question overall test is typically high (=0.90), but when the test is used in parts, such as the trauma domain in this study, the reliability of a portion of the test is considerably less because of the smaller number of questions. In recent years, the reliability of the fifty-five-question trauma domain has been approximately 0.70, indicating that substantial test variability is from chance alone. This means that the outcome measure used to assess medical knowledge in trauma is not a perfectly reliable instrument, and this may be part of the reason that we did not find more substantial differences between the various resident groups.
In our preliminary review of the educational benefits of a focused trauma course, with the OITE trauma domain used as an outcome measure, we found that, on the average, the scores of the PGY-2 orthopaedic residents did not increase after taking a focused trauma course. However, a trauma rotation before the OITE, and in particular the combination of both a course and a trauma rotation, did lead to an improvement in trauma test scores. These data suggest that the knowledge required to improve performance on the OITE is acquired gradually during training and is not measurably improved over short time periods, even with focused intense educational exposures. This information highlights the importance of residents committing to the process of continual study as part of their life-long learning goals. It is possible that current trends in educational courses to further engage the learner in interactive activities, decreasing the amount of didactic lectures, will better improve resident medical knowledge, but this remains to be proven. Further evaluation and study is necessary to identify the best methods to increase the medical knowledge of orthopaedic residents.