Residency education requires the acquisition and development of numerous
traits, including knowledge, technical and interpersonal skills, compassion,
and the ability to respect and work with other individuals in the health-care
system. Recently, the Accreditation Council for Graduate Medical Education has
fostered the Outcome Project, a process that has required residency programs
to include in their curricula six core competencies that have been identified
as essential components of resident education and to demonstrate their
effectiveness in teaching them.
One of these competencies is medical knowledge, and there are currently
numerous mechanisms to assess a resident's knowledge during residency
training, including observations in a clinical setting, oral examinations,
objective structured clinical examinations, and in-training examinations. The
Orthopaedic In-Training Examination (OITE) is given once a year and measures
factual recall as well as interpretive and problem-solving skills. An
association has been demonstrated between performance on the OITE and on
specialty board certification examinations, especially at the higher and lower
extremes1-4.
Individual residency programs may use the in-training examination to assist in
not only the evaluation of a resident's acquisition of medical knowledge over
time but also in the determination of the department's overall teaching
effectiveness. The United States Medical Licensing Examination (USMLE) Step-1
and Step-2 Clinical Knowledge examinations measure skills similar to those
measured in the in-training examination, and scores on these examinations are
used by many orthopaedic residency training programs, including our own, as a
factor in resident selection.
Many factors may influence how residents perform on the in-training
examination. These include the study habits and abilities of the resident, the
breadth and teaching effectiveness of the faculty, the educational curriculum
of the program, an exposure to an appropriate mix of patients that allows for
surgical and nonsurgical treatment as well as continuity of care, and a
balance among clinical workload, study time, and recreation.
The purpose of this investigation was to analyze the in-training
examination scores of the individuals in our orthopaedic residency training
program. We sought to determine whether there was a relationship between the
scores on the USMLE examination taken during medical school and performance on
the OITE and to evaluate the OITE scores each year, over time. The latter was
done with the intent of determining our effectiveness in improving the
knowledge base of our residents, relative to their peers, as measured by the
yearly examination.
The Penn State Department of Orthopaedics and Rehabilitation has a
four-year residency program after a preliminary year of training and, during
the time-frame in the present study, had three orthopaedic residents per year.
During the preliminary year of general surgery, the residents had a one-month
rotation in the orthopaedic department. The major teaching hospital is the
Penn State Hershey Medical Center, a level-I trauma center. One additional
resident per year worked in the department's Musculoskeletal Research
Laboratory, and, during the eight years reviewed in the present study, seven
residents worked in the laboratory. Six residents worked in the laboratory for
one year, and five of them went into the laboratory after postgraduate year 1
and were designated as year-in-training 0 (YIT 0). Another resident completed
his research year after three years of general surgery and was also listed as
YIT 0. This is consistent with Residency Review Committee guidelines for the
examination, which require that the designated year in training reflect the
individual's current year of clinical orthopaedic training, as opposed to his
or her postgraduate year. The final resident entered the laboratory for two
years, after having completed one year of orthopaedic residency, and was
listed as a YIT 1 for both years that he took the examination while working in
the laboratory. These individuals were required to attend all conferences with
the other residents but had no clinical responsibilities during the day. Their
on-call responsibilities were inconsistent but minimal. In some years they
were never on call, while in other years they would take as many as three
nights of call per month.
During this period, the department had two separate match numbers that
resulted in two residents who would begin their orthopaedic training after a
preliminary year of general surgery, while a third resident, from a separate
match number, would enter the laboratory after that year. Therefore, there
were consistently three residents per year in the YIT-1 year, with the
exception of one year in which a fourth resident completed one year of
orthopaedic training and subsequently left the program. The academic
qualifications of the residents who matched into the laboratory year were
comparable with those in the rest of the program.
During the period under review, there was one department chairman with a
stable faculty and clinical and didactic curriculum. The clinical faculty
comprised fellowship-trained orthopaedic surgeons who were representative of
all subspecialties except shoulder. The department was fully accredited by the
Residency Review Committee during the entire period under consideration.
Our educational curriculum did not provide for specific lectures and/or
instruction for the in-training examination. Residents were expected to
prepare for it either independently or in small groups. The examinations were
administered within our department, and they were taken either in the
orthopaedic library or in the adjacent faculty offices. One of the faculty
members proctored the examination each year, and residents were allowed up to
6.5 hours to complete it, as recommended by the American Academy of
Orthopaedic Surgeons Evaluation Committee, the creator of the examination.
The records of each orthopaedic resident who took the examination from
November 1993 through November 2000 were reviewed, and the SAS statistical
program (version 8.2; SAS Institute, Cary, North Carolina) was used to
calculate correlation coefficients, corrected for reliability, and 95%
confidence intervals to assess the relationship, if any, between the OITE
year-in-training percentiles and the USMLE Step-1 and Step-2 three-digit
scores. Complete correlations could be performed for twenty of thirty-one
residents. For two residents, we had only Step-1 scores, while for three
residents we had only Step-2 scores. These correlations could not be performed
for the remaining six residents who had completed their training earlier in
the study, as they had taken Part 1 and/or Part 2 of the National Board of
Medical Examiners (NBME) certification examinations, which were reported as a
percentile and reflected a candidate's performance relative to a single
reference group and were not equated across time. In addition, in-training
examination scores were evaluated longitudinally from YIT 1 through 4, with
use of a one-way analysis of variance with multiple comparisons. P values of
<0.05 were considered to be significant.
Complete data, defined as both USMLE Step-1 and 2 scores and all OITE
scores, were available for eleven residents in YIT 4, fourteen in YIT 3,
seventeen in YIT 2, and twenty-two in YIT 1. There was evidence of a
significant moderate-sized correlation between USMLE Step 2 and all OITE score
percentiles (p < 0.05). With the numbers available, no correlation was
detected between the USMLE Step-1 scores and the OITE percentiles. The mean
OITE scores were in the 66th percentile for YIT 1, the 53rd percentile for YIT
2, the 57th percentile for YIT 3, and the 50th percentile for YIT 4. For the
purposes of the following discussion, we refer to anyone who completed a year
in the laboratory as a laboratory resident and to those who did not
participate in the research year as clinical residents. Residents in the
laboratory for one year scored in the 88th percentile during that year (as a
YIT 0). When they took the examination in November after completing their
laboratory year (as a YIT 1), they scored in the 86th percentile. The mean
OITE score of the laboratory residents when they took the examination as a YIT
4 was in the 48th percentile (Fig.
1 and Table I).
The mean score for clinical residents in YIT 1 was in the 58th percentile,
and the mean score for the laboratory residents in YIT 1 was in the 86th
percentile; the difference was significant (p < 0.05). The decrease in the
scores of the laboratory residents from YIT 1 to 4 (the 86th to the 48th
percentile) was significant (p < 0.05). The mean scores for the clinical
residents decreased from the 58th to the 51st percentile from YIT 1 to 4, but,
with the numbers available, the difference was not significant (p > 0.05).
The difference in OITE scores for YIT-4 laboratory residents (48th percentile)
and YIT-4 clinical residents (51st percentile) was also not significant.
For many years, orthopaedic surgery has had a highly competitive residency.
After surveying residency directors from various specialties, Wagoner et
al.5 reported that
the more competitive specialties (>85% match rates), such as orthopaedics,
tended to rely on more objective data, such as NBME scores, transcripts, and
membership status in Alpha Omega Alpha, a medical school academic honor
society. In our investigation, a significant moderate-sized correlation was
identified between USMLE Step-2 scores and performance on the OITE. Although
we did not investigate whether there was a correlation between USMLE scores
and the pass rate for the American Board of Orthopaedic Surgery (ABOS) Part-I
certification examination, Case and
Swanson6 reported a
strong relationship between the pass-fail outcome on the orthopaedic
board-certification examination and NBME scores, especially the Part-II
scores. To our knowledge, this work has not been repeated with the USMLE
Step-1 or Step-2 examinations. Because the individuals who were eligible to
take the NBME examination represented only a subset of those who now take the
USMLE examination, and the design of the USMLE differs from the NBME
examination, the continued correlation of performance on USMLE Steps 1 and 2
and the ABOS Part-I certification examination should not be assumed.
The USMLE Step-1 and Step-2 Clinical Knowledge examinations are licensure
examinations, designed to certify that an individual possesses at least the
minimum knowledge necessary to enter the practice of medicine. It is important
to note that our study analyzed the relationship between the three-digit USMLE
scores and the OITE percentiles and that neither NBME nor USMLE percentiles
were used in the calculations. USMLE scores are reported by the NBME as a
three-digit standard score equated across time and examination form so that
identical three-digit scores imply equivalent levels of performance. USMLE
percentiles were not used because individuals with comparable levels of
achievement and three-digit scores would be expected to have differing
percentile scores as the total ability of the comparison cohort changed over
the time-period of this investigation.
The data in the present report raise several questions that warrant
analysis. Our residents who had spent a year in the laboratory had
significantly higher scores during YIT 1 than the full-time clinical
residents, and this may be attributable to several factors. Although the
laboratory year provided them with a very modest clinical exposure, they
participated in the daily morning orthopaedic conferences throughout the year,
had minimal clinical work responsibilities in the evening, and ample time for
evening reading. It would appear that the year in the laboratory allowed them
to develop a fund of knowledge that resulted in scores during YIT 1 that were
higher than those of the clinical residents. These superior scores, however,
clearly did not continue throughout their training, and we believe it is
important to consider why the mean scores of the laboratory residents
decreased as they progressed through the program. One possibility is that the
laboratory residents simply had an additional year of study prior to taking
the examination, and they scored higher during YIT 1 because they had more
time to prepare. It could also be secondary to the didactic educational
curriculum, if it had not been satisfactory for the more senior-level
residents. However, it was identical for the clinical residents in YIT 1
through 4, and their scores did not decrease over time.
Finally, we believe that one must also consider the effect of the
difference in work hours, fatigue, and study time throughout the year for our
laboratory residents as they progressed through their clinical years. Although
the work hours of the residents in our program were not documented during this
time-period, it is well accepted by our faculty and residents that there was a
substantial difference in these factors for the residents working in the
laboratory and those with full clinical responsibilities during the year. The
clinical residents typically started their work responsibilities early in the
morning, predictably worked well into the evening, and had
"on-call" responsibilities every third night. Although this may
not be the case in all orthopaedic residency programs, our residents do not
have time for study during the day. Their potential reading time was limited
to evenings and weekends and, therefore, was dictated by clinical workload
hours. As our laboratory residents did not have substantial clinical
responsibilities during their research year, they were more rested on a daily
basis and routinely had time for evening study. This allowed them to
accumulate a relatively superior fund of knowledge that resulted in higher
OITE scores during their first year in training. As they continued through the
residency program, however, they were subjected to the same clinical
responsibilities and work-load as the clinical residents and were not able to
maintain the higher scores. It is appropriate to consider, therefore, the
effect of work hours on resident performance in general, and, specifically, on
cognitive activities.
Numerous investigators have evaluated the effect of resident work hours on
mood and
attitude7-16,
psychomotor and neuropsychological test
performance12,13,17,18,
and simulated and actual clinical
tasks7,17,19-23.
From those investigations, it can be concluded that sleep deprivation and
fatigue have clear, deleterious effects on mood, and, although it appears that
the performance of short-term tasks that require manual dexterity, reaction
time, and short-term memory are reasonably maintained, tasks that require
sustained vigilance and concentration are the most sensitive to sleep loss and
fatigue. Previous studies by
Wilkinson24,25
demonstrated that moderate sleep deprivation resulted in adverse effects on
serial reaction time and vigilance tasks only when the measurement duration
approached thirty minutes.
Donnell26
demonstrated that performance accuracy in a sleep-deprived population did not
deteriorate until fifty minutes of testing had elapsed. Although those studies
did not attempt to measure academic performance, we believe that the data have
negative implications for our clinical residents, and others, with regard to
their ability to have productive evening study time.
Other investigators have looked specifically at the relationship between
sleep loss and learning and academic performance. Jacques et
al.27 evaluated the
consequences of sleep loss, hours worked, and call schedule on cognitive
performance, measured by the American Board of Family Practice in-training
examination, and demonstrated a decline in test scores with decreasing sleep
on the night before the examination for residents in each year of training. It
is of note that the loss of one night's sleep resulted in a decrease in test
scores that was equivalent in magnitude to the difference in scores between
the first and the third year of training.
The results of the investigations described above have implications for the
potential for productive study time during residency training. The data
suggest that fatigue, such as might occur after a night on call with
inadequate rest, limits the ability to read productively and retain
information for short periods of time. Therefore, a one-in-three on-call
schedule, such as ours, for the most part afforded the resident the
opportunity to study, at best, one night in three.
Many, if not most, orthopaedic residency programs place importance on USMLE
scores in the selection of medical students for their residency program. Like
the OITE, the USMLE measures factual recall as well as interpretive and
problem-solving skills. As we utilize in-training scores as a measure of
medical knowledge, it is reasonable to ask whether there is an association
between the scores on the USMLE Step-1 and Step-2 Clinical Knowledge
examinations and the scores on the OITE. While we did not find a correlation
between USMLE Step-1 scores and OITE percentiles, a positive correlation was
noted between the USMLE Step-2 score and OITE percentile, suggesting that
substantial deviations from this relationship during an individual's residency
require analysis. For example, we believe that when a resident in our program
scores quite high on the USMLE Step 2 but relatively low on the OITE, a review
of individual study habits and the balance between clinical workload and
reading time is warranted. In addition, any factors in the personal life of
the resident that might compromise his or her accumulation of medical
knowledge should be identified. On the other hand, we also believe that an
individual who performs relatively poorly on the USMLE Step 2 is more likely
to do less well on the OITE, and, in this case, we need to be more proactive
in supporting the resident's accumulation of medical knowledge.
Historically, the vast majority of applications that we have received for
our residency program contain USMLE Step-1, but not Step-2, scores. It may be
that our orthopaedic department's selection process has been overly dependent
on these scores in the past. It is clear from these data that, within our own
program and during the period of this study, there was no association between
the Step-1 scores and performance on the OITE. Although we identified a
significant correlation between Step-2 USMLE scores and resident in-training
examination scores, we also believe that it would be inappropriate to
overemphasize USMLE Step-2 scores in resident selection. As noted previously,
the USMLE is a licensing examination and is not designed to predict future
performance on other examinations nor is it designed to predict resident
performance or, ultimately, the quality of the practice of medicine. Test
results can be biased by medical school characteristics and curricula that are
patterned on a USMLE format, resulting in substantial advantages to those
students taking the
examination28,29.
Fine and Hayward30
reported that the performance of internal medicine residents, as measured by
faculty evaluations, was most strongly correlated with honors during the
medical school internal medicine clerkship, as opposed to NBME scores.
Erlandson et al.31
found no correlation between NBME scores and resident performance, and Warrick
and Crumrine32
identified a negative correlation between NBME scores and anesthesiology
in-service examination scores. Therefore, although we identified a correlation
in our own program between USMLE Step-2 and in-training examination scores, we
believe that other aspects of the candidate's application warrant important
consideration as well.
Clearly, there are many factors, in addition to medical knowledge, that
influence a resident's performance and overall competency as a physician. The
purpose of this study was not to emphasize the importance of medical knowledge
relative to any other competency but, rather, to evaluate our own training
program's experience with the USMLE and the OITE. A review of the literature
on sleep deprivation strongly suggested that fatigue secondary to a busy
on-call schedule of one night in three clearly compromises potential
productive study time, and the faculty in our department are fully supportive
of the recently created limitations on resident work hours. We have recently
implemented a "night-float" system for the junior-level residents
and have restructured the on-call schedule for the remaining residents to be
one night in five. As numerous changes that might affect the educational
experience of our residents have occurred in our department during the past
several years, and residency education is clearly evolving rapidly, we believe
that it is essential to monitor the performance of our residents not only in
medical knowledge but in each of the core competencies. We believe that the
data in the present report will serve as an appropriate baseline for future
comparison as we continue to try to enhance the educational experience of our
residents.