Passing the written and oral examinations is a requirement for
certification for the American Board of Orthopaedic Surgery (ABOS). Residents
and residency program directors alike consider passing "The
Boards" to be a priority. Part I of the ABOS examination consists of
over 300 multiple-choice questions designed to test the candidate's knowledge
in general orthopaedics, basic science, and the application of this knowledge.
Part II is an oral examination administered to evaluate the candidate's
competence in areas such as data gathering and interpretation, diagnosis,
treatment, and technical skills. Passing the ABOS Part-I examination the first
time is crucial to avoid delays in taking Part II and attaining board
certification. In 2002, 553 (89%) of 623 first-time test-takers passed the
ABOS Part-I examination. If one were to include repeat examinees, 637 (79%) of
805 passed1. The
passing rate after one or more failures is dramatically lower than that for
first-time examinees. In 2002, there were 182 repeat test-takers, of whom
ninety-eight failed (a 54% failure rate), demonstrating the importance of
passing the first
time1. This suggests
that inadequate training and preparation of an orthopaedic knowledge base for
this examination during residency may be difficult to correct after an initial
failure of the ABOS Part-I examination.
The ABOS Part-I examination represents one standardized test in a long
battery of examinations already taken by graduating residents, which include
the Standardized Admission Test; Medical College Admission Test; United States
Medical Licensing Examination (USMLE) Steps I, II, and III; and Orthopaedic
In-Training Examination (OITE) administered during postgraduate-years (PGY)-1,
2, 3, 4, and 5. Since orthopaedic residency applications are extremely
competitive (in 2000, there were 1116 candidates for 554 positions), it can be
inferred that orthopaedic residents have generally performed well on these
previous
examinations2.
Indeed, it may be the unofficial policy of some residency programs to
interview only applicants with USMLE Step-I scores above an established cutoff
unless the applicant is already known to the department faculty. Nevertheless,
every year more than 10% of postresidency orthopaedic surgeons fail the ABOS
Part-I examination and fail to attain "Board-Eligible" status,
which is defined as candidates who have successfully passed Part I and are
waiting to take Part
II1.
In addition, performance on the ABOS Part-I examination is important for
the residency program itself. For example, one of the few quantitative
guidelines that the Orthopaedic Residency Review Committee (RRC) uses in its
review of accredited residency programs is whether 75% or more of the
graduates of a residency program pass the Part-I Examination on their first
try3. Furthermore,
performance of past residents on this test is regarded by residency directors
and applicants as an important indicator of the quality of a residency's
educational program. A program's consistently poor performance may be spread
by word of mouth by potential applicants. For these reasons, programs with
high failure rates may have greater difficulty attracting their top
candidates.
Passage of the ABOS examination is important not only to residency programs
but also to individual surgeons. While the board-certification process is
strictly voluntary for the orthopaedic surgeon, some hospitals require it for
hospital privileges and many practices require it for partnership. In
addition, patients may check web sites such as
or
to see whether a physician is "Board Certified."
Although the annual OITE and the ABOS Part-I examinations both test
orthopaedic knowledge, differences in psychometrics between the two tests may
limit the ability of the score on the OITE to predict performance on the ABOS
examination. Specifically, questions for the OITE are submitted by a group of
orthopaedic surgeons from different subspecialties and are used in proper
balance by the examination committee to comprise the examination. In
comparison, the ABOS Part-I examination is composed of questions that were
previously administered to board-certified orthopaedic surgeons, analyzed
statistically, and selected for inclusion on the basis of their
psychometrics.
Given the importance of passing the ABOS Part-I examination, we sought to
review ten years of orthopaedic resident standardized test data at our
institution in order to search for a correlation between scores on previous
standardized tests and performance on the ABOS written examination. We
specifically chose the USMLE Step-I examination as a score for correlation
with passage of the ABOS Part-I examination, since it is the most recent
competitive test taken by the orthopaedic applicant. We tested the hypothesis
that performing above a specific threshold score on this examination would
correlate with a high likelihood of passing the ABOS Part-I examination. In
addition, we also reviewed the OITE scores of the same cohort during the
third, fourth, and fifth years in training to see whether scoring below a
certain level on this test might represent a warning sign of possible failure
on the ABOS Part-I examination.
The scores of all graduates of our residency program over a ten-year period
for whom USMLE Step-I, OITE, and first-attempt ABOS Part-I scores were
available were included in the data analysis. During the study period, the
National Board of Medical Examiners (NBME) test was changed and renamed the
USMLE examination. Therefore, it was necessary to convert the NBME scores into
USMLE scores for statistical evaluation. According to the National Board of
Medical Examiners, NBME scores (mean [and standard deviation] 500 ±
100) can be transformed into USMLE scores (mean, 200 ± 20) by a simple
linear equation: USMLE = 200 +
([NBME-500]/5)4. For
example, a score of 600 on the NBME is equivalent to 220 on the USMLE. Also,
like most standardized tests, the scaling of the NBME Part-I scores changed
from year to year on the basis of the population's performance. In addition,
shorter tests were administered starting in 1999, causing standard deviations
to increase. Accepting these limitations, the NBME score conversion was used
for all statistical analysis.
Correlation coefficients between the ABOS Part-I percentiles and the USMLE
Step-I and OITE scores for PGY-3, 4, and 5 were calculated with use of MedCalc
(version 7.2.0.2; MedCalc Software, Mariakerke, Belgium). The strength of any
significant correlation was graded as low (0.1 < r < 0.39), medium (0.40
< r < 0.7), or high (0.70 < r <
1.0)5. For this
analysis, CART (categorical and regression tree) datamining software (version
5.0; Salford Systems, San Diego, California), which can isolate patterns and
relationships among data, was used to evaluate the raw data. The CART method
categorizes and tabulates outcome (pass or fail) for every possible cutoff of
a numerical variable, and it chooses from among the cutoffs the value that
minimizes false classifications. Relative risks and 95% confidence intervals
were calculated with use of SAS software (version 8.2; SAS Institute, Cary,
North Carolina).
During the study period, sixty-five residents graduated from the program
and sixty-four of them who met the inclusion criteria provided the study
population. Throughout this period, the OITE was consistently proctored by a
faculty member within our department. There was a 6.5-hour time limit imposed
by the American Academy of Orthopaedic Surgeons (AAOS). No disciplinary
actions were taken against residents receiving low scores, although the
results for all of the residents were discussed with the Residency Program
Director, and residents with low scores were encouraged to study and improve
their scores. Among this group of sixty-four graduates, fifty-eight residents
(91%) passed and six residents (9%) failed the ABOS Part-I examination. The
mean USMLE Step-I score (and standard deviation) was 211 ± 17.3 (range,
172 to 243). For the Orthopaedic In-Training Examination, the mean PGY-3
percentile was 45.4 ± 26.4 (range, 6 to 98), the mean PGY-4 percentile
was 44.5 ± 27.2 (range, 4 to 98), and the mean PGY-5 percentile was
43.2 ± 27.1 (range, 3 to 99).
Correlation coefficients (r) were calculated
(Table I) in order to determine
relationships between the ABOS Part-I percentile scores and the USMLE and OITE
percentile scores. The correlation coefficient for the USMLE Step-I score and
the ABOS Part-I percentile score was 0.38 (p = 0.002). Correlation
coefficients were also calculated for the OITE percentile scores for years 3
(r = 0.52, p < 0.0001), 4 (r = 0.49, p < 0.0001), and 5 (r = 0.69, p
< 0.0001).
We used the CART software to identify any trends or relationships between
these data. We identified a USMLE Step-I score of 204 as a critical number.
Below this score, there was a 16% chance of failure (three of nineteen) on the
ABOS Part-I examination. With a score of 204 or higher, there was a 7% chance
of failure (three of forty-five) and therefore a 93% chance of passing
(forty-two of forty-five). Consequently, the relative risk of failing the ABOS
Part-I examination for students with USMLE Step-I scores below 204 was 2.63
(95% confidence interval, 0.48 to 14.38) times that of students scoring above
204.
When analyzing the OITE data with use of the CART software, several
relationships became apparent. There was a high risk of failure (63%; five of
eight failed) on the ABOS Part-I examination when a resident scored below the
twenty-ninth percentile for PGY-3 and below the twentieth percentile for
PGY-5. These so-called low-scoring residents had a relative risk of failure
that was 91.7 (95% confidence interval, 8.0 to 1053) times that of the other
residents. All failures occurred among those who scored below the
thirty-second percentile for PGY-3 and below the twenty-seventh percentile for
PGY-4 (Table II). No failures
occurred (fifty passed) when either the PGY-3 score was above the
thirty-second percentile or the PGY-4 score was above the twenty-seventh
percentile (Table III).
We hypothesized that individuals who score better on standardized tests in
the past should continue to do so throughout their training. However, the
findings for the USMLE Step-I scores only weakly support our hypothesis. The
correlation coefficient for the USMLE Step-I and ABOS Part-I scores was 0.38,
a low-to-moderate correlation, although significant. However, poor performance
on the OITE was a much stronger predictor of failure on the ABOS Part I. This
would be expected since the examination is more recently taken (within three
years of the ABOS compared with at least seven years for the USMLE Step I) and
because both the OITE and ABOS examinations test orthopaedic knowledge.
Our data suggest that the USMLE Step-I score is not a strong predictor of
individuals at increased risk for failing the Boards. Stated another way, the
difference between a 16% potential ABOS failure rate (a USMLE score of
<204) and a 7% potential ABOS failure rate (a USMLE score of >204) may
not be large enough to deny a residency spot to an otherwise well-qualified
applicant. There may have been extenuating circumstances (passing of a loved
one, divorce, etc.) that precluded ideal performance on the USMLE Step-I
examination. Alternatively, these individuals may have been less stimulated to
learn basic medical science data, leading to a poorer performance on the
earlier examination, whereas an interest in orthopaedics may motivate them to
study more intensively in their field of interest. We chose not to include
USMLE Steps II and III, since many of the candidates who take these
examinations are already accepted into residency programs and there is less
pressure to do well on these examinations.
The stronger message from our data is the importance of tracking OITE
performance. In our cohort, all failures occurred among residents scoring
below the thirty-second percentile during PGY-3 and below the twenty-seventh
percentile during PGY-4. Obviously, each residency program is likely to have
its own unique thresholds for poor OITE performance predictive of failure on
the ABOS Part I. Program directors, and especially residents, should be aware
that individuals are at risk for failure on the ABOS Part-I examination with
performance below these levels, and they should devote additional time and
energy to preparation for the PGY-5 OITE and ultimately the ABOS
examination.
In summary, this analysis demonstrates that achievement on the USMLE Step-I
examination does not guarantee passage of the ABOS Part-I examination. In our
program, the USMLE scores of residents who failed the ABOS Part-I examination
were as high as 231. However, the present study supports close tracking of
OITE performance by residency program directors. Residents who scored in the
lowest third on the OITE in PGY-3 and 4 stood a much greater chance of failing
the ABOS Part I, and we believe that residents who consistently score below
the thirtieth percentile on the OITE should receive focused attention by the
residency program faculty. It must be stated, however, that it is ultimately
the responsibility of each resident to learn the fundamentals of orthopaedic
surgery and to adequately prepare for these examinations.