The ABOS is one of three sponsoring organizations represented on the
Residency Review Committee for Orthopaedic Surgery; the other two
organizations are the Council on Medical Education of the American Medical
Association and the American Academy of Orthopaedic Surgeons. An orthopaedic
resident also serves on the committee. The Residency Review Committee
functions autonomously under the direction of the Accreditation Council for
Graduate Medical Education (ACGME).
According to the ABOS, the goal of orthopaedic residency education is to
prepare a resident to be a competent and ethical practitioner of orthopaedic
surgery1. During
their orthopaedic residency, applicants for certification by the ABOS must
have received, and successfully completed, the following
preparation1:
A. Education in the entire field of orthopaedic surgery, including
inpatient and outpatient diagnosis and care as well as operative and
nonoperative management and rehabilitation.
B. The opportunity to develop, through experience, the necessary cognitive,
technical, interpersonal, teaching, and research skills.
C. The opportunity to create new knowledge and to become skilled in the
critical evaluation of information.
D. Education in the recognition and management of basic medical and
surgical problems.
E. An evaluation of ethical performance.
The ABOS was founded on January 7, 1934, the year after the establishment
of the Advisory Board for Medical Specialties, the American Academy of
Orthopaedic Surgeons, and the certifying boards for ophthalmology,
dermatology, and obstetrics and gynecology. The ABOS, founded to "serve
the best interests of the public and of the medical
profession,"1
was created by the American Orthopaedic Association, the American Academy of
Orthopaedic Surgeons, and the American Medical Association (Section on
Orthopaedic Surgery).
In 1934, the ABOS initiated certification without an examination, which
required ten years of orthopaedic surgery practice. In 1935, the ABOS
introduced the first written examination for certification. In 1937, the Board
became a member of the Advisory Board for Medical Specialties, now called the
American Board of Medical Specialties.
The ABOS currently is composed of eighteen directors elected from
diplomates of the Board and one public member. Directors are nominated by the
American Orthopaedic Association, the American Medical Association, and the
American Academy of Orthopaedic Surgeons, and they serve a ten-year term
without salary. The Board consists of many standing committees, including the
Written Examination Committee and the Oral Examination Committee. The mission
of these certification committees is to produce the best possible examinations
to fairly and accurately evaluate the competence of candidates for
certification.
Examination Construction
The Part-I written certification examination evaluates a candidate's
knowledge of general orthopaedics and the basic sciences of orthopaedics as
well as a candidate's ability to use this information for problem-solving in
the diagnosis and treatment of patients. It is a secured examination,
consisting of 300 to 330 multiple-choice questions, given over seven hours of
testing divided into two sessions. The examination covers acute and chronic
disease as well as disorders and injuries of the musculoskeletal system,
including the diagnosis and management of congenital, developmental,
infectious, inflammatory, neurological, vascular, metabolic, neoplastic,
degenerative, and traumatic conditions affecting the limbs and the spine. The
examination also covers the basic sciences of anatomy, pathology, physiology,
biochemistry, genetics, embryology, microbiology, immunology, pharmacology,
and molecular biology as they apply to the musculoskeletal system.
The questions are written by the ABOS Question-Writing Task Force, a group
of forty orthopaedic surgeons consisting of both private and academic
practitioners. The questions are then reviewed by a second group, the ABOS
Field Test Task Force, consisting of twenty-six practicing orthopaedic
surgeons who also represent both private and academic practices. Surgeons from
across the United States are represented on these two task forces
(Fig. 1).
The life of an ABOS written-certification question begins in August when
question assignments are made to the members of the Question-Writing Task
Force. The task force members submit their questions, all with evidence-based
references, to the National Board of Medical Examiners (NBME) in Philadelphia.
The NBME is contracted by the ABOS to edit and review questions for technical
flaws and to provide professional expertise in developing, printing, and
evaluating the written examination.
The NBME edits and corrects grammatical and technical flaws in the
submitted questions. All questions are then placed into a standard style and
format to provide consistency for the examination. The questions are further
divided into drafts on the basis of content: adult trauma, adult disease,
basic and applied science, pediatric trauma, pediatric disease, and
rehabilitation.
The ABOS Question-Writing Task Force meets in Philadelphia in the spring,
and every question is reviewed for accuracy, content, and relevancy. The NBME
then reedits the items and enters them into the ABOS question library. The
examination drafts are then administered to the members of the ABOS Field Test
Task Force and are scored by the NBME for question statistics. The Field Test
Task Force then meets in the fall in Chicago to review all of the items
selected for the examination. The questions are again reviewed for their
relevancy and accuracy with the aid of the question statistics, which are
based on how the questions actually performed when administered to the members
of the Field Test Task Force.
The NBME then assembles the examination on the basis of the ABOS content
domains and valid psychometrics; the content domains are set by the ABOS each
year. The current content domains are shown in a table in the Appendix.
During the winter meeting of the ABOS, the Written Examination Committee
reviews each question and makes final item selections. In the spring, the
chairman of the Written Examination Committee and the executive director of
the ABOS review page proofs and approve the final draft of the
examination.
The examination is administered in July to the candidates.
Annual Examination Key Validation
After the examination, the NBME performs a key validation process on every
question to identify any potentially defective items. Items can be identified
as defective if they have a very high or extremely low percentage of correct
values or if they have a statistically poor discrimination value. This means
that candidates who scored in the upper percentiles of the overall examination
tended to get the question wrong as opposed to candidates who scored poorly on
the overall examination and tended to get the question correct. All of the
defective items are reviewed by the ABOS Written Examination Committee chair,
the ABOS executive director, and members of the NBME. Items determined to be
defective are deleted from the final scoring.
After the final scoring of the examination, the ABOS Written Examination
Committee reviews the results and sets the passing standard. The passing
standard is based on a Rasch bank scale calculated in
logits2. Test
equating by the use of a question bank scale permits the examinees tested in
different years to be evaluated on a common scale. This effectively eliminates
variations in test difficulty as well as variations in the average proficiency
of examinees. There has been and continues to be a fixed minimum passing
standard based on specific content-based and compromised standard-setting
procedures that a group of practicing surgeons (the Standard Setting Task
Force) has determined must be known by qualified candidates. An in depth
review of the passing standard is performed on a regular basis because, as the
knowledge base in the practice of orthopaedic surgery continues to evolve, it
is anticipated that the standard will also change.
The candidates are notified of their results in the fall.
The final cost to the ABOS to create, edit, categorize, review, print,
administer, and statistically analyze each question is approximately $2000 per
question.
2006 Examination Psychometrics
In July 2006, the written certification examination was administered to 741
examinees. There were 321 questions administered, and six questions were
deleted in the key validation process because of defects; their deletion
enhanced the test score validity. Therefore, 315 questions contributed to the
total examination score.
The candidates consisted of 633 United States and Canadian medical school
graduates taking the examination for the first time and eighty-nine United
States and Canadian medical school graduates who were repeating the
examination. Thirteen international medical school graduates took the
examination for the first time, and six international medical school graduates
repeated the examination.
The items on the examination were analyzed psychometrically by evaluating
the mean discrimination index; this is the average point biserial correlation
coefficient of item score with total score. In essence, this analyzes the
questions to see how well items discriminated between those candidates who
obtained high scores and those with low scores on the examination. The target
for this value on an examination of this type is between 0.10 and 0.30. For
the 2006 examination, this value was 0.16.
The questions were also analyzed for the internal consistency reliability
coefficient (KR20). This analysis reflects the consistency of the
scores and calculates the standard error of measurement for the examination.
It determines the precision of examination scores or how much an examinee's
score would vary across repeated testing with different questions on the same
content. For the 2006 examination, the KR20 was 0.90.
The passing score was calibrated with use of the Rasch bank scale model,
which has been utilized by the ABOS since 1980. The Rasch bank passing score
for the 2006 examination was 1.13 logits, the same score that has been
utilized since 2000. It corresponded to a raw score of 211 and 67% correct.
This logit score of 1.13 was then equated to a standard score by means of a
linear transformation technique. With use of the standard score, a mean and
standard deviation of the Rasch score for the reference group could be
calculated. The mean standard score for the 2006 examination was 200, and the
standard deviation was 20 standard score points for the reference group. The
standard error of measurement was 9 standard score points, which meant a
candidate's true proficiency on the examination was ±9 standard score
points. This means that, if a candidate took this examination on the same
content with different questions in repeated testings, his or her score would
vary ±9 standard score points. This allows a candidate to understand
normative data as to where he or she stands relative to the mean score of the
other candidates taking the examination. The Rasch bank passing score for the
2006 examination of 1.13 logits corresponded to a standard score of 170.
The results from the 2006 examination revealed that 87.3% of all of the
candidates taking the examination passed.
Table I reveals the pass rates
for the individual subgroups taking the examination in 2006 and for the
previous five years.
In October, the candidates received their individual performance report for
the 2006 examination, which included the pass-fail decision, the minimum
passing standard score for this year's examination, their standard score, and
their percentile rank. Last year, the candidates also received their
individual performance report for the individual content areas (see
Appendix).
The program directors also received a report on how their individual
candidates performed on the examination as well as aggregate information about
the performance of the first-time takers from their specific program (see
Appendix).
The ABOS recognizes that knowledge is important for competence in our
specialty, but that knowledge does not equal competence. Unlike the Part-I
written examination, which tests exclusively orthopaedic knowledge, the
Part-II oral examination tests the application of knowledge, diagnostic
acumen, surgical techniques, outcomes, and ethics and professionalism. It
tests the applicant's ability to apply knowledge in a safe and appropriate
way.
The present oral examination has been developed over the last seventeen
years. Previous to 1993, the oral examination involved testing over four basic
content areas: pathology, pediatrics, adult, and trauma, with use of standard
cases for all candidates. In 1993, the first practice-based oral examination
was administered. The board of directors at the time, as well as the ABOS
today and during the interim period, believed that the best way to examine an
orthopaedic surgeon is to evaluate his or her own practice. The oral
examination of the ABOS is the only specialty board examination that is
totally based on the candidate's own practice. This oral examination evolved
over the years to the present examination, for which the candidate submits a
notarized case list collected over a six-month period beginning the July prior
to the examination. The case list is collected with use of a software package
entitled SCRIBE (Data Harbor Solutions, Hinsdale, Illinois), which has been
used and constantly updated since 1999. Presently, it is an Internet-based
collection system, and cases are entered with use of CPT (Current Procedural
Terminology) codes. It generates the candidate's practice profile and
complication list. Directors of the ABOS and other oral examiners select
twelve cases for the candidate. The candidate then selects ten of the twelve
cases to present at the time of the oral examination. All pertinent
information concerning these cases, including preoperative, postoperative, and
follow-up notes as well as imaging studies, laboratory studies, intraoperative
arthroscopy photographs, operative reports, and consultant notes, are required
for each case. The ABOS stringently adheres to HIPAA (Health Insurance
Portability and Accountability Act) regulations regarding privacy in the
gathering and use of this information. Pass rates have ranged from 86% to 93%
over the past decade, with 90% of the 656 candidates passing the oral
examination in 2006.
The examination is one hour and forty-five minutes in length, divided into
three thirty-five-minute segments with a five-minute break in between each
segment. During each segment, the candidate is examined by two examiners who
are matched to the candidate for areas of his or her stated expertise
(Fig. 2). For example, if a
candidate identifies his or her special area of practice as spine surgery, at
least one of the two examiners is a practicing orthopaedist who dedicates a
significant part of his or her practice to spine surgery. The examiners are
provided the complete case list as well as a graphic analysis of the
candidate's practice profile and complications.
The decision on pass-fail is based on the candidate's performance as
assessed independently by the six examiners without any caucus of the
examiners. For each presented case, the candidate is graded on data gathering,
diagnosis and interpretive skills, treatment plan, technical skills, and
outcomes. At the conclusion of each segment, the examiners grade the
candidate's surgical skills and handling of complications as well as his or
her ethics and professionalism. Each candidate therefore receives
approximately 100 to 130 grades, which are averaged and adjusted on the basis
of the known severity or leniency of the examiners. The ABOS has worked with
Measurement Research Associates (Chicago, Illinois) for over fifteen years to
develop this grading system.
A large number of new examiners are routinely recruited from both private
and academic practice, and educational methods have been developed to enhance
the training of oral examiners. Directors of the ABOS sit in on the
examinations as observers and evaluate the examiners' performance. The Board
provides feedback to the examiners on how to improve their testing methods.
Scores are adjusted on the basis of the examiners' grading severity, and
examiners receive yearly feedback on their examination performance within
three months after a given examination.
The oral examination has incorporated, over the last several years, all of
the core competencies outlined by the ACGME—to include communication and
interpersonal skills, professionalism, ethics, patient care, knowledge,
systems-based practice, and practice-based learning and improvement—in
the scoring of the examination. Last year, the ABOS provided to the residency
programs and candidates the rating definitions for the various categories by
which candidates will be evaluated and graded. These are available to the
candidates who are taking the examination and have been provided to all
program and residency directors. They can also be obtained from the ABOS
office. Several articles related to the oral examination have recently been
published3,4.