With all of the constraints facing residency programs today, such as increasing work-hour restrictions, a changing economy and its accompanying financial pressures, and the current generation of physicians' enhanced interest in lifestyle quality, it is imperative for orthopaedic educators to identify and select the best possible residents to fill our residency programs. Several recent editorials from major orthopaedic journals have discussed identifying resident quality and maximizing success during residency. One such commentary stressed that continuously refining and improving resident education and training are of the utmost importance to the future of orthopaedic surgery (and, indeed, to the future of medicine) because doing so today will facilitate the continued recruitment of top medical students to orthopaedic surgery1. In another recent editorial, in The Journal of Bone and Joint Surgery (American Volume), Deputy Editor Marc Swiontkowski remarked that the Orthopaedic Residency Review Committee's expanded role within the Accreditation Council for Graduate Medical Education (ACGME) has recently enabled us to attain a number of important goals in postgraduate education with respect to interviewing and selecting candidates for residency as well as educating and counseling residents during residency2. In order to build on these improvements, Swiontkowski stressed that we must continue to innovate and improve all aspects of orthopaedic surgery postgraduate education. Therefore, it is important for orthopaedic surgery postgraduate training programs to evaluate the means by which they identify and select the candidates who are likely to succeed during residency and, ultimately, in practice. Moreover, identifying predictors or factors associated with success during orthopaedic surgery residency is critical knowledge for program directors, selection committees, and students.
Currently and for the last twenty years, resident selection decisions have been made by a committee of faculty at our institution that reviews all of the applications and then generates a consensus ranking of all of the applicants. The number of applicants is typically large and the means by which such decisions are made is subjective and specific to our individual program. Some of the criteria considered by our committee over the years have included the completion of a rotation at our institution, class rank, interview, performance, the quality of letters of recommendation, student transcripts, the letter from the dean of the applicant's medical school, and the scores on standardized licensing examinations3-7.
A number of studies have described the difficulty we currently face in predicting which students are most likely to succeed during residency3,8. Having prior empirical knowledge of which preresidency selection factors significantly correlated with success in an orthopaedic surgery residency is therefore desirable for program directors because such knowledge will aid their selection committees in making better informed decisions when narrowing down a large applicant pool for a limited number of residency positions.
The primary aim of this study was to identify which preresidency selection factors and metrics within the training program were associated with what we considered to be a good overall performance in an orthopaedic surgery residency program on the basis of strict criteria. The secondary aim of the study was to identify which preresidency selection factors were associated with successful performance for individual metrics within the program.
This study was reviewed and approved by the institutional review board at our institution. One hundred and fifty medical students matched or transferred into our ACGME-accredited orthopaedic surgery training program between 1983 and 2002. Of those, 147 who completed the program and had complete residency application data available were evaluated. During the first fifteen years of the study period, the program consisted of six residents per year and for the last five years it was expanded to twelve residents per year. During the course of the training, all residents were under the direction of the same residency program director and underwent a similar didactic and clinical training program. In the study, resident files containing data from the resident application and residency performance were reviewed. Information obtained from a typical resident file included medical school transcript, medical school diploma, scores on standardized licensing examinations (National Board of Medical Examiners and United States Medical Licensing Examinations), curriculum vitae, the letter from the dean of the medical school, a personal statement from the applicant, letters of recommendation, an interview day evaluation sheet (which included a check box for having completed a rotation at our institution), and a letter of notification from the dean's office in the event of election to Alpha Omega Alpha (AOA, the honorary medical society).
The criteria for defining successful metrics during residency were agreed on by the senior authors of the study and were based on previous literature as well as their combined twenty-five years of experience as residency program directors. Individual measures of success during residency included scores on the Orthopaedic In-Training Examination (OITE), successful completion of Part I of the American Board of Orthopaedic Surgery (ABOS) certification examination on the first attempt, a mean clinical performance score based on evaluations following each clinical rotation during the residency, being named an executive chief resident (selected by the residency program director on the basis of the following criteria, which required that the resident [1] must be considered a role model for others, [2] possess a good fund of knowledge, [3] have leadership skills, and [4] have the ability to multitask), and the number of peer-reviewed articles the resident had published during residency training (Table I). The clinical performance evaluation (CPE) was our program's precursor to the current system of core competencies and took into account patient care skills, medical knowledge, surgical skills, interpersonal relationships, and systems-based practices. The residents were graded on a scale of 1 to 10 for each clinical rotation, and the score represents a mean for each of the postgraduate years.
Preselection variables available in the resident files that were used to correlate with successful performances in various aspects of resident duties included scores on part one of the standardized licensing examination, election to the national medical school honor society Alpha Omega Alpha, class rank (a lower number is better), U.S. News and World Report ranking of the medical school attended9 (a lower value is better), the number of peer-reviewed articles published before residency, military service, completion of an additional degree, number of honors grades earned during preclinical and clinical years of medical school, the letter of recommendation from the medical school dean (a numerical value was given on the basis of key words within the last sentence of the text; a higher value is better), completion of a previous rotation at our institution, having a previous career, participation in a varsity sport in college, being a semiprofessional musician, holding a peer-elected office in a national or college-wide organization, and substantial charitable involvement (Table II).
Data from the application file and residency records were reviewed exclusively by members of our clinical research staff who were not familiar with any of the residents being evaluated. We performed two different analyses. The first analysis compared preresidency selection criteria and certain performance measures during the residency program with our own subjective classification of good performance or poor performance overall. The criteria for performance classification were based on the current ACGME core competencies and comprised the following: scoring in the clinical performance evaluations (patient care and interpersonal skills), passing the ABOS certifying examination (medical knowledge) on the first attempt, and disciplinary action taken against the resident (professionalism), i.e., probation or repeating of a year during his or her training. A resident was considered to be a poor performer in the residency program if any of the following occurred: failure of the ABOS examination, any formal disciplinary action received during the training program, or scoring in the lowest 5% in clinical performance evaluation. We used the Student t test to test the differences in continuous variables and the Fisher exact test to examine the categorical variables in good compared with poor performers. A p value of =0.05 was considered significant.
Second, we looked at the preresidency selection factors that were associated with success in individual metrics in various aspects of resident performance. Differences in resident outcomes between all dichotomous predictors were determined with use of the Student t test. For continuous predictors, we measured the association with successful outcomes by the regression coefficient from a simple linear regression model. Correlation coefficients were also derived from these same models. We applied the standard level of significance of 0.05. Only significant associations are reported. The Fisher exact test was used to measure the associations between dichotomous outcomes and dichotomous resident characteristics.
The demographic makeup of our urban, university-based orthopaedic residency program over the twenty-year period studied included 141 men (96%) and six women (4%). Of those 147 residents who trained at our institution, 106 (72%) were white, thirty-seven (25%) were Asian, two (1.5%) were Hispanic, and two (1.5%) were black.
According to our criteria for overall performance, 138 residents (94%) were considered good in their overall performance during their training program and nine (6%) were considered poor. None of the preresidency selection variables were associated with what we considered good performance by a resident (Table III). The only identifiable factor that was correlated with good performance during the residency was the average OITE score. Residents who were considered to be good performers scored, on the average, in the 55th percentile compared with the 35th percentile for those who were considered to be poor performers during their residencies (p = 0.007) (Table IV).
With regard to specific aspects of resident performance, we found associations with the following preresidency selection factors. (1) The residents who attended a higher-ranked medical school (p = 0.043) and had higher scores on part one of the standardized licensing examination (p = 0.001) scored higher on their in-training examination (medical knowledge) (Table V). (2) The residents who completed a rotation at our institution during the fourth year of medical school (p = 0.02), those with a history of substantial charitable participation (p = 0.010), and those who had a lower ranking (more desirable) in the National Residency Matching Program match list (p = 0.010) performed better on their clinical rotations as measured by the clinical performance evaluation (patient care, medical knowledge, professionalism, and systems-based practice) throughout their residencies (Table V).
The only other associations between preresidency selection factors and individual measures were that varsity athletes in college were significantly more likely to be appointed to executive chief resident during residency (40% compared with 15% who were not varsity athletes; p = 0.045) and a positive association between the number of peer-reviewed articles published during residency and the number of such articles published prior to residency (p = 0.008) (Table V).
In this retrospective review of a large, university-based residency program over twenty years, we found only a higher mean percentile score on the OITE to be correlated with an overall good performance during residency on the basis of simple criteria. While one would expect OITE scores to be associated with better evaluations by the attending surgeon, we can envision a clinical practice evaluation that defines good performance of a surgeon despite the fact that he or she scored poorly on standardized testing. In addition, we identified several preresidency selection factors that correlated with a successful performance with regard to certain aspects within our orthopaedic surgery residency program. Our results indicate that purely academic preresidency selection factors, including medical school rank and scores on part one of the standardized licensing examination, are positively correlated with higher average percentile scores on the OITE. Since students at higher-ranked medical schools often have higher Medical College Admission Test scores, it is logical that these students, as well as those who scored highest on part one of the standardized licensing examination, would be most likely to succeed on another standardized test such as the OITE. People who do well on standardized tests usually continue to do so throughout their training. However, other, nonacademic preselection factors that constitute a well-rounded individual, including the completion of a rotation at our institution where the faculty could get to know the applicant to a greater degree, and substantial charitable involvement, correlated with nonacademic (clinical) success during training. Interestingly, we did not find a positive correlation between any factor and passing the ABOS examination on the first try. This may have been due to the very high number of residents who successfully completed Part I on the first try (142 of 147 candidates, or 97%). When we used very strict criteria for successful completion of our training program, we found an association between residents with consistently poor performance on the OITE and an overall poor performance in the program.
Clinical performance measures, such as patient care, systems-based practice, and professionalism, are strongly influenced by individual life experiences that foster teamwork, commitment to excellence, and a strong work ethic. Therefore, perhaps, it makes sense that residents who are well-rounded (charitable involvement and varsity sports) would perform better than those who have not had these experiences. The correlation between lower rank on our department's National Residency Matching Program rank list and better clinical performance is not a surprise to us and reinforces the strength of our selection process. This result also indicates that, during the study period, our selection committee did a good job of interviewing prospective candidates and selecting those who would succeed.
Residents who were former varsity athletes received higher retrospective faculty evaluations on knowledge and were more likely to be named executive chief resident than were those who had not been collegiate varsity athletes, suggesting that the leadership skills, team player mentality, sense of individual responsibility, accountability, and time management skills often seen in varsity athletes facilitated successful performance and positive evaluations during residency. It is not surprising that students with a greater number of articles published before residency would generate a greater volume of articles published during residency as they have already shown an interest in academic pursuits.
We chose to look at preselection factors associated with positive performance in this study. While some studies have found that excellent medical students make excellent residents10,11, and numerous studies have found that students who perform well on standardized testing during medical school will likewise perform well on standardized testing during residency12-15, most of the literature has shown that success as a resident and as a clinician in the most complete sense does not necessarily correlate with superior academic performance during medical school14-20; consequently, further studies on resident education are needed.
There are many definitions of the various attributes of a good doctor; some are cognitive, some are ethical, and some are interpersonal, depending on whom you ask21-23. For example, physicians on the average regard honesty, responsibility, and trustworthiness as the most important qualities in a good physician, while the public regards being knowledgeable and keeping up-to-date as the most important qualities23. We chose to evaluate specific measures of successful performance as an orthopaedic resident in this study using a variety of criteria that were both academic (standardized testing scores, passing Part I of the ABOS certifying examination, and the number of articles published) and nonacademic (being named executive chief resident and being highly ranked after clinical rotations). Analogous measures of success were used in other similarly designed studies in the literature24,25.
Only a few prior studies have attempted to correlate preselection factors from the residency application with success during residency. In one study, Erlandson et al. looked at 103 surgical house officers (forty-one in general surgery and sixty-two in specialty surgery) who entered their residency program between 1975 and 197926. Success was measured by a monthly faculty evaluation assessing performance during residency and by their American Board of Surgery In-Training Examination (ABSITE) score through the second year of the program. The predictors examined included scores on part one of the standardized licensing examination, preclinical course honors, clinical course honors, surgical clerkship honors, AOA membership, published research, medical school grading system, medical school rating, and National Resident Matching Program rank. They found that medical school honors, AOA membership, and medical school grading system were significant indicators of success during residency. A limitation of that study is its limited definition of success, whereby only two outcomes were analyzed, and one of those outcomes, in-training examination scores, was only analyzed through the second year. Like Erlandson et al., we also used certain preresidency factors to predict success during residency, but our study does not support the finding of Erlandson et al. Our study employed a more comprehensive definition of success, both overall and with regard to specific metrics, and focused on residents in one surgical specialty (orthopaedic surgery) over a much longer study period (twenty years). Further, our findings were more specific and correlated with various specific aspects of performance during residency.
In another predictors study, Fine and Hayward retrospectively examined the residency application files of 123 internal medicine residents at their home institution from 1989 to 199219. Success was measured as the final overall evaluation score submitted to the American Board of Internal Medicine for each resident by the program. Preselection factors looked at in that study included sex, internal medicine clerkship grade, number of honors in nonmedicine clerkships, AOA membership, number of articles published, score on part one of the standardized licensing examination, and medical school reputation. Fine and Hayward found that the intern selection committee score given to each residency applicant correlated only somewhat with their overall evaluation scores. In addition, the only factors that were found to be significantly correlated with the final evaluation scores of the residents were honors achieved in an internal medicine clerkship and graduation from the medical school where the residency program was located. Our study had a larger cohort size (147 residents) and longer study period (twenty years), identified several additional predictors of success, and used a more comprehensive definition of success, including OITE scores, passing the ABOS examination the first time, executive chief resident status, and contemporary clinical performance evaluations by the faculty.
Dirschl et al. reviewed the application files of fifty-eight orthopaedic surgery residents from 1983 to 1997, in order to determine which criteria in a resident's application had the highest correlation with performance as an orthopaedic resident25. Resident performance was measured by faculty rankings of cognitive, affective, and psychomotor abilities as well as by performance on the OITE and the ABOS examination. Dirschl et al. found that the number of honors grades during clinical clerkships in medical school was the most predictive of resident performance, followed by AOA membership. They also found a correlation between the number of fine-motor extracurricular activities and psychomotor performance. By contrast, the cohort size in our predictors study was about three times greater, and our study period was more than five years longer. We did not find a correlation between honors grades during years 1 and 2 or during the clinical clerkships (years 3 and 4) and success during residency. Again, we found a number of additional correlations in our study.
Daly et al. sought to identify predictors of residency and postresidency success in otolaryngology in a retrospective cohort study of thirty-six residents from twenty-four medical schools over a ten-year period24. Success was defined as ranking in the highest tertile of a numerical ranking of the residents by seven otolaryngologists who had been on the faculty for the length of the study period. They found that having more than one article published during residency and having scores of 570 or higher on part one of the standardized licensing examination were associated with being favored by >50% of the evaluators. Postresidency success was predicted by having an exceptional trait as mentioned in the letter of recommendation for residency and AOA membership (p = 0.02). Some limitations of their study were its very small sample size (thirty-six residents), its relatively short study length (ten years), and its relatively arbitrary definition of resident success (the faculty providing the rankings were not given any uniform guiding criteria when assigning an overall rank for each resident). We also found that scores on part one of the standardized licensing examination were predictive of resident success, and we found numerous additional predictors of success in our program.
One limitation of the current study is that the data were obtained retrospectively, and are therefore subject to considerable bias. Second, since some of these data were compiled contemporaneously, we were limited to the materials available in each resident's file, and it is possible that certain preselection criteria were not provided by the residents in their application files. Despite this limitation, we still had more data than any other published study of this type, and our data were both complete and thorough because of excellent record-keeping. Furthermore, the criteria that define a successful resident were subjectively determined by the senior authors of the study. Although subjective, this decision was based on previous literature as well as on the authors' combined twenty-five years of experience as residency program directors. This is, however, why we chose to present the data both as a dichotomous model (good compared with poor performers) and as individual measures of successful performance, as well as to present the factors that correlate with each measure, so that the readers can use this information as they like. While we use the terms successful and unsuccessful and good and poor, it should be noted that all of the residents in this study graduated from the program and all but one eventually passed the ABOS certifying examination and are, to our knowledge, currently in practice. Thus, being considered a poor resident performer does not preclude one's ability to perform successfully within the medical community once training has been completed. Finally, we must interpret any of the results with caution as most, if not all, of the preresidency factors measured in this study are correlated and, thus, it is difficult to tease out the independent effect of any one specific factor.
Possible future applications of this study would be to help orthopaedic surgery program directors to fine-tune the selection criteria for their residency programs and possibly to establish a structured process to select residents possessing criteria in their applicant file that are predictive of success during residency. Such a process was attempted and validated in an obstetrics and gynecology residency program27, but these findings have yet to be replicated in orthopaedic surgery residencies. This study may also have useful future applications with respect to benchmarking and possibly revealing flaws in the current metrics used in resident evaluation. Future studies should be conducted to build on our results. One such study might include a broad-scale multi-institutional investigation to examine whether these findings are reproducible in other orthopaedic surgery residency programs throughout the United States.