The physical examination (PE) is an essential part of the clinical encounter in every field of medicine. Within orthopaedic surgery, the physician must often assimilate information from multiple diagnostic modalities to interpret the patient’s history and PE findings. In order to determine clinically relevant findings, the physician must have a mastery of the musculoskeletal PE, which is fundamental for making appropriate treatment decisions. The PE is often underemphasized in residency education. A perhaps greater concern is that there may be underutilization of PE skills in clinical practice, with a corresponding overutilization of imaging technology. As recently described on National Public Radio1, the fading art of the PE and technology’s replacement of skills are universal issues in medicine. These concerns have also been recently echoed by Shelbourne with regard to the knee examination2.
The importance of the PE in a resident’s education is clearly evident from a review of the core competencies designated by the Accreditation Council for Graduate Medical Education (ACGME). In July 2002, the ACGME instructed programs to require competency in six areas: patient care, interpersonal and communication skills, medical knowledge, professionalism, practice-based learning, and systems-based practice. In a survey of orthopaedic residency program directors and residents, patient care was considered to be the most important of these six areas3. Patient care competency includes not only the operative procedure but preadmission and follow-up care as well as nonoperative outpatient diagnosis and treatment. Residents should have two half-days of outpatient clinical experience per week during which they are directly supervised by faculty in performing preoperative and postoperative assessments4. Thus, up to 20% of a resident’s time should be spent examining patients and receiving instruction on PE skills. The amount of time that a resident spends in the clinical setting can be regulated by the ACGME. However, the level of instruction in proper musculoskeletal PE skills, a hands-on discipline, cannot be directly controlled.
The most effective way to teach and assess a resident’s PE skills remains unclear at this point. Most literature on this issue has addressed how to incorporate general musculoskeletal education within a training program at the undergraduate or medical student level5–11. Some studies have shown the effectiveness of computer-based modules in teaching performance of the musculoskeletal PE12,13, of the use of trained patients as teachers14, and of teaching15 of core skills to medical students by medical professionals other than faculty. Research dealing with training at the residency level comes mainly from physiatry16,17 and rheumatology18,19 and further highlights the need for improvement.
Although online PE demonstrations and textbooks describing a generalized approach to the musculoskeletal PE do exist, we assume that residents are learning their PE skills by practicing in the clinical setting under direct observation by the attending physician, who can provide feedback regarding the resident’s technique. However, we have no data indicating how often this occurs in the clinical setting, where volume and efficiency are becoming increasingly important. With more stringent work-hour restrictions, residents spend less time examining patients in the clinical setting. Although there has been much concern regarding how work-hour restrictions will affect a resident’s total operative experience20,21, there has been little research evaluating the effect on clinical experience. Furthermore, we have no means to evaluate a resident’s PE skills on graduation. Although the Orthopaedic In-Training Examination (OITE) and board certification examinations can assess knowledge regarding a description of a particular maneuver or pathology, a multiple-choice question will not provide proof that a resident can perform the PE correctly.
Our interest in evaluating the musculoskeletal PE skills of residents came about serendipitously. Our emphasis had been focused on how we teach medical students the performance of the musculoskeletal PE. We created a multiple-choice musculoskeletal PE test to assist us in evaluating our medical students’ musculoskeletal PE knowledge and understanding of skills. We wanted to use this test to track the effect of changes that we were making in our three-week musculoskeletal clinical course for second-year students. As part of the validation process of this test, we administered it to our entire residency, expecting excellent scores. The scores that we received were unexpectedly low, inspiring us to take a closer look at how well we are training residents in the PE.
The purpose of the present study was to evaluate all facets of the musculoskeletal PE as they pertain to an orthopaedic residency: attitudes toward it, knowledge of it, and ways in which we can improve how we teach it and track the results of those changes.
Surveys
We created a five-question survey (Table I) to assess attitudes toward musculoskeletal PE training and sent it to 110 orthopaedic department chairs throughout the U.S. via regular mail. We also employed an online survey company to send reminder e-mails over the following two months. We created a similar survey to assess residents’ attitudes regarding the musculoskeletal PE training that they were receiving (Table II). We administered this survey to twenty-four of our orthopaedic residents in postgraduate year [PGY] 2 through 5.
Multiple-Choice Test
After obtaining institutional review board approval, we created a thirty-question multiple-choice decision-making musculoskeletal PE test based on the most pertinent skills from each orthopaedic subspecialty. Sample questions and answers are shown in Table III. In an effort to establish content validity, we also sent these questions to all of the U.S. orthopaedic department chairs along with our survey. We asked the department chairs to rate the importance of each subject on a scale of 0 to 10, with 0 = not important, 5 = important, and 10 = very important.
We administered this test to the thirty orthopaedic residents in PGY 1 through 5 of our program. To compare our residents’ PE knowledge with their overall cognitive musculoskeletal knowledge, we also compared the test results to our program’s overall percentage score on the OITE for the most recent year.
Curriculum Changes
As in many orthopaedic programs, our residency has a set of goals and objectives for each rotation. These objectives are in line with the ACGME core competencies, knowledge expectations, and surgical skills expectations. On reviewing the objectives, we realized that there were no objectives involving the PE skills to be learned during the individual rotations. Therefore, we developed a PE skills checklist for each rotation (trauma, musculoskeletal oncology, pediatrics, spine, infection, general and prison service, joints, foot and ankle, shoulder, sports, and hand). With the help of subspecialists in each rotation area, we created a checklist of specific PE requirements in which each resident must exhibit competency before completing the rotation. The resident is provided with the appropriate checklist at the start of each rotation. At some time near the completion of the rotation, each attending physician is asked to directly observe the resident performing a complete PE.
To assess compliance with the above changes, we have modified the electronic evaluation that each resident and attending physician completes at the end of a rotation. Both evaluations now include a question asking specifically whether the attending physician directly observed the resident examining a patient.
Development of the Objective Structured Clinical Examination (OSCE)
Our final step was to develop an OSCE for our residents. The goal was to evaluate residents’ communication, history-taking, PE, and decision-making skills and to track our program’s progress in developing and improving residents’ overall musculoskeletal PE skills. Our institution is fortunate to have a state-of-the-art clinical skills laboratory, which houses fourteen ambulatory examination rooms for standardized patient encounters. Examination rooms are equipped with ceiling-mounted cameras, and encounters may be digitally recorded for off-site viewing as well as monitored on site.
We developed four clinical scenarios, again with assistance from subspecialists. The scenarios involved an upper extremity case, a lower extremity case, a trauma case, and a spine case. We trained the standardized patients for these encounters by providing them with a written explanation of the scenario and then spending two different training sessions with them. The first session lasted ninety minutes, and the second session lasted thirty minutes and occurred immediately prior to the OSCE.
We created checklists that included the key components of the history that we wanted the resident to elicit as well as the relevant PE skills. Each checklist included twelve to fifteen key points that we expected the resident to ask for and obtain during the encounter. The musculoskeletal PE checklist also included the basic skills: inspection, palpation, joint range of motion, strength testing, and any special tests pertinent to the clinical scenario. To ensure that the checklists were reproducible and to add to their validity, we based them on the Medicare guidelines for coding a level-3 new-patient visit and ensured that we included all pertinent aspects of the musculoskeletal PE that would be required for accurate documentation. We developed a point system for the checklists, and each resident would obtain numerical scores on completion of the OSCE that included subscores for history-taking skills and for PE skills as well as the overall score.
We evaluated the interrater reliability of our checklists to verify that our raters were able to detect actual changes in performance of the history-taking and PE skills. This evaluation involved having three raters view the same video of each scenario and use the same checklists for the history-taking and PE. We calculated the intraclass correlation coefficient (ICC) model 2,1 coefficients to determine whether we could be confident of the musculoskeletal OSCE results assessed by a single randomly chosen rater selected from a population of experienced raters. We also calculated the ICC model 2,3 coefficients to determine the improvement in reliability achieved by averaging the scores from three raters to assess performance on the OSCE22,23.
The residents were given ten minutes with each standardized patient to take a complete history and perform the relevant PE. There was a warning alarm at eight minutes. Each encounter was videotaped for later review. Immediately following each scenario, the resident completed a five-question decision-making test based on the encounter. This test was taken on a computer outside the examination room and was designed to assess the resident’s ability to synthesize information obtained from the encounter. The questions involved appropriate diagnoses, subsequent steps in evaluation, and initial treatment plans. The answer key had been pre-entered, and the computer program automatically calculated the score for later appraisal.
Each resident was also rated on communication and interpersonal skills by the standardized patient with use of a previously validated score card24. After each encounter, the standardized patients entered this information into the computer system, which again calculated the score for subsequent assessment.
Videos taken from two different angles were available. The videos were reviewed by subspecialists in the appropriate area who used the checklists as a guide. The resident was given points for asking correct questions and for performing different aspects of the PE with proper technique. The predetermined scoring system for the checklists was utilized to determine overall scores.
The overall score on completion of an OSCE scenario thus had four components: the history-taking and the PE as graded with use of the developed checklists, the five-question post-test, and the evaluation by the standardized patient. These four subscores for each scenario, the overall score for each scenario, and the overall score for the entire OSCE were all reported. On the day of the OSCE, the resident was given password-protected access to view his or her videos and the scores given by each standardized patient. This provided the ability for the resident to self-assess his or her performance, although this was optional. The residents also had the opportunity to review their performance on the OSCE by meeting with the residency education director to obtain direct feedback.
Source of Funding
There was no external funding for this study.
Surveys
Fifty-four (49%) of the 110 U.S. orthopaedic department chairs responded to our survey (Table I). Eighty-one percent believed that their residents spent enough time, an average of approximately one-third of the week, in the clinical setting. Although approximately 92% thought that educational time dedicated to teaching PE skills was important, nearly 63% reported that there was not enough time in the clinical setting to observe a resident examining a patient. Moreover, nearly 70% of respondents stated that their residents’ PE skills were routinely observed and critiqued in the clinical setting only rarely (8.3%) or occasionally (60.4%).
The resident survey raised similar concerns. The twenty-four residents in PGY 2 through 5 completed the survey (Table II). Although 100% of the residents reported that they spent enough time in the clinical setting, only 20% believed that faculty took enough clinic time to teach PE skills. Only 5% believed that there was enough time in the clinical setting to devote to teaching and observing PE skills. Finally, 100% stated that PE skills were routinely observed and critiqued in the clinical setting only rarely (50%) or occasionally (50%).
Multiple-Choice Test
All thirty residents in our program (PGY 1 through 5) completed the thirty-question multiple-choice decision-making musculoskeletal PE test. We administered this test to the residents in the process of establishing content validity for another test, and we fully expected the residents to perform well. However, the mean test score was 76% for all residents and 81% when interns were excluded. Excluding the interns (who had a mean score of 63%), the mean scores did not differ significantly according to the year in training, ranging from 78% to 86%. The content of the test was validated by the survey of the department chairs regarding the level of importance of each question (which again had a 49% response rate). All content was given a score of at least 5 and was thus deemed to be important, and the mean score for the questions was 7.12. This test was administered in the spring of 2009, and the scores were noticeably lower than our program’s mean OITE score in 2008, which was 93%. Given that our residents consistently scored above 90% on the OITE, we revised the test for our medical students at a much more basic level.
OSCE
Twenty-two of the twenty-four residents in PGY 2 through 5 participated in the OSCE; the remaining two residents were unavailable because of prior commitments. The mean overall score, which averaged all four components (history, PE, five-question post-test, and standardized patient evaluation) for each scenario, was 66%. The mean score for the trauma scenario (78%) was significantly better than those for the shoulder (67%), spine (64%), and knee (59%) scenarios (p < 0.05, Fisher exact test) (Table IV).
When all scenarios were combined, the mean score was 53% for the history component of the OSCE, 60% for the PE, 64% for decision-making (the five-question post-test), and 90% for communication skills (Table V). The only significant difference according to years in training involved communication skills, which were better for residents in PGY 5 compared with PGY 2 (p = 0.03).
OSCE Checklist Reliability
A set of three expert raters independently assessed the history-taking and PE skills of the twenty-two residents as demonstrated on the videotapes taken at each musculoskeletal OSCE station. A different set of raters was used for each of the four stations (knee, spine, shoulder, and trauma). Table VI provides the ICC 2,1 and 2,3 values and corresponding 95% confidence intervals for the PE and history-taking components of each of the four scenarios. The ICC 2,1 values for the history-taking and PE components ranged from 0.27 (fair) for the trauma PE to 0.90 (almost perfect) for the spine history; all of the remaining ICC 2,1 values indicated substantial reliability. Averaging the scores from the three raters improved the reliability to almost perfect for all ICC 2,3 values except that for the trauma PE, which improved from fair to moderate. Trauma rater 2 appeared to provide generally higher scores than raters 1 and 3, and pairwise correlations and graphing confirmed this and showed rater 2 to have a low interrater correlation with considerable discrepancies when compared with raters 1 and 3.
The impetus for this study stemmed from our residents’ poor performance on the multiple-choice PE test that we had created to assess medical students’ knowledge on completion of their musculoskeletal course. The test provided us with a baseline for our residents’ PE knowledge. More importantly, however, it opened our eyes to our residents’ lack of knowledge in this area. The fact that we noticed no significant improvement in scores with increasing years in training further highlighted the need for redesign of the curriculum.
The results of the surveys of department chairs and residents draw attention to the fact that the normal clinical environment may not be the best setting for residents to hone their PE skills. Although both department chairs and residents agreed that the PE is an important component of training, there was believed to be insufficient time in the clinical setting to devote to teaching and observing residents as they actually examine patients.
Thus, the dilemma facing those involved in orthopaedic residency education today is finding time in the current health-care system to teach the PE. With the ACGME’s new competency-based approach to residency education, assessment of performance has become a main area of interest, and direct observation is offered as a tool to assess knowledge and skills4. Although success in acquiring medical knowledge can be determined with a written examination, other equally valued competencies—primarily patient care, interpersonal and communication skills, and professionalism—may be best evaluated by observing and assessing residents interacting with patients and providing feedback on performance. Multiple studies have demonstrated the value of directly observed clinical experiences in increasing the comfort and ability of learners in the areas of PE and history-taking25–27. Members of various other specialties28–32 have already published their experiences with the use of direct observation and called for its use in their training programs. We are aware of no such reports in the orthopaedic literature, but we believe that direct observation will be an important topic as we strive to improve residents’ education in the musculoskeletal PE.
We are aware of no guide for teaching or structuring a curriculum in the musculoskeletal PE for orthopaedic residents. Although video programs demonstrate performing a PE of various normal joints, these programs often do not include abnormal findings, which residents also need to see. Moreover, watching a specific maneuver and performing it are different processes. We have created PE skills checklists that residents are given when they start each rotation. We hope that by emphasizing the components of the checklist at that time, residents will seek out certain PE findings while in the clinical setting to ensure that they can recognize these findings and accurately perform the relevant tests. Residents are to review the checklists with their attending physician midway through each rotation and again on completion of the rotation. It is also our goal for the attending physician to directly observe one patient encounter during each resident’s rotation. We instituted these changes in July 2010. Although it is too early to report results, we believe that we are creating a cultural change within our residency program, with an increased emphasis on the musculoskeletal PE.
The OSCE concept has been utilized for years and has gained widespread acceptance in medical curricula33. For instance, the OSCE is currently part of the medical licensure process in Canada and Australia, and it has also been incorporated into Step 3 of the United States Medical Licensing Examination (USMLE)33,34. Many disciplines have implemented OSCEs as part of their training programs35–39. We are aware of only one previous report in the literature regarding its use in orthopaedics40. However, given the importance of direct observation of skills and subsequent feedback, we judged that instituting an OSCE in our curriculum to evaluate musculoskeletal PE, interpersonal, and communications skills was worthwhile. In an effort to investigate the use of OSCEs in other orthopaedic programs in the U.S., all 113 members of the Association of Residency Coordinators in Orthopaedic Surgery (ARCOS) were asked whether they employed standardized patients or a structured clinical encounter at their institution to supplement PE training. The response rate was 78% (eighty-eight of 113), and only three other programs (2.6%) offered OSCE-type PE training.
Our OSCE results are important for a number of reasons. First, they highlighted the deficiencies noted in the PE skills of our residents. The mean overall score for all residents was 66%, and the mean score on only one of the four scenarios (78% for trauma) represented a passing score of >70%. Second, the OSCE has provided us with a benchmark for tracking our residents’ results and the improvement following our curriculum changes.
Our study has several limitations. First, we do not have complete validation data for our scenarios, our checklists, or the point system that we created to score the OSCE. Although we employed the assistance of subspecialists from a large academic medical center, included the five standard components of a normal PE, and based our history-taking checklists on the Medicare guidelines for a satisfactory new-patient history, the OSCE was still developed in a subjective fashion and will undergo further validation as we move forward. We obtained good reliability data for our checklists, as the ICC values for all but one of the history-taking and PE subscores for the scenarios (the trauma PE) indicated substantial or almost perfect reliability41. There were discrepancies between trauma PE rater 2 and raters 1 and 3, which placed the reliability data for this subscore in the fair category. Overall, however, obtaining validity evidence has proven to be difficult for all tools developed for the direct observation and assessment of clinical skills in trainees26,27. Therefore, our findings truly apply only to our program, and we must be careful not to generalize our results to all residencies.
Second, because we utilized standardized patients in our OSCE, much of each encounter depended on how well the standardized patient understood the case and on his or her ability to simulate the relevant PE findings. We used experienced standardized patients and instructed them on two separate occasions prior to the OSCE. However, they were not medical personnel and likely made mistakes in their role during the encounter. Although this may have led the resident to interpret the PE findings incorrectly, and thus negatively affected the score on the five-question post-test, the ability of the resident to perform the PE properly should not depend on the responses or behavior of the standardized patient. Despite these weaknesses, we are pleased with our first OSCE.
Finally, the results of our survey were weakened by the 49% response rate of the department chairs. We cannot generalize that the remaining one-half would have responded in a similar fashion.
We hope that we have demonstrated a need for reform in how we teach the musculoskeletal PE. With the trend toward overutilization of advanced imaging technology, the PE is indeed at risk of becoming a lost art. We hope to reverse this trend. As Shelbourne stated recently, “a new emphasis on educating physicians (of all kinds) about how to correctly and thoroughly examine the knee (and other joints) is needed.”2 Our work presented here is the first step in creating a cultural change within orthopaedic residencies today, to a model in which the PE receives a much greater emphasis. We envision creating a more standardized format for teaching the musculoskeletal PE that can be modeled by many programs. We intend to continue to refine our OSCE, PE checklists, and decision-making PE test. We have received sponsored funding and started work on an online video database for the musculoskeletal PE. These videos will demonstrate a normal PE for each anatomic area as well as abnormal PE findings and testing maneuvers, allowing for year-round self-study by residents and medical students. Finally, we will continue to work on changing the culture during rotations so that we can find some time to teach the PE in the clinical setting and directly observe our residents’ performance. We hope that our experience will stimulate others to examine how they are teaching the PE and to evoke a similar cultural change.