Abstract
Background:
The purpose of the present study was to evaluate musculoskeletal knowledge among graduating medical students and physician assistant students with use of a National Board of Medical Examiners (NBME) examination. We hypothesized that there would be no difference in scores between the two groups. In addition, we looked for relationships between examination scores and both the student-reported musculoskeletal experiences and the school-reported musculoskeletal curriculum.
Methods:
One hundred and forty-four students from three medical schools and ninety-one students from four physician assistant schools were included in the present study; both groups were graduating students in the final semester of education. The National Board of Medical Examiners Musculoskeletal Subject Examination (NBME MSK) was utilized to assess musculoskeletal knowledge.
Results:
The mean examination score (and standard deviation) was 73.8% ± 9.7% for medical students and 62.3% ± 11% for physician assistant students (95% confidence interval [CI], −13.8 to 0.00; p < 0.05). Medical students with an interest in orthopaedics as a career scored significantly higher than those without an expressed orthopaedic interest, and medical students without an expressed career interest in orthopaedics scored significantly higher than physician assistant students (p < 0.05). Among medical students, a longer duration of a clinical rotation in orthopaedics was associated with a higher examination score (p < 0.05). The average number of hours of preclinical musculoskeletal education in the first two years of school was significantly higher for medical schools (122.1 ± 25.1 hours) than for physician assistant schools (89.8 ± 74.8 hours) (p < 0.05).
Conclusions:
Graduating medical students scored significantly higher than graduating physician assistant students on the NBME MSK. This may be related to multiple factors, and further studies are necessary to evaluate the overall musculoskeletal clinical competence of both groups of students.
Musculoskeletal conditions comprise a substantial expenditure of the health care budget of the United States1. Musculoskeletal conditions are the second most common reason for visiting a doctor and represent the most common cause of severe long-term pain and physical disability1-3. Musculoskeletal symptoms account for approximately 20% of both primary care physician visits and emergency room visits1,2. Kurtz and Salsberg, as well as the U.S. Department of Health and Human Services, have reported an anticipated shortage of orthopaedic surgeons over the next twenty years4. Other providers, such as primary care physicians, physician assistants, and nurse practitioners, may need to assume increased responsibility for the nonoperative treatment of the musculoskeletal needs of society, but concerns have been raised by numerous investigators with regard to the musculoskeletal knowledge base of these providers2,3.
In the past forty years, physician assistants and nurse practitioners have become an integral component of the health care team, and they often function in a very independent manner, with only indirect physician supervision. Therefore, physician assistants and nurse practitioners represent a body of health care professionals who potentially will increasingly contribute to the nonoperative management of patients with musculoskeletal conditions in the future. The purpose of the present study was to determine the extent of musculoskeletal knowledge among graduating medical and physician assistant students with use of the National Board of Medical Examiners Musculoskeletal Subject Examination (NBME MSK). We hypothesized that there would be no difference in the scores for the two groups. In addition, we evaluated the relationships between the students’ examination scores and their clinical musculoskeletal rotations.
We solicited participation from all medical schools and physician assistant programs in Pennsylvania. A standardized e-mail with a preformatted template was sent to all seven allopathic medical schools in Pennsylvania, and one agreed to participate in the study. Orthopaedic educators from two other medical schools, outside of Pennsylvania, were aware of the present investigation and expressed interest in participating. Students in the final semester of education from three medical schools (Penn State College of Medicine, the University of Michigan Medical School, and the Mayo Medical School) and four physician assistant schools (Pennsylvania College of Technology, Mayo School of Health Sciences, Marywood University, and DeSales University) participated in the present study. The present study received approval from the institutional review boards at each of the participating medical and physician assistant schools. All students at these schools were asked to participate via class-wide e-mails that were sent by the class presidents. Students received at least two such e-mails in a standardized template. Participation was entirely voluntary, and students were informed that taking the examination would not influence course grades or academic transcripts in any way. All students were compensated $50 for participation. We stratified the response rate of students for the different programs and the individual schools.
Medical Students
A total of 145 medical students were recruited from a group of 345 students. The overall response rate for medical schools was 42.0% (145 of 345), with individual rates of 38.4% (forty-nine of 125) for Penn State College of Medicine, 52.9% (ninety of 170) for the University of Michigan Medical School, and 12% (six of fifty) for the Mayo Medical School.
Physician Assistant Students
A total of 105 students were recruited from a group of 132 students. The overall response rate for physician assistant schools was 79.5% (105 of 132), with individual rates of 62% (eighteen of twenty-nine) for Marywood University, 85% (thirty-four of forty) for DeSales University, 87% (twenty-six of thirty) for Pennsylvania College of Technology, and 82% (twenty-seven of thirty-three) for Mayo School of Health Sciences.
Timing, Structure, and Administration of the Examination
Because all testing was done in the final semester of school, students had completed the majority of clinical training at the time of the examination. The NBME MSK was utilized for the present investigation. Before the start of the examination, students received a packet that included a consent form and a demographics questionnaire (see Appendix). Verbal consent was obtained prior to the administration of the examination.
The NBME MSK was created after the NBME was approached by the Project 100 Educational Initiative, a subcommittee of the U.S. Bone and Joint Decade4. Funding was provided to ensure that this examination would be administered free of charge to medical schools for two years after its creation. Content was finalized and approved in 2006 by a team of clinicians and basic scientists. The NBME MSK was the first NBME fixed-form, secure, web-based subject examination written to specifically evaluate musculoskeletal knowledge. Pilot testing commenced in 2007, and, by August 2010, 1177 students had taken the examination.
The examination is composed of seventy-five items: fourteen basic science questions (18.7%) and sixty-one clinical questions (81.3%). The content is outlined in the Appendix. The basic science questions on the examination are classified as normal processes (20% to 25%) and abnormal processes (75% to 80%). The content for the clinical questions included a variety of musculoskeletal disease categories (e.g., infection, trauma, and inflammatory conditions) as well as physician tasks (e.g., diagnosis and management, health maintenance). The reliability of the examination, based on seventy-four scored items and 205 pilot examinees, was 0.86. The mean percent correct score (and standard deviation) for this group of >1000 students was 69.7% ± 13.2% (range, 28% to 99%5,6).
The examination is structured similarly to other NBME subject examinations as well as the United States Medical Licensing Examination. A clinical vignette is introduced for the majority of the items, and questions may be accompanied by radiographs or images that contain relevant information. Answer choices are multiple-choice, and the use of double negatives and multiple correct answer choices is avoided.
Students were allowed 115 minutes to complete the proctored examination, which was computer-scored by the NBME, and a percent score, based on the number of correct items for seventy-five questions, was calculated for each participant. Students were not penalized for guessing incorrectly. The NBME sent the scores to the local research investigator at each institution, who then forwarded the de-identified scores to the principal investigator (K.B.). All of the scoring was conducted by the NBME. None of the participating programs in the present investigation had been utilizing the NBME MSK routinely, and no practice sessions or examination questions had been provided to students prior to the examination sessions. None of the authors were involved in the design or selection of test items. Program directors and medical education office staff were contacted to complete a survey on preclinical and clinical musculoskeletal education (see Appendix).
Additional Variables Examined
Clinical Musculoskeletal Education
The presence or lack of a mandatory musculoskeletal clerkship or rotation by the student was assessed along with the length of student elective clinical musculoskeletal training in orthopaedics, rheumatology, and physical medicine during clinical training.
Demographic Information, Career Interests, and Musculoskeletal Experience
All student participants completed a personal survey outlining individual career interests and the extent of musculoskeletal experience to that point in their training (see Appendix).
Statistical Methods
The primary outcome of interest was the difference in musculoskeletal examination scores between medical students and physician assistant students.
Equivalence analysis was utilized to ascertain whether the two study groups were equivalent in terms of their performance on the musculoskeletal examination. In general, equivalence trials are designed to show that groups differ by no more than a specific amount, known as the equivalence margin7,8. If groups differ by more than the equivalence margin, the hypothesis is rejected, and the two groups cannot be regarded as equivalent. A power analysis was conducted. With a sample size of sixty-three per group (126 total) and a maximum allowable limit of no clinical significance of 5%, it was determined that we would have 85% statistical power to reject the null hypotheses in favor of the alternative hypothesis that the two treatment groups are equivalent in terms of percentage correct on the examination, assuming that the expected difference in means was zero and the common standard deviation was 9% with use of two one-sided tests with a significance level of 0.05 per test.
For binary student demographic and other variables, the Pearson chi-square test was used to compare the medical students with the physician assistant students. For normally distributed continuous variables, such as student age, the two-sample t test was used to compare the two groups. For continuous variables that were not normally distributed, such as the clinical musculoskeletal rotation length, the Wilcoxon-Mann-Whitney test was used to compare the two groups.
For correlation analyses involving ordinal variables, such as clinical musculoskeletal rotation length, and for continuous variables, such as musculoskeletal examination scores, the Stuart tau-c correlation coefficient (and 95% confidence interval [95% CI]) was calculated.
The present study was powered to detect a difference between medical students and physician assistant students for the primary outcome (the difference in musculoskeletal examination scores). Therefore, all other outcomes examined were exploratory in nature, and no adjustments for multiple hypothesis testing were done. All hypothesis tests were two-sided, and all data were analyzed with use of statistical software.
Source of Funding
NBME funded test administration and scoring. Compensation paid to participants was from internal department funding.
Participant Characteristics
A total of 250 students completed the NBME MSK, including 145 medical students and 105 physician assistant students. The breakdown of student programs and sex is shown in Table I. One medical student did not complete the demographics questionnaire (see Appendix), leaving 144 medical students available for inclusion. Fourteen physician assistants did not complete the demographic items on the questionnaire, leaving ninety-one physician assistants available for inclusion in the study. A total of twenty-two physician assistants (including the fourteen who were excluded from the study and eight others who were included in the study) did not indicate their academic degree. The average age was 26.1 ± 5.8 years for the ninety-one physician assistant students and 26.9 ± 2.2 years for the 144 medical students. There was a significantly greater proportion of female physician assistant students (69.2%; sixty-three of ninety one) as compared with female medical students (44.4%; sixty-four of 144) (Z = −4.44; p < 0.05). Ten (11%) of the ninety-one physician assistant students had a future career interest in orthopaedics, compared with twelve (8.3%) of the 144 medical students. There was no significant difference between the groups in terms of the percentage of students interested in orthopaedic surgery as a specialty (exact odds ratio [OR], 1.36; 95% CI, 0.5 to 3.6; p > 0.05).
NBME MSK Scores
Scores by Educational Program
The mean NBME MSK score (and standard deviation) was 73.8% ± 9.7% for the medical students and 62.3% ± 11.0% for the physician assistant students. This difference was significant as analyzed with equivalence analysis (95% CI, −13.8 to 0.00; p < 0.05).
Scores by Specialty Choice
One hundred and forty-four medical students responded to the career interest question. Medical students who were interested in orthopaedics (N = 12) performed significantly better on the musculoskeletal examination than did medical students who were not interested in orthopaedics (N = 132) (81.1% compared with 72.9%) (95% CI, −13.79 to −2.57) (p < 0.05). All ninety-one physician assistant students responded to the career interest question. For physician assistant students, no significant difference was detected between students who were interested in orthopaedics and those who were not interested in orthopaedics. Medical students who did not list orthopaedics as a career interest scored significantly better than physician assistant students, regardless of the interest among physician assistant students in orthopaedics as a career choice (95% CI, −13.33 to −7.89) (p < 0.05).
Preclinical Musculoskeletal Education Reported by Schools
The average number of hours of preclinical musculoskeletal education in the first two years of school was significantly higher for the medical schools (122.1 ± 25.1 hours) than for the physician assistant schools (89.8 ± 74.8 hours) (p < 0.05). The preclinical musculoskeletal education was further categorized into components, including anatomy, rheumatology, clinical skills, physical examination, orthopaedics, and radiology.
The medical school preclinical instruction included an average of 59.8 hours of musculoskeletal anatomy, 9.9 hours of rheumatology, 8.2 hours of clinical skills, 6.0 hours of radiology (with only one of the three medical schools having a distinct musculoskeletal radiology component), 5.9 hours of orthopaedics, and 4.6 hours of physical examination. The physician assistant school preclinical instruction included an average of 32.8 hours of musculoskeletal anatomy (constituting the largest proportion of the preclinical curriculum), 26.5 hours of physical examination, 12.1 hours of orthopaedics, 9.4 hours of rheumatology, 9.1 hours of radiology, and 7.6 hours of clinical skills.
The effect of the preclinical curriculum on musculoskeletal test scores was examined. Among physician assistant students, more hours of clinical skill instruction were significantly related to higher musculoskeletal test scores (Stuart tau-c, 0.5; p < 0.05). In addition, a greater amount of instruction in rheumatology (Stuart tau-c, 0.67; p < 0.05) and orthopaedics (Stuart tau-c, 0.29; p < 0.05) was associated with significantly higher musculoskeletal test scores. No significant association was found for anatomy (Stuart tau-c, 0.10; p > 0.05). Time spent on teaching physical examination was related to lower musculoskeletal test scores (Stuart tau-c, −0.28; p < 0.05). Among medical students, with the numbers available for study, none of the components of the preclinical musculoskeletal curriculum were associated with performance on the NBME MSK.
Clinical Musculoskeletal Experience
None of the three participating medical schools had a required musculoskeletal rotation for students during the clinical years. A four-week elective was offered at all medical schools. However, one and two-week-long orthopaedic rotations also were available for students at two of the three medical schools as part of the student general surgery clerkship during the third year of medical school. Two of the three medical schools also offered rheumatology as an elective, and only one school offered a physical medicine clinical elective. Only one of the four physician assistant schools required a clinical musculoskeletal rotation. Like the medical schools, all physician assistant schools offered orthopaedics as an elective clinical rotation. Two of the four physician assistant schools also offered a clinical rheumatology elective.
An analysis of the data self-reported by the students revealed that eighty-nine medical students (61.8%) had taken at least one musculoskeletal clinical rotation prior to this examination, compared with forty-two physician assistant students (46.2%) (OR, 0.54; 95% CI, 0.31 to 0.95; p < 0.05). However, physician assistant students reported longer-duration musculoskeletal rotations (p < 0.05, Wilcoxon-Mann-Whitney test) (Fig. 1). Among medical students, longer duration of clinical musculoskeletal training was associated with higher musculoskeletal examination scores (Stuart tau-c, 0.25; 95% CI, 0.10 to 0.40; p < 0.05). Interestingly, with the numbers studied, this relationship was not observed among physician assistant students.
In the present study, our goals were to assess and compare musculoskeletal knowledge of graduating medical school and physician assistant school students. Our interest in performing this comparison was based on (1) previous studies that used the Cognitive MSK Examination2, in which medical students and first-year residents scored lower than or comparably with chiropractor interns and physical therapy and osteopathic students9-13, (2) the increasing number of physician assistants practicing in the United States14, (3) the projected orthopaedic workforce shortage, and (4) our personal belief that physician assistants can play an important role in providing primary musculoskeletal care to patients.
In the current sample, medical students received an average of 122 hours of preclinical musculoskeletal education and physician assistant students received an average of ninety hours; this difference was significant. All of the physician assistant schools had mandatory preclinical musculoskeletal education, and one of the schools also had a required clinical rotation in orthopaedics. However, physician assistant schools had only 55.1% of the number of hours devoted to musculoskeletal anatomy compared with medical schools.
Although no passing score has been established for the current NBME MSK, previous NBME clinical science examinations have scaled scores with a mean (and standard deviation) of 70 ± 8. On the basis of these parameters, medical students were significantly more likely to pass the musculoskeletal examination compared with physician assistant students (66.2% compared with 28.6%; p < 0.001). Medical students averaged 73.8% on the seventy-five-item NBME MSK, and it would appear, therefore, that the medical student score on this examination is comparable with other, more routinely utilized NBME clinical examinations15. However, in medical school, these so-called shelf examinations are given at the conclusion of required clinical rotations, and student assessments incorporate both clinical performance and test scores. The medical schools and physician assistant schools in the present investigation do not have consistent expectations for a minimum threshold of musculoskeletal medical knowledge and clinical expertise for students, and validated assessment methods for basic clinical competence were lacking. The present study did not establish or measure a threshold level of required knowledge or expertise, and, in the absence of such criteria and assessment tools, we could not state if the clinical competency of the students who participated in the present investigation was acceptable or even different or if it was associated with the musculoskeletal knowledge measured here. Although not evaluated in the present study, we suspect that the majority of medical and physician assistant schools in North America do not have consistent expectations and requirements for meaningful, demonstrated clinical competency in the care of patients with musculoskeletal disease, and we encourage the development of this evaluation methodology.
Medical students scored better than we anticipated on this examination, and we were surprised by the level of difference in scores between the physician assistant students and the medical students. Although one possible explanation for the higher-than-expected performance of medical students was improvement in musculoskeletal medical school education since the publication of the study by Freedman and Bernstein2 in 1998, we believe that other equally or more likely explanations exist. First, this examination is a very different test than the Cognitive MSK Examination developed by Freedman and Bernstein, and a direct comparison of the two is not appropriate or our intent. Therefore, we cannot state if musculoskeletal knowledge of graduating medical students is any different than it was ten years ago. Nor can we evaluate the impact of any of the curriculum changes noted above. In addition, we hypothesized that the resources, such as review books and online question banks, that are available to medical students to help them prepare for their NBME Step-1 and Step-2 examinations may have provided an advantage for them on the NBME MSK and could explain the high performance of students who did not have any previous clinical musculoskeletal experience.
It is important to distinguish between medical knowledge and clinical competence. Medical knowledge is only one of the components of the six core competencies outlined by the Accreditation Council for Graduate Medical Education16. The other five components are patient care, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. Graduating medical students and new residents are often less comfortable with certain physical examination and clinical skills needed for the diagnosis and management of musculoskeletal disease12,17. Ensuring an adequate medical knowledge base, though essential to overall clinical competence, is not enough by itself. We emphasize that medical knowledge was the only competency evaluated in the present study, and it should not be equated with clinical proficiency in the diagnosis and primary treatment of musculoskeletal conditions.
Despite the fact that the medical students scored significantly better than the physician assistant students on this examination, it is worth noting that there are several areas in which the physician assistant students received more extensive training, such as physical examinations, and for which competency may not have been measured. The limitations of the present study included the appropriateness of generalizing the results as only three medical schools and four physician assistant programs participated in the study. Institutions deciding to participate already may have had a greater focus on musculoskeletal education or concerns about the musculoskeletal knowledge of graduates. Participation in the present study was voluntary for students and, as such, constituted a low-stakes examination, which arguably may have affected student performance and effort during the examination process. Therefore, the true musculoskeletal competence of students may not be fully reflected in the scores. An additional limitation of the present study was the difference in the response rate between medical students (41.7%) and physician assistant students (77.1%). However, we believe that, despite this difference, we had representative samples of each group, which was exemplified by a similar number of students in both groups being interested in orthopaedics.
One may argue that it was inappropriate for us to compare these two groups of students. Indeed, the NBME MSK was designed and intended to be used for assessment of musculoskeletal knowledge among medical students and by design is broader in content coverage compared with examinations utilized in prior studies on musculoskeletal knowledge among health care professionals2. Medical students have more years of formal medical education and have undergone a more competitive and selective acceptance process. However, from a quality-of-care perspective, we believe that it is important to recognize that the recent physician assistant graduate can provide care to patients with indirect supervision from a physician after the graduate has passed the certification examinations. It also is important to recognize that there are no specific requirements for additional musculoskeletal training. We believe that it is appropriate for patients to expect all health care practitioners to have a basic fund of musculoskeletal knowledge that allows them to provide appropriate, safe, and efficient care. In fact, this opinion was also emphasized in an official statement by the American Association of Physician Assistants (AAPA): “…Because they [physician assistants] train using similar curriculum, training sites, faculties and facilities, physicians and PAs [physician assistants] develop a similarity in medical reasoning during their schooling that eventually leads to standardized thought in the clinical workplace; PAs think like doctors.”18
In summary, we assessed musculoskeletal knowledge of graduating medical students and physician assistant students with use of a recently developed examination created by the NBME. The finding that medical students performed significantly better than physician assistant students on this examination may be related to a variety of factors.
The student questionnaire on musculoskeletal knowledge, the content outline of the NBME MSK, and the program director survey on musculoskeletal curriculum are available with the online version of this article as a data supplement at jbjs.org.
Note: The authors wish to acknowledge the following groups and individuals for having made this research possible. The authors thank all participating institutions, Dr. Carpenter (Chair of Orthopaedics) and Dr. Larry Gruppen (Chair of the Department of Medical Education) of the University of Michigan Health System for having made research funds available to support the project at Michigan, Ms. Carolyn Cole Brown (our Department Administrator) for assistance in running the study, and the National Board of Medical Examiners for their generous contribution and for assistance with the study.
Craton
N;
Matheson
GO. Training and clinical competency in musculoskeletal medicine. Identifying the problem. Sports Med.
1993;15:328-37.
Freedman
KB;
Bernstein
J. The adequacy of medical school education in musculoskeletal medicine. J Bone Joint Surg Am.
1998;80:1421-7.
Woolf
AD;
Walsh
NE;
Akesson
K. Global core recommendations for a musculoskeletal undergraduate curriculum. Ann Rheum Dis.
2004;63:517-24.
U.S. Bone and Joint Decade. USBJD and NIH discuss increased awareness, collaboration and partnership. .
Snapinn
SM. Noninferiority trials. Curr Control Trials Cardiovasc Med.
2000;1:19-21.
Schuirmann
DJ. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. J Pharmacokinet Biopharm.
1987;15:657-80.
Matzkin
E;
Smith
EL;
Freccero
D;
Richardson
AB. Adequacy of education in musculoskeletal medicine. J Bone Joint Surg Am.
2005;87:310-4.
Humphreys
BK;
Sulkowski
A;
McIntyre
K;
Kasiban
M;
Patrick
AN. An examination of musculoskeletal cognitive competency in chiropractic interns. J Manipulative Physiol Ther.
2007;30:44-9.
Stockard
AR;
Allen
TW. Competence levels in musculoskeletal medicine: comparison of osteopathic and allopathic medical graduates. J Am Osteopath Assoc.
2006;106:350-5.
Childs
JD;
Whitman
JM;
Sizer
PS;
Pugia
ML;
Flynn
TW;
Delitto
A. A description of physical therapists’ knowledge in managing musculoskeletal conditions. BMC Musculoskelet Disord.
2005;6:32.
Sandefur
R;
Febbo
TA;
Rupert
RL. Assessment of knowledge of primary care activities in a sample of medical and chiropractic students. J Manipulative Physiol Ther.
2005;28:336-44.
American Academy of Physician Assistants. 2008 AAPA physician assistant census report. .
National Board of Medical Examiners (NBME). .
Accreditation Council for Graduate Medical Education (ACGME), Common program requirements: general competencies. .
Lynch
JR;
Schmale
GA;
Schaad
DC;
Leopold
SS. Important demographic variables impact the musculoskeletal knowledge and confidence of academic primary care physicians. J Bone Joint Surg Am.
2006;88:1589-95.
White
GL
Jr;
Egerton
CP;
Myers
R;
Holbert
RD. Physician assistants and Mississippi. J Miss State Med Assoc.
1994;35:353-7.
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. One or more of the authors, or his or her institution, has had a financial relationship, in the thirty-six months prior to submission of this work, with an entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. Also, one or more of the authors has had another relationship, or has engaged in another activity, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.