Abstract
Update
This article was updated on September 5, 2012, because of a previous error. On page e113(1), the text had listed one of the authors as “Andrea Polinijo.” The text now reads .“Andrea Polonijo.”
An erratum has been published: J Bone Joint Surg Am. 2012;94(18):e141.
Background:
Despite advances in understanding the “systems-based practice” competency in resident education, this topic has remained difficult to teach, assess, and document. The goal of this study was to perform a needs assessment and an analysis of the current state of systems-based practice education in orthopaedic residency programs across the U.S. and within our own institution.
Methods:
A sample of orthopaedic educators and residents from across the U.S. who were attending the 2010 American Orthopaedic Association (AOA) Effective Orthopaedic Educator Course, AOA Resident Leadership Forum, and AOA Council of Residency Directors meeting were surveyed to determine (1) which aspects of systems-based practice, if any, were being taught; (2) how systems-based practice is being taught; and (3) how residency programs are assessing systems-based practice. In addition, an in-depth case study of these issues was performed by means of seven semi-structured focus group sessions with diverse stakeholders who participated in the care of musculoskeletal patients at the authors’ institution. A quantitative approach was used to analyze the survey data. The focus group data were analyzed with procedures associated with grounded theory, relying on a constant comparative method to develop salient themes arising from the discussion.
Results:
“Clinical observation” (33%) and “didactic case-based learning” (23%) were reported by the survey respondents as the most commonly used teaching methods, but specific topics were taught inconsistently. Competency assessment was reported to occur infrequently, and 36% of respondents reported that systems-based practice areas were not being assessed by any methods. The focus group discussions emphasized the need for standardized experiential learning that was closely linked to the patient’s perspective. Orthopaedic faculty members were uncomfortable with their knowledge of this competency and their ability to teach and assess it.
Conclusions:
Teaching the systems-based practice competency occurs inconsistently, and formal assessment occurs infrequently. In addition to formal teaching, learning systems-based practice will be best achieved experientially and from the patient’s perspective.
Since implementation of the results of the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project1 in 2001, graduating residents in the U.S. are required to demonstrate appropriate knowledge, skills, and behaviors in six “core competencies,” and residency programs are required to measure and document their students’ accomplishments in them. Specifically, the “systems-based practice” competency requires residents to demonstrate an awareness of, and responsiveness to, the larger health care system as well as the ability to effectively call on system resources to provide care that is optimal and effective.
More specifically, the ACGME has stated six expectations within systems-based practice for which residents must achieve and demonstrate competency: (1) Work effectively in various health care delivery settings and systems relevant to their clinical specialty. (2) Coordinate patient care within the health care system relevant to their clinical specialty. (3) Incorporate considerations of cost awareness and risk-benefit analysis in patient and/or population-based care as appropriate. (4) Advocate for quality patient care and optimal patient care systems. (5) Work in interprofessional teams to enhance patient safety and improve patient care quality. (6) Participate in identifying systems errors and implementing potential systems solutions.
Despite advances in understanding the overall competencies and developing appropriate assessment tools2, systems-based practice has remained a difficult subject area to teach, assess, and document3-6. The ACGME competencies were created a decade ago, and yet fourth-year medical students have been reported to be largely unfamiliar with them. In one investigation, seventy-three of 193 medical students reported having no knowledge of any of the physician competencies and only 10% reported knowledge of systems-based practice7. Residents focus more on learning medical knowledge and patient care, and perceive other areas as less important8. In addition, few educators feel sufficiently comfortable with our complex, ever-changing, and expanding health care system to serve as experts in systems-based practice and assessors of resident knowledge in this competency9. Therefore, the challenges are to determine (1) what information needs to be learned to fulfill this competency, and (2) how best to teach this information, the optimal delivery method, and how to effectively assess resident learning.
In addition, health care costs, quality, and safety have come under increased public scrutiny in the past decade, and the health care “system” and associated regulations have become increasingly complex to understand and navigate. The knowledge, judgment, and communication skills required for competency in systems-based practice are important for a resident’s development as a health care professional, and emphasis needs to be placed on acquisition of these skills during residency training.
As a result, we engaged in a needs assessment and an analysis of the current state of systems-based practice education in orthopaedic residency programs across the U.S. and within our own institution. To accomplish this, we sought to determine which, if any, aspects of systems-based practice were being taught in the U.S., and by what means systems-based practice was being taught and assessed. In addition, we evaluated our own institution and local community to gain a deeper understanding, from other stakeholders in the health care system, of deficiencies in our awareness of and responsiveness to the larger health care system and the ability to effectively call on system resources to provide care that is optimal and effective.
Survey of Orthopaedic Professionals
Survey information was collected from orthopaedic residents and educators who attended the 2010 American Orthopaedic Association (AOA) Effective Orthopaedic Educator Course, the AOA Resident Leadership Forum, and the AOA Council of Residency Directors meeting. Written survey information was collected from sixty-nine senior orthopaedic residents and sixty-two orthopaedic faculty members attending the conferences. The response rate of the survey was 57%.
Based on our review of the literature and experiences in resident education, eleven important systems-based practice areas were presented for consideration: “practice management,” “hospital finance,” “professional finance,” “scope of practice of mid-level providers and physician oversight responsibilities,” “physician employment options,” “role of other non-physician members of the health care team,” “orthopaedic professional societies and infrastructure,” “federal and state regulations,” “quality measures, systems error and patient safety,” “roles and relationships between different health care facilities and guidelines for admission,” and “health care plans and access.” Respondents were asked to report whether their residency program taught these topics (options were “yes,” “no,” or “don’t know”). In addition, respondents were asked how systems-based practice was being taught. Possible responses included “clinical observation,” “didactic case-based learning,” “online program,” “self-directed study,” and “quality improvement project.” Finally, participants were asked how each competency area was assessed. The choices were “global evaluation form,” “written or web-based test,” “360 evaluation,” “other,” and “not assessed.”
Cross-Sectional Focus Groups
Seven semi-structured focus group sessions were conducted with diverse stakeholders who participated in the care of musculoskeletal patients at the institution of the senior author (K.P.B.). Focus group participants included (1) faculty and (2) residents from the Department of Orthopaedics and Rehabilitation; (3) nursing leadership (outpatient clinic, inpatient orthopaedic floor, and operating room); (4) social work, inpatient physical therapy, and utilization review staff; (5) risk management, quality leadership (including the Chief Quality Officer and Chief Medical Officer), and legal counsel staff; (6) senior hospital administrative leadership; and (7) community orthopaedic surgeons with whom orthopaedic residents complete a rotation.
Groups were selected to encourage a willingness of focus group members to speak freely. A semi-structured focus group protocol was used to support open conversation among the participants regarding their understanding of systems-based practice competency, desired learning outcomes, gaps in residents’ knowledge and behavior, current and proposed methods of teaching and assessment, and anticipated implementation barriers. Each session was led by three of the investigators (K.P.B., D.D.P., and S.J.-S.), with the exception of the resident session, from which the Department Chairman (K.P.B.) excused himself. Discussions were recorded and transcribed verbatim. Procedures most closely associated with grounded theory were used to analyze focus group data by relying on a constant comparative method to develop salient themes arising from the discussion.
Source of Funding
Funding for this study was provided by the Woodward Endowment for Medical Science Education. That organization played no role in the investigation.
Survey of Orthopaedic Professionals
What Is Being Taught?
The mean rate of affirmative responses for the eleven important systems-based practice topics areas was only 56% (Table I). There was agreement between the two groups of respondents, with a mean of 54% of residents and 57% of orthopaedic educators giving an affirmative response. Teaching in five of the eleven areas (“roles and relationships between different health care facilities and guidelines for admission,” “hospital finance,” “professional finance,” “scope of practice of midlevel providers and physician oversight responsibilities,” and “health care plans and access”) was reported by <50% of total respondents.
How Is Systems-Based Practice Being Taught and Assessed?
Overall, “clinical observation” (33%) and “didactic case-based learning” (23%) were reported by the respondents as the most commonly used methods of instruction. Online programs (10%), self-directed study (7%), and quality improvement projects (5%) were not reported as often (Table II).
Competency assessment was reported to occur infrequently, and for ten of the eleven topics, residents were more likely than faculty to report that assessment did not occur (Table III). Use of a “global evaluation form,” “written or web-based test,” “360 evaluation,” and “other” were reported by 8%, 8%, 4%, and 3% of the total respondents, respectively. Thirty-six percent of respondents, 29% of educators and 42% of residents, reported that systems-based practice areas were not being assessed by any methods.
To summarize the survey data, clinical observation was reported to be the most common learning tool; without being coupled with a clear assessment plan, this appears to be a passive approach to teaching. Systems-based practice is being taught passively and inconsistently, and assessment of this competency occurs infrequently.
Cross-Sectional Focus Groups
Building on the data gathered from the survey of orthopaedic educators across the U.S., focus groups were used to gain an in-depth understanding of the issues related to systems-based practice in a more comprehensive and instructive way.
The seven focus group sessions revealed a substantial difference in understanding of the concept of systems-based practice across the stakeholder groups. Orthopaedic faculty demonstrated poor understanding of, and frustration with, this competency. Although residents and faculty largely focused on efficiency in task completion, nursing and allied health professionals emphasized the need to teach residents effective communication and collaboration skills as well as an understanding of complex processes internal and external to the hospital. Administrators emphasized deficits in residents’ understanding of hospital operations, finance, law, and insurance as these relate to clinical practice, whereas quality and safety leaders identified the need for improved understanding of one’s place in the larger health care system and how systems-related issues impact patient care. Community orthopaedic surgeons prioritized the need for residents to learn how to run an efficient practice and develop appropriate time management skills. Several of our focus groups involved people with little or no direct responsibility for teaching or assessing residents, such as some members of the allied health or administrative staff. Members with such responsibilities reported that teaching methods were variable and inconsistent. They also reported that methods of assessment were inadequate. There was consensus regarding the need for standardized experiential learning, closely linked to the patient’s perspective, which does not currently exist. Anticipated difficulties were noted both in defining systems-based practice for the attending physicians who must teach it and in engaging task-oriented residents in learning these necessary skills.
Our review of the literature indicated that there is an increasing amount of research in teaching the systems-based practice competency and that a variety of approaches have been taken. Didactic lecture series10 are feasible, and the additional use of audience response systems has been reported to support learner engagement and provide feedback to the lecturer11. “Morbidity and mortality” conferences12 have been used by a number of programs across different specialties. David and Reich13 developed a curriculum in systems-based practice and managed care in a monthly workshop format for internal medicine residents, and they concluded that it can increase residents’ understanding of, and comfort with, various topics in managed care and systems-based practice.
Eskildsen14 reported the use of a web-based module to provide training and assess competency in systems-based practice but stated, “Ideally, this product should be used in the context of a clinical rotation so that learners can expand their learning and acquire transferable skills using this online tool as a starting point.” In this statement, the author points out that a substantial pitfall of an entirely web-based system is that, at its core, the systems-based practice competency focuses on the members of the health care system as a team and the ability of residents to function within that team. A web-based system most often encourages individualized learning in an isolated environment, and it should therefore not be the sole method of teaching a competency that, at its very core, is team and systems-based. Computer-based learning is appropriate and convenient for well-defined and individual learning experiences dealing with topics such as basic patient safety information and health care system regulations. In these cases, information can be organized easily into learning modules, and learning can be assessed with web-based pretests and posttests with good results15.
The use of group projects to teach systems-based practice can strengthen residents’ competency in teamwork with other health care providers as well as with non-physician members of the hospital staff16,17. In addition to enhancing collaboration skills, such programs have the potential to identify inefficiencies in the system and result in quality improvement projects.
Simulation also has been reported as a method to teach systems-based practice. Larkin et al.18 reported that the development and implementation of a two-year pilot “human factors curriculum” focusing on interpersonal and communication skills, systems-based practice, and professionalism resulted in significant improvement in empathetic communication by junior surgical residents. Wang and Vozenilek19 also described utilization of a simulation-based curriculum to address systems-based practice competencies pertinent to emergency medicine, and they noted that the curriculum allowed residents to interact with the health care team in a more realistic way than in a case-based discussion curriculum.
Finally, experiential systems-based practice learning assignments for first-year medical residents provide opportunities for residents to build rapport with non-physician providers and other members of the health care team with whom they work on a daily basis. Through this approach, residents have reported an increased appreciation of non-physician members of the health care system, awareness of care opportunities, and efficiency in various patient care tasks20. Buchmann et al.21 also highlighted the value of using opportunities in daily clinical work to teach systems-based practice concepts to radiology residents. In addition, Sutkin and Aronoff22 reported improved resident appreciation for the training and responsibilities of office staff after residents experienced two days with the “front office” staff. Turley et al.23 demonstrated a significant increase in posttest compared with pretest knowledge after designing and implementing a five-day systems-based practice rotation focusing on the interrelationships of patient care, clinical revenue, and the physician’s role within the health care system.
Peters et al.24 utilized a web-based curriculum that incorporated exercises requiring interviews and consultations with other members of the heath care system and that encouraged the communication and mutual understanding that is essential to the systems-based practice competency. The content focused on health care system finance and quality improvement. In the process of completing a business improvement plan and a root cause analysis, residents became immersed in the practical experiences of their hospital, allowing them to better understand systems of care. Residents who participated in this program demonstrated improvements in knowledge and self-assessed competency that were greater than those in a control group of residents, making the project not only a valuable learning experience for the residents but also a potential source of cost savings for the hospital.
We were not surprised by the survey data collected in this investigation. Although many respondents to our survey reported teaching some aspect of systems-based practice, the data suggest that such teaching is inconsistent and variable across orthopaedic training programs. Most programs rely on passive teaching approaches using clinical observation, without assessment of the resident’s knowledge of systems-based practice. We conclude that systems-based practice competency is not being assessed routinely for any of the topics, and that the emphasis on “clinical observation” for teaching and the lack of assessment indicate a deficit in the learning experience of our residents in the area of systems-based practice. It is particularly noteworthy that orthopaedic faculty demonstrated significant frustration with, and poor understanding of, this competency (“I never learned this; how can I teach it?”), yet clinical observation was reported as the most common method of teaching.
We believe that the most critical piece of information gathered from this study is related to the gap between the investigators’ perceived goals of teaching and learning systems-based practice and what we learned from other health care system members regarding what were considered to be deficiencies in orthopaedic resident (and faculty) knowledge and performance. Specifically, orthopaedic faculty and residents were perceived by other health care members as lacking in (1) communication and collaboration skills, and (2) understanding of complex processes internal and external to the hospital. Therefore, although we believe that resident education in topics such as Medicare, rehabilitation hospitals, and oversight requirements for midlevel providers is extremely important, it is also essential to provide a learning environment in which residents experience the health care system through the eyes of the patient and other health care professionals—that is, in the “real world” setting with the entire health care team. In other words, much as patient care cannot be learned simply by means of lectures or studied materials, systems-based practice competency cannot be achieved exclusively through these same methods. Although core knowledge about systems-based practice is essential, competency in systems-based practice requires residents to learn how to interact effectively within the system on behalf of patients. Such knowledge is best learned experientially through a curriculum that requires interaction with various aspects of the very system that they are to learn.
This study included the use of representative members of different stakeholder groups in focus group sessions. In the interest of creating a manageable discussion, these representatives were chosen for their leadership roles and experience. The reported deficits in orthopaedic resident competency in systems-based practice did not necessarily reflect the assessment of all members of these groups. In addition, the focus group sessions reflected the opinions of individuals working within one academic medical center and a nearby private practice orthopaedic group, and they may not be representative of the opinions of individuals at other institutions.
Based on information collected from this investigation, we are in the process of completing specific learning objectives, a curriculum, and methods of instructional delivery involving reading materials and assessment tools. The latter will include both an examination with multiple-choice questions as well as a 360-degree assessment tool specifically focusing on resident performance in systems-based practice. Finally, our residents now complete two one-week-long “health systems rotations” during their postgraduate year-1 schedule. These two weeks are incorporated into their three months of orthopaedics during that year. This is a semi-structured experience in which we intentionally provide specific learning opportunities but also give residents the opportunity to explore new areas as they arise from their unique experiences.
We view our work as an innovative step in providing better learning experiences in systems-based practices for our residents. Certainly, some aspects of the health care system can be formally taught or learned by lectures, web-based modules, reading, and other traditional means. However, we believe that critical aspects of systems-based practice must be seen through the eyes of non-physician providers and patients. By spending time with patients and other members of the health care system, residents will experience systems-based practice in ways that are not possible through more traditional teaching. We recommend that the system be used to teach residents about systems-based practices. Only then will we find ways to realize the implied promise of developing systems-based practices that will improve the health of our patients.
Graham
MJ;
Naqvi
Z;
Encandela
J;
Harding
KJ;
Chatterji
M. Systems-based practice defined: taxonomy development and role identification for competency assessment of residents. J Grad Med Educ.
2009;1(
1):49-60.[PubMed][CrossRef]
Didwania
A;
McGaghie
WC;
Cohen
E;
Wayne
DB. Internal medicine residency graduates’ perceptions of the systems-based practice and practice-based learning and improvement competencies. Teach Learn Med.
2010;22(
1):33-6.
Elwood
D;
Kirschner
JS;
Moroz
A;
Berliner
J. Exploring systems-based practice in a sample of physical medicine and rehabilitation residency programs. PM R.
2009;1(
3):223-8. .
Frenk
J;
Chen
L;
Bhutta
ZA;
Cohen
J;
Crisp
N;
Evans
T;
Fineberg
H;
Garcia
P;
Ke
Y;
Kelley
P;
Kistnasamy
B;
Meleis
A;
Naylor
D;
Pablos-Mendez
A;
Reddy
S;
Scrimshaw
S;
Sepulveda
J;
Serwadda
D;
Zurayk
H. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet.
2010;376(
9756):1923-58. .
Smith
CS;
Morris
M;
Langois-Winkle
F;
Hill
W;
Francovich
C. A pilot study using cultural consensus analysis to measure systems-based practice performance. Int J Med Ed.
2010;1:15-18.
Wasnick
JD;
Chang
L;
Russell
C;
Gadsden
J. Do residency applicants know what the ACGME core competencies are? One program’s experience. Acad Med.
2010;85(
5):791-3.
David
RA;
Reich
LM. The creation and evaluation of a systems-based practice/managed care curriculum in a primary care internal medicine residency program. Mt Sinai J Med.
2005;72(
5):296-9.
Harris
EE;
Abdel-Wahab
M;
Spangler
AE;
Lawton
CA;
Amdur
RJ. Results of the Association of Directors of Radiation Oncology Programs (ADROP) survey of radiation oncology residency program directors. Int J Radiat Oncol Biol Phys.
2009;74(
2):327-37.
Mitchell
JD;
Parhar
P;
Narayana
A. Teaching and assessing systems-based practice: a pilot course in health care policy, finance, and law for radiation oncology residents. J Grad Med Educ.
2010;2(
3):384-8.
Varkey
P;
Karlapudi
S;
Rose
S;
Nelson
R;
Warner
M. A systems approach for implementing practice-based learning and improvement and systems-based practice in graduate medical education. Acad Med.
2009;84(
3):335-9.
Davison
SP;
Cadavid
JD;
Spear
SL. Systems-based practice: education in plastic surgery. Plast Reconstr Surg.
2007;119(
1):410-5.
David
RA;
Reich
LM. The creation and evaluation of a systems-based practice/managed care curriculum in a primary care internal medicine residency program. Mt Sinai J Med.
2005;72(
5):296-9.
Eskildsen
MA. Review of web-based module to train and assess competency in systems-based practice. J Am Geriatr Soc.
2010;58(
12):2412-3. .
Kerfoot
BP;
Conlin
PR;
Travison
T;
McMahon
GT. Web-based education in systems-based practice: a randomized trial. Arch Intern Med.
2007;167(
4):361-6.
Delphin
E;
Davidson
M. Teaching and evaluating group competency in systems-based practice in anesthesiology. Anesth Analg.
2008;106(
6):1837-43.
Patterson
BR;
Kimball
KJ;
Walsh-Covarrubias
JB;
Kilgore
LC. Effecting the sixth core competency: a project-based curriculum. Am J Obstet Gynecol.
2008;199(
5):561..
Larkin
AC;
Cahan
MA;
Whalen
G;
Hatem
D;
Starr
S;
Haley
HL;
Litwin
D;
Sullivan
K;
Quirk
M. Human Emotion and Response in Surgery (HEARS): a simulation-based curriculum for communication skills, systems-based practice, and professionalism in surgical residency training. J Am Coll Surg.
2010;211(
2):285-92.
Wang
EE;
Vozenilek
JA. Addressing the systems-based practice core competency: a simulation-based curriculum. Acad Emerg Med.
2005;12(
12):1191-4. .
Eiser
AR;
Connaughton-Storey
J. Experiential learning of systems-based practice: a hands-on experience for first-year medical residents. Acad Med.
2008;83(
10):916-23.
Buchmann
RF;
Deloney
LA;
Donepudi
SK;
Mitchell
CM;
Klein
SG. Development and implementation of a systems-based practice project requirement for radiology residents. Acad Radiol.
2008;15(
8):1040-5.
Sutkin
G;
Aronoff
CK. Resident front office experience: a systems-based practice activity. Med Educ Online.
2008;13:6.
Turley
CB;
Roach
R;
Marx
M. Systems survivor: a program for house staff in systems-based practice. Teach Learn Med.
2007;19(
2):128-38.
Peters
AS;
Kimura
J;
Ladden
MD;
March
E;
Moore
GT. A self-instructional model to teach systems-based practice and practice-based learning and improvement. J Gen Intern Med.
2008;23(
7):931-6.
Disclosure: One or more of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of an aspect of this work. None of the authors, or their institution(s), have had any financial relationship, in the thirty-six months prior to submission of this work, with any entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. Also, no author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.