Abstract
Background:
Acquisition of surgical skills through laboratory-based training and simulation is appealing to surgical training programs. The purpose of this study was to provide baseline information on the current use of surgical skills training laboratories in orthopaedic resident education and to determine the interest in expansion of these facilities and training techniques.
Methods:
The creation of the survey was a collaborative effort between the authors and the American Academy of Orthopaedic Surgeons (AAOS). Two online versions of the Surgical Skills Simulation survey were created, one (with twenty-three items) specifically for program directors and one (with fourteen items) for orthopaedic residents. The survey was sent via e-mail to 185 program directors and 4549 residents. Data were retrieved and analyzed by the AAOS Department of Research and Scientific Affairs.
Results:
Eighty-six (46%) of the 185 surveys distributed to orthopaedic surgery residency directors and 687 (15%) of the 4549 distributed to orthopaedic surgery residents were completed. Seventy-six percent of the program directors reported having a surgical skills laboratory, and 46% of these reported having a structured surgical skills laboratory curriculum. Fifty-eight percent of program directors and 83% of residents believed that surgical skill improvement by orthopaedic residents was not being objectively measured. Both 80% of program directors and 86% of residents agreed that surgical skills simulations should become a required part of training, and 82% and 76% were interested in a standardized surgical skills curriculum. Eighty-seven percent of program directors identified a lack of available funding as the most substantial barrier to development of a formal surgical skills program at their institution.
Conclusions:
There was strong agreement among both program directors and residents that surgical skills laboratories and simulation technology should be a required component of orthopaedic resident training. At the present time, the most substantial barrier to adoption of surgical skills laboratories and a formalized surgical skills curriculum is the lack of funding.
Laboratory-based training to acquire and improve surgical skills through surgical simulation is of increasing importance to surgical residency training programs1. The appeal of learning basic skills through repeated practice away from the direct patient care environment is clear and compelling, although such practice is not a replacement for traditional methods of acquiring surgical skills in the operating room. Recent concerns about patient safety, resident work hour restrictions, and better technology have further increased interest in laboratory-based simulation2. In general surgery, a well-defined curriculum for simulated surgical skills has been developed, and since July 2012 the General Surgery Residency Review Committee of the Accreditation Council for Graduate Medical Education (ACGME) has required that general surgery training program resources must include simulation and skills laboratories. In addition, these facilities must address acquisition and maintenance of skills with a competency-based method of evaluation3.
Orthopaedic surgery has made less progress in adopting laboratory-based training techniques, and only recently have the Orthopaedic Surgery Residency Review Committee and the American Board of Orthopaedic Surgery (ABOS) begun the process to require a curriculum and laboratory-based training. However, the extent to which laboratory skills training has been incorporated into orthopaedic residency training as well as the interest in and obstacles to further expansion of these training techniques are unknown. The purpose of this study was to provide baseline information on the availability and the current use of surgical skills training laboratories in orthopaedic resident education and to determine the interest in expansion of these facilities and training techniques. The study was conducted through a national survey of orthopaedic program directors and orthopaedic residents.
Survey Development
The survey was developed to prepare for the November 4, 2011, Orthopaedic Surgery Simulation Summit conference sponsored by the American Academy of Orthopaedic Surgeons (AAOS). This conference had representatives from orthopaedic accreditation bodies, residency directors, and representatives from orthopaedic subspecialty societies, and its goals were to advance surgical simulation and laboratory-based skills training throughout the continuum of orthopaedic resident education. The survey was developed by the authors as a tool to inform the conference participants about the current and future use of skills laboratories and simulation in orthopaedic resident education. The results were reported at the conference by one of the authors.
Subjects and Survey Administration
Two online versions of the Surgical Skills Simulation survey were created, one (with twenty-three items) specifically for program directors and the other (with fourteen items) specifically for orthopaedic surgery residents. The survey was sent via e-mail to 185 program directors and 4549 residents on June 14, 2011. Within the e-mail, the invitees were provided with a brief explanation of the purpose of the survey and were asked to click on a link that would lead them to the appropriate version of the survey. E-mail reminders were sent to all invitees on a weekly basis starting on June 21, 2011. The survey was closed on September 15, 2011.
Data Analysis
Data were retrieved and analyzed by the AAOS Department of Research and Scientific Affairs. All responses are presented as counts, percentages, or means. The 95% confidence interval (CI) is presented for the responses to each survey item. The CI indicates that, if this survey were repeated 100 times, we could expect the responses to fall within this range ninety-five times. The size of the CI will vary depending on the number of responses to the item.
Demographics
Eighty-six (46%) of the 185 surveys distributed to orthopaedic surgery residency directors and 687 (15%) of the 4549 surveys distributed to orthopaedic surgery residents were completed. There was an even distribution of respondents among postgraduate years in the latter group.
Facilities
Surgical skills facilities were not available at nearly one-quarter of training sites (Table I). The training facility size also varied widely among the programs that did have such facilities, with 25% of facilities being <500 ft2 (46.5 m2), whereas 48% were >1000 ft2 (93.0 m2).
Access and Equipment
Resident access to the skills facility varied, with 48% of programs with such facilities reporting scheduled access only, whereas 30% provided residents with twenty-four-hour access (Fig. 1). Most programs (85%) had surrogate bone models (e.g., Sawbones [Pacific Research Laboratories, Vashon, Washington]) available, and 69% of programs reportedly had the ability to obtain human cadaveric specimens on request. Basic surgical instruments and arthroscopy towers were widely available (in 80% of programs). C-arm fluoroscopy was available in only 22% of skills laboratories (Fig. 2).
Funding
Forty-three percent of program directors were unsure of the source of the start-up funding of their institution’s skills lab, and nearly 48% were unaware of the annual budget or operating expenses. Departmental and hospital financial support were the largest contributors to both start-up funding and ongoing operating funding (Figs. 3-A and 3-B). Program directors were also asked how much they would pay to obtain one or more surgical simulators that help residents to learn basic skills. As shown in Figure 4, the most common responses were between $1000 and $15,000, with only 15% willing to invest >$30,000.
Curriculum
Of the program directors who reported having a surgical skills laboratory, 54% reported no structured surgical skills laboratory curriculum beyond basic anatomy and intermittent exercises with surrogate bone models. Furthermore, in 53% of the programs that did have such a curriculum, residents had less than one protected hour per week of exposure to the curriculum (Fig. 5).
Learners and Instructors
The majority of program directors indicated that the primary intended learners were orthopaedic junior residents (95%) and orthopaedic senior residents (70%). The majority of respondents (98%) indicated that structured surgical skills instruction was provided by orthopaedic faculty members (Fig. 6). For both of these questions, the percentages totaled >100% because respondents were able to indicate all answers that applied.
Assessment
Fifty-eight percent of responding orthopaedic program directors believed that improvement in surgical skills by orthopaedic residents was not objectively measured in their program. Of those who thought that it was measured, the most commonly reported method was faculty evaluation forms (Fig. 7). A significantly greater proportion (83%, t[79] = −6.70, p < 0.001) of responding residents indicated that surgical skill improvement was not objectively measured.
Accreditation and Certification
When asked whether surgical skills simulation should become a required part of residency training, 80% of program directors agreed (Table II). However, when asked whether surgical skills simulation should become part of board certification or recertification, only 33% agreed (Table III).
Barriers to Progress
The greatest perceived barrier to progress in this area was money. Eighty-seven percent of program directors identified lack of available funding as the most substantial barrier to development of a formal surgical skills program at their institutions. Lack of a curriculum was the second most common obstacle (Fig. 8).
Comparison of Program Directors with Residents
There were more similarities than differences between the two groups. A higher percentage of program directors (43%) than residents (17%) believed that orthopaedic resident surgical skills improvement was objectively measured (t[79] = −6.70, p < 0.001). Both program directors (80%) and residents (86%) agreed that surgical skills simulation should become a required part of future orthopaedic resident education (t[681] = 0.23, p = 0.817). In both groups, the percentage of support for making surgical skills simulations part of the certification process was sharply lower (33% and 31%). As expected given the agreement that surgical skills simulation should be a part of resident education, both program directors (82%) and residents (76%) were interested in the establishment of a standardized surgical skills curriculum (t[770] = 3.57, p < 0.001).
With an ever-changing health-care environment, surgical educators face the challenge of training competent future surgeons amid mounting clinical, financial, and regulatory burdens. Issues related to patient safety cast a negative light on the current educational techniques in which learners are placed in the operating room as near novices. Increased restrictions on resident work hours, an emphasis on operating room efficiency, and the financial pressures in the current health-care environment further restrict the tolerance for a new resident’s initial skills learning curve being in the operating room.
There is mounting evidence that this learning curve can be safely advanced outside of the operating room by existing simulation technology, with the possibility of an associated decrease in intraoperative technical errors and thus a reduction in patient morbidity. Most of this evidence demonstrates the construct and/or face validity of simulations. For instance, in orthopaedic surgery, studies involving cadaver-based simulations of carpal tunnel surgery have shown that it is possible to distinguish between the performance of residents with different experience levels4. Most of the evidence that simulation can improve actual operating room performance has been in general surgery and involved use of the Fundamentals of Laparoscopic Surgery box trainer for laparoscopic cholecystectomy. In one of many studies on this device, Seymour et al. demonstrated that surgical simulation training improved the operating room performance of general surgery residents during actual laparoscopic cholecystectomy5. Although there is less evidence of direct transfer of simulation skills to the operating room in orthopaedics, Howells et al. demonstrated a significant and objective improvement in performance of arthroscopic Bankart suture repair utilizing an arthroscopic shoulder simulator6.
In contrast to general surgery, in which laboratory-based skills training is required by the associated boards and residency review committees, current Orthopaedic Surgery Residency Review Committee program requirements do not have such a mandate, although one is being developed. The American Board of Surgery currently requires completion of the Fundamentals of Laparoscopic Surgery skills-based curriculum for board certification7. The American Board of Orthopaedic Surgery does not utilize a direct assessment of surgical skills for initial board certification or maintenance of certification.
This survey sampled the opinions of both orthopaedic surgery residency program directors and orthopaedic residents. The results strongly suggested that there is substantial room for progress in laboratory-based surgical skills education of orthopaedic residents. Not surprisingly, there was variation in the availability of, size of, and access to surgical skills laboratories. Nearly one-quarter of all program director respondents indicated that they did not have a surgical skills laboratory at all, and over one-half indicated they did not have a curriculum for skills education. The majority of respondents reported that less than one hour per week of protected time was dedicated to a surgical skills curriculum. Many of the existing laboratories were very small.
Assessing the progress of individual residents as they learn surgical skills and providing formative feedback should be an important part of laboratory-based skills training. According to the survey responses, resident surgical skills were most commonly assessed by means of faculty evaluations. Assuming that this response often meant an evaluation at the end of a rotation, this does not represent a very objective or timely assessment of technical skills. Interestingly, 83% of all responding orthopaedic surgery residents felt that orthopaedic surgical skill improvement was not objectively measured at all. These responses indicate that developing objective measures of technical skill in orthopaedic surgical education is an unmet need of considerable importance.
It is interesting that 80% of responding orthopaedic surgery residency directors and 86% of residents felt that a laboratory-based surgical skills curriculum should be a part of residency training but that the percentage decreased significantly (to 33% and 31%, respectively) when they were asked whether similar methods should be utilized in the board certification and recertification process. The strong agreement among both program directors and residents concerning the need for a curriculum-based surgical skills curriculum should lend support to the future development and requirement of such programs. The biggest obstacle to progress in these areas was clear from the survey; program directors indicated that lack of available funding was the most substantial barrier to the development of a formal surgical skills program at the reporting institutions. It is possible that if skills training were a requirement for accreditation rather than an optional luxury, institutional financial support would be more readily available. It is also obvious that substantial energy should be applied to development of affordable, effective, and validated training modalities.
The results of the current survey can be compared with those of a similar survey of general surgery program directors that was published in 20072. The two surveys were separated by approximately five years. The skills laboratory space available and the time spent by general surgery residents in the skills laboratory at the time of the previous survey were similar to the values reported by orthopaedic program directors in the present survey. However, even five years previously, the general surgery program directors reported greater access to the laboratory, greater use of simulation, and greater interest in a standardized curriculum.
The low response rate is a limitation of the present study. Only eighty-six (46%) of the 185 program directors and 687 (15%) of the 4549 residents invited to participate responded. Unfortunately, such response rates are typical of this type of study. A survey of both orthopaedic program directors and residents regarding arthroscopic training in U.S. residency programs had response rates of 15.9% and 11.1%, respectively8. Another study in which orthopaedic program directors and residents were surveyed regarding their attitude toward resident work hours had response rates of 56% and 30%, respectively9. Another limitation of the present study is that some of the questions could have been interpreted in different ways by different respondents. For instance, some program directors who were asked, “Is there a curriculum for surgical skills training?” may have considered a resident anatomy course sufficient for a positive response and others may not have.
Despite the limitations, the present survey showed that there was strong agreement among both program directors and residents that surgical skills laboratories and simulation technology should be a more important and required portion of resident education and training. Both groups agree that a formal national curriculum should be established. At the present time, the most substantial barrier to the widespread adoption of surgical skills laboratories and formalized motor skills curricula is the lack of available funding. The results of this survey indicate that orthopaedic educators should find cost-effective solutions to improve surgical skills training, including objective assessment tools, for orthopaedic surgery residency curricula.
Sanfey
HA;
Dunnington
GL. Basic surgical skills testing for junior residents: current views of general surgery program directors. J Am Coll Surg.
2011 Mar;212(
3):406-12.[CrossRef]
Kapadia
MR;
DaRosa
DA;
MacRae
HM;
Dunnington
GL. Current assessment and future directions of surgical skills laboratories. J Surg Educ.
2007 Sep-Oct;64(
5):260-5.[CrossRef]
Accreditation Council for Graduate Medical Education. Surgery program requirements. .
Van Heest
A;
Putnam
M;
Agel
J;
Shanedling
J;
McPherson
S;
Schmitz
C. Assessment of technical skills of orthopaedic surgery residents performing open carpal tunnel release surgery. J Bone Joint Surg Am.
2009 Dec;91(
12):2811-7.[CrossRef]
Seymour
NE;
Gallagher
AG;
Roman
SA;
O’Brien
MK;
Bansal
VK;
Andersen
DK;
Satava
RM. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg.
2002 Oct;236(
4):458-63; .[CrossRef]
Howells
NR;
Auplish
S;
Hand
GC;
Gill
HS;
Carr
AJ;
Rees
JL. Retention of arthroscopic shoulder skills learned with use of a simulator. Demonstration of a learning curve and loss of performance level after a time delay. J Bone Joint Surg Am.
2009 May;91(
5):1207-13.[CrossRef]
Okrainec
A;
Soper
NJ;
Swanstrom
LL;
Fried
GM. Trends and results of the first 5 years of Fundamentals of Laparoscopic Surgery (FLS) certification testing. Surg Endosc.
2011 Apr;25(
4):1192-8. .[CrossRef]
Hall
MP;
Kaplan
KM;
Gorczynski
CT;
Zuckerman
JD;
Rosen
JE. Assessment of arthroscopic training in U.S. orthopedic surgery residency programs–a resident self-assessment. Bull NYU Hosp Jt Dis.
2010;68(
1):5-10.[PubMed]
Immerman
I;
Kubiak
EN;
Zuckerman
JD. Resident work-hour rules: a survey of residents’ and program directors’ opinions and attitudes. Am J Orthop (Belle Mead NJ).
2007 Dec;36(
12):E172-9; .
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. One or more of the authors, or his or her institution, has had a financial relationship, in the thirty-six months prior to submission of this work, with an entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. No author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.