Competence as an orthopaedic surgeon requires knowledge, judgment, and technical skills. Although performing surgical procedures is a major portion of an orthopaedic surgeon's work, assessment tools to measure performance in specific technical skills have been lacking.
Adequate documentation of technical competence in surgery has come under increased scrutiny in recent years. New restrictions in work hours, with a shorter work week for residents, an emphasis on operating-room efficiency, and concerns about patient safety and medical errors may limit the ability of a teaching faculty to provide graded responsibility with adequate acquisition of skills during a five-year orthopaedic surgery residency program. Residency programs in the United States are required by the Accreditation Council for Graduate Medical Education to document competence in six areas: patient care, medical knowledge, professionalism, practice-based learning and improvement, systems-based practice, and interpersonal communication skills; however, documentation of competence in specific technical skills is also important.
In order to minimize rates of medical errors and adverse events, particularly during surgery, other surgical specialty training programs have begun to use simulation both to teach and to test technical skills when performing procedures1-3. Lentz et al.4 documented the use of inanimate and porcine models for surgical teaching and skills evaluation of obstetrics/gynecology residents. They used the Objective Structured Assessment of Technical Skills (OSATS), which assessed both global ratings and checklist evaluations. Significant improvement was documented for residents advancing from the postgraduate year (PGY)-1 level to the PGY-4 level for technical skills in performing laparoscopic salpingotomy and open oophorectomy. Using anesthetized pigs, Martin et al.5 assessed the construct validity of the assessment of basic surgical technical skills for reliability and validity in general surgery. A detailed task-specific checklist, a seven-item global rating scale, and a pass/fail judgment were used by board-certified general surgeons to assess each skill. This study showed that the OSATS can reliably and validly assess surgical skills.
The American Board of Surgery is now requiring general surgery residents to complete the Fundamentals of Laparoscopic Surgery curriculum (developed and validated by the Society of American Gastrointestinal and Endoscopic Surgeons) and to have successfully passed the standardized Fundamentals of Laparoscopic Surgery examination as a prerequisite for board certification6. The Fundamentals of Laparoscopic Surgery examination measures cognitive knowledge, case and problem-management skills, and manual dexterity7. Skills testing is growing quickly in other areas through the use of simulation8,9. To our knowledge, no reliable, valid measures of specific procedural skills critical for residents' success in practice have been developed in the field of orthopaedic surgery.
The present study was designed to examine an integrated approach to assess technical skills in orthopaedic surgery residents performing carpal tunnel release. The residents were assessed with use of several different measures, as outlined in Figure 1. First, residents completed a web-based knowledge test assessing surgical anatomy, preoperative evaluation of carpal tunnel syndrome, interpretation of the literature, surgical steps, surgical incision, dictation of an operative report (see Appendix), and complications. Next, they participated in an Objective Structured Assessment of Technical Skills (OSATS) with use of assessment measures that have been validated in the general surgery literature. Each resident performed a carpal tunnel release on a cadaver specimen. Performance was evaluated by two examiners (A.V.H., S.McP.) with a use of a detailed checklist score, a global rating scale, and a pass/fail assessment. In addition, the time needed for completion of the procedure was recorded. The purpose of the present study was to evaluate the validity and reliability of each testing measure in the integrated assessment of the competence of orthopaedic surgery residents in performing carpal tunnel release surgery. Our hypothesis was that performance on the knowledge test, detailed checklist, global rating scale, and pass/fail assessment would vary significantly among residents with different levels of training and would provide a consistent and reliable measure of technical competence in performing carpal tunnel release.
Following institutional review board approval, residents in the Orthopaedic Surgery Residency Program at the University of Minnesota participated in an upper-extremity motor skills examination. Twenty-eight residents consented and participated; four residents were unavailable for participation. With regard to the level of training at the time of the examination, there were two PGY-1 residents, eight PGY-2 residents, five PGY-3 residents, seven PGY-4 residents, and four PGY-5 residents, and two PGY-6 hand fellows. All residents were in good standing at the time of the examination.
Knowledge Test
Prior to the OSATS examination, all residents completed a 100-point knowledge test with use of a web-based computer assessment module. The module was created with use of WebCT/Vista ()10 on a University of Minnesota server and was administered with password-protected access with centralized automated scoring. The test content was written by a board-certified orthopaedic surgeon (A.V.H.) with subspecialty certification in hand surgery. The test underwent three iterations prior to administration to the residents. The first version was taken by four board-certified orthopaedic surgeons (including three of the authors [A.V.H., M.P., and S.McP.]) with subspecialty certification in hand surgery to evaluate the questions and to verify the content as appropriate for understanding and completing carpal tunnel release surgery. The second version was taken by three medical students for usability testing with written feedback on both technical aspects of the web-based learning format as well as question content. The third version was taken by a senior-level board-certified orthopaedic surgeon with subspecialty certification in hand surgery to verify that the content was appropriate for the skills necessary to understand and complete carpal tunnel surgery. The final version was administered to all orthopaedic surgery residents as part of the present study.
The knowledge test was scored on a 100-point scale with the following sections. The anatomy section (32 points) reviews pertinent surgical anatomy, including correct identification of labeled pictures of tendons, ligaments, arteries, and nerves in or near the carpal tunnel. The preoperative evaluation section (18 points) includes the indications for surgery, pertinent physical examination findings, electrodiagnostic testing interpretation, and patient examples used to decide whether carpal tunnel release is indicated. The interpretation of literature section (2 points) provides a link to a randomized study examining the surgical treatment of carpal tunnel syndrome and asks the resident to interpret the results as they apply to a patient example. The surgical steps section (16 points) requires the resident to identify correct patient positioning, topical landmarks, and layers of dissection. The surgical incision section (6 points) is an exercise in which the mouse is used to control the image of a scalpel on the computer screen and the resident is asked to indicate the location of the planned incision on a picture of the hand with use of the "scalpel" tool of the mouse. The dictated operative report section (10 points) asks the resident to use a microphone headset and to dictate an operative report for a carpal tunnel release with use of a standardized operative report template. The complications section (16 points) tests whether the resident recognizes a nerve or artery laceration and asks how it can be prevented.
At the time of the knowledge test, each resident was asked to self-report the number of carpal tunnel operations that he or she had observed prior to the OSATS examination as well as the number of carpal tunnel operations that he or she had performed prior to the OSATS.
OSATS Testing
The OSATS was given individually to all twenty-eight residents on a single day. After a brief orientation, each resident performed carpal tunnel surgery on a cadaver specimen in a mock operating room setting. Each resident was given a standard instrument tray and performed the surgery independently, with no assistants. Each resident was evaluated by two board-certified orthopaedic surgeons (A.V.H., M.P.) with subspecialty certification in hand surgery. The same two examiners independently evaluated all twenty-eight residents with separate rating forms for the detailed checklist (see Appendix), the global rating scale, and the pass/fail assessment. The time to completion was recorded for each resident from the time of incision to the time of the final suture.
The detailed checklist is a deconstructed list of all steps of a surgical procedure that quantitatively assesses whether or not each step has been completed correctly (with 1 point indicating that the step has been completed correctly and 0 points indicating that it has not been completed correctly). As shown in the Appendix, each detailed step of carpal tunnel release is included. Content was verified by two board-certified orthopaedic surgeons (A.V.H., M.P.) with subspecialty certification in hand surgery. Because the detailed checklist does not delineate whether any adverse steps or events had occurred, another section was added to document adverse events that may have occurred.
The second evaluation tool, the global rating scale, was validated by Reznick et al.11. The scale has six dimensions, each of which is related to some aspect of the operative procedure. Each dimension is graded on a 5-point scale with middle and extreme points anchored by explicit descriptors. Each 5-point scale is scored from 1 (poor performance) to 5 (good performance). Our modified global rating scale (see Appendix) used the same dimensions for "Respect for Tissue" and "Instrument ID and Handling" as did the scale used by Reznick et al.11. We combined the dimensions of "Motion and Flow" of the operation, and we added the dimensions of "Quality of Incision," "Quality of Suturing," and "Quality of Knots." These six dimensions were scored on a 1 to 5-point scale, for an overall possible total score of 30 points.
The third OSATS measure, the pass/fail assessment, was a subjective independent assessment by the examiner, asking the examiner to make a pass/fail judgment. For the nine residents who failed, adverse events were documented, including surgical release of the Guyon canal without release of the carpal tunnel, palmar arch injury, adverse handling of the soft tissues, and surgical release of the palmar aponeurosis without division of the transverse carpal ligament.
At the time of the motor skills evaluation, the total time for completion of the carpal tunnel surgery was recorded.
Statistical Methods
All scores from the computerized knowledge test were downloaded and entered into a study database. Data resulting from the OSATS and the residents' self-reported experience with the carpal tunnel release procedure were recorded on paper forms and were manually entered into the database.
To assess the internal consistency reliability of the OSATS measures, we calculated the Cronbach alpha coefficient for scores resulting from the detailed checklist and the global rating scale. To assess the inter-rater reliability of scores emanating from the same two measures, we used the interclass correlation coefficient. We explored the correlation between checklist scores, global rating scores, and the pass/fail decision with use of Spearman rho.
To assess the discriminant validity of scores from the knowledge test and the OSATS measures, we conducted one-way analysis of variance with postgraduate year of training (six levels) as the between-subjects factor. Using analysis of variance, we also explored the association between completion time, the number of previous carpal tunnel release procedures observed and performed, and training year. Finally, a review of the knowledge test data as compared with the OSATS pass/fail rates identified a score of =70 as a cutoff point. To determine whether knowledge predicted skill performance, we reviewed the pass/fail rates for residents who had a score of =70 on the knowledge test. We set the criterion level for significance at p < 0.05 for all tests.
Source of Funding
This project was funded internally by an education grant from the University of Minnesota for the purchase of laboratory materials.
The present study assessed the construct validity of an integrated approach to assessing competence in procedural skills. All assessment measurements that were used (the knowledge test, the global rating scale, the detailed checklist, and the pass/fail assessment) significantly and positively correlated with the level of orthopaedic training. The results suggest that both knowledge testing and cadaver skills testing discriminate between novice and accomplished residents. Failure on the knowledge test can predict failure on OSATS testing; however, the presence of knowledge does not necessarily predict success in technical skills (OSATS) testing.
The present study used tools that have been validated previously for general surgery procedures and demonstrated construct validity for use as assessment tools to evaluate residents performing carpal tunnel release. The global rating scale, detailed checklist, and pass/fail assessment correlated highly with each other as well as with the residents' level of training. The greatest gains were in two areas: knowledge gained between the PGY-1 and PGY-2 years and an increase in pass/fail decisions between the PGY-2 and PGY-3 years. All residents at or above the PGY-3 level competently performed the procedure, whereas nine out of ten residents at the PGY-2 level or below failed. In our residency program, the hand rotation occurs during the PGY-3 year, so the significant increase in pass rate above the PGY-3 level indicates substantial learning on the hand rotation. That is, all residents who completed the hand rotation passed the motor skills examination as the skills test was taken near the end of the academic year.
Although the cadaver testing assessments correlated with each other, four residents passed the knowledge test but failed the surgical skills examination. The knowledge test was not predictive for passing the surgical skills examination. However, the knowledge test plays an important role in the integrated approach of assessment as it assesses the cognitive domain as a prerequisite for performing the technical domain. The knowledge test can be used as a screening tool to assess whether the resident demonstrates an adequate knowledge base to be prepared for the surgical skills test, but it is not a substitute for motor skills testing. This finding challenges whether knowledge testing alone is sufficient for the assessment of surgical skills.
Any method that is used to assess technical skills must generate reliable and valid scores. The strength of the present study is that two of the OSATS measures that were used have been previously validated in the general surgery literature. When these tools were modified for use in carpal tunnel release, they were found to be similar12. Martin et al.5 described the use of three measures during OSATS testing: the global rating scale, the detailed checklist, and the pass/fail decision. The global rating scale is an assessment tool that has been used by multiple authors13-17. It is used for a qualitative assessment of surgical skill. The detailed checklist measures the completion of each deconstructed step in the procedure. The global rating scale measures the quality of technical skills common to most surgical procedures. Neither of these tools measures safety or assesses any adverse events that may occur during the procedure. Therefore, we added a checklist of possible adverse events that could guide the examiner in the pass/fail determination. Last, measurement of the time to complete the task measures efficiency, but this did not directly correlate with level of training and did not appear to identify the expert surgeon, as some of the younger surgeons simply completed the task quickly, but without expertise.
The strengths of the present study include not only the use of validated assessment tools but also the rigor that was used as in a high-stakes examination format. All residents were rated by the same two examiners. No verbal feedback was given to the residents, and a simulated operating room environment was used.
One weakness of the study is that the two examiners may have been biased in their assessment of the residents. The residents, and their level of training, were known to the examiners. Future work for validation of these assessment tools will need to include blinded examiners (i.e., examiners who are unfamiliar with each resident's previous work or year in training). Another weakness of the study is that only two PGY-1 residents were able to participate in the study. Furthermore, the study only examined carpal tunnel release, which is a relatively easy procedure to perform; more complex procedures may discriminate better, particularly at higher training levels. Last, although the OSATS measures (the global rating scale and the detailed checklist) have been previously validated in the general surgery literature, the carpal tunnel release knowledge test has not been previously validated.
Development of an Objective Structured Assessment of Technical Skills (OSATS) is an important and necessary step for the advancement of orthopaedic surgery resident education. Other surgical specialties have developed technical competence testing as a required part of resident education, such as the Fundamentals of Laparoscopic Surgery6,7. Fundamentals of Laparoscopic Surgery testing measures cognitive knowledge, case and problem-management skills, and manual dexterity as assessed with a timed skills test. The present study provides an example of a similar integrated approach to assess the cognitive knowledge and technical skills associated with the performance of carpal tunnel release surgery. This study also provides a framework on which other assessment tools can be validated to evaluate orthopaedic surgery residents in training.