Beginning in 1975, arthroplasty registries were introduced in several Scandinavian countries. Once the benefits of registries became apparent in Scandinavia, many other countries developed their own national registries. Each nation designed a registry to meet the needs of its own orthopaedic community, patient population, and circumstance. The aim of most national arthroplasty registries is to monitor implant survival so that poorly performing implants can be removed from the market and to improve the outcomes of arthroplasty. This information also allows surgeons to be guided to the most effective prostheses. A secondary aim is to encourage good practice by self-reflection and to improve the outcomes of arthroplasty surgery. Although units are often identified openly, a perceived need for anonymity dictates that data related to individual surgeons are not released to the public. This anonymity ensures that surgeons are not dissuaded from engaging with the national registry or from attempting challenging cases in which the outcomes are likely to be poorer.
Our aim is to encourage continual improvement in the quality of care provided to joint replacement patients in Scotland by the promotion of positive change in individual surgeons. The Scottish Arthroplasty Project (SAP) uses statistics to engage surgeons in the process of audit and the resultant improvements in patient care. When the Scottish Arthroplasty Project was first established in 1999, we took advice from our Scandinavian colleagues that the orthopaedic community should own and control the project and should take responsibility for the data and consequent change. The project is led by orthopaedic surgeons and reports to the Scottish Committee for Orthopaedics and Trauma (SCOT). The Scottish Arthroplasty Project Steering Committee (SAPSC) is composed of orthopaedic surgeons, an anesthesiologist, patient representatives, and community medicine representatives. The project is supported by the Chief Medical Officer for Scotland and is wholly funded by the Scottish Government through the budget allocated to the Information Services Division (ISD) (www.isdscotland.org), with no supplementary industry funding.
Since the inception of the Scottish Arthroplasty Project, our policy has been to provide high-quality data on the activities of individual surgeons and orthopaedic units by reporting complications resulting in readmission that can then be used at a local level to promote positive change. A generic Information Services Division dataset from which Scottish Arthroplasty Project extracts arthroplasty-relevant data is made available to the public with the publication of an annual report through our website (www.arthro.scot.nhs.uk). The first report, providing a comparison of Scottish data with international data, was published in 2002. Thereafter, data collection continued, and the first individual feedback for surgeons was issued in 2003.
We report on easily identifiable end points of public concern: death, dislocation, wound infection, revision arthroplasty, and venous thromboembolism (VTE). These are outcomes that our patients understand and identify with. The annual report includes national information about the number of hip and knee joint replacements being performed and also charts the complication rates of each unit.
The Scottish population of 5.2 million is served by twenty-four orthopaedic units. Data are collected by the ISD on every arthroplasty case that is undertaken, and all arthroplasty surgeons therefore contribute to the Scottish Arthroplasty Project without having direct involvement in the data collection process. As many as 300 orthopaedic consultants may contribute to the Scottish Arthroplasty Project in any one year because of surgeon retirements, locum tenens, and new appointments. Simple monitoring of activity and quality may influence clinical practice but can be insufficient to create meaningful change. With the support of the Scottish Committee for Orthopaedics and Trauma, the Scottish Arthroplasty Project introduced a feedback and review system to allow both individual surgeons and orthopaedic units to look at quality issues.
Data are routinely collected locally and consist of a composite of patient name, surgeon name, and procedure date. These data are then submitted electronically to the Scottish Arthroplasty Project by means of Scottish Morbidity Record forms (SMR01) that are completed at the time of hospital discharge and are part of the National Health Service (NHS) administrative dataset. Every three months, each surgeon is provided with a personalized report detailing every operation that he or she has done. This gives individual surgeons the opportunity to evaluate their own data. In addition, 2.5% of all notes are centrally audited annually to ensure the accuracy of local coding. The submitted data are then collated and analyzed by staff at the Information Services Division.
The Scottish Arthroplasty Project aims to improve the overall outcome for patients who are managed with arthroplasty by auditing surgeons rather than the medical devices that they implant. Although the Scottish Arthroplasty Project does not collect implant-specific data, it is able to perform indirect implant surveillance by identifying poorly performing implants used by individual surgeons or orthopaedic units. An audit of a surgeon's or unit's complications is undertaken when they trigger an outlier alert, and, if an implant is the common cause of the complications, then the Scottish Arthroplasty Project Steering Committee is alerted via the feedback and review system. The Scottish Arthroplasty Project does not collect patient-reported outcome measures (PROM) data. Although the Scottish Arthroplasty Project only collects limited data, it is able to demonstrate a measurable change in practice. Patients are identified by the Community Health Index (CHI) number, which is an individual national identification number that is used throughout NHS Scotland. This allows the Scottish Arthroplasty Project to match a patient's procedure to hospital admission anywhere in Scotland both before and after the procedure. The Scottish Arthroplasty Project therefore captures all readmissions, even if the patient is readmitted to another unit within Scotland. Patients do not formally consent for their data to be held by the ISD. No sensitive personal data relating to the patient are available to anyone other than the Scottish Arthroplasty Project analyst and the patient's surgeon. All patients are managed within the NHS, and the Scottish Arthroplasty Project data are collected as part of the patient's overall care.
From 2003 to 2009, Shewhart control chart methodology1 was used to present complication data and to identify variation. Orthopaedic units or individual consultants were case mix-adjusted by diagnosis only and were plotted in relation to Shewhart control limits, three standard deviations above and below the mean. If an orthopaedic unit or surgeon was outside this statistical limit, that unit or surgeon was identified as an outlier and was asked to perform a review of complications. In the control charts, five-year aggregated outcomes were necessary to achieve significance. Therefore, the data presented to outliers were historical. In 2010, the Scottish Arthroplasty Project started using cumulative sum of outcomes (CUSUM) methodology to allow us to identify unusual runs of complications, including poorly performing implants, in real time.
CUSUM is a sequential statistical analysis technique with an excellent visual impact when displayed graphically. It allows the rapid identification of increasing complication rates, of any etiology, among surgeons. It has been in use in the U.K. since as early as 1954 for industrial quality-control analysis2. This graphical tracking measure of adverse events has been used repeatedly for analyzing the learning curve in cardiothoracic surgery since the 1990s3 and has been shown to be a superior form of statistical analysis for identifying complications4. There are many forms of CUSUM, and the one used by the Scottish Arthroplasty Project is a likelihood-based scoring method that adjusts for the estimated risk of complication. The exact method of calculation of CUSUM may vary each time it is applied to a different clinical setting, depending on the availability and accuracy of case-mix control variables. If individual patient risk can be calculated preoperatively, then the drop or rise in CUSUM postoperatively will reflect this risk. The statistical modeling used by the Scottish Arthroplasty Project only has access to routine data, so our case mix adjustment is minimal and only takes account of age, sex, osteoarthritis, and rheumatoid arthritis. A surgeon therefore can be identified as an outlier as a result of other complex case mix factors alone. A detailed comprehensive case note review supporting a difficult case mix can form the basis of an acceptable response to the Scottish Arthroplasty Project Steering Committee. In this way, we believe that case mix is taken into account but does not become a statistical façade for true outliers to hide behind.
If a surgeon's complication rate is close to or below average, his or her CUSUM will remain close to zero. The point at which the surgeon is identified as an outlier is called the prediction limit. Setting the statistical triggers at an appropriate level allows us to balance the risk of false alerts and the risk of not detecting an unacceptable change in the complication rate. Because the prediction limit is arbitrarily set and can be changed simply by changing the statistical criteria, an outlier surgeon is not necessarily a poor surgeon and should not be labeled as such5. The Scottish Arthroplasty Project has chosen a control limit of two, which will result in approximately 10% of surgeons becoming outliers for at least one complication type in a year. This control limit allows detection of outliers from as few as four complications in quick succession while identifying a similar number of outliers as was the case with the methodology previously employed by the Scottish Arthroplasty Project. If the CUSUM rises to the prediction limit line, then the surgeon will be alerted to his or her outlier status and will be asked to review his or her complications and to complete and return an Action Plan to the Scottish Arthroplasty Project Steering Committee. The CUSUM is reset when the prediction limit is reached and a satisfactory response to the Scottish Arthroplasty Project Steering Committee is received from the surgeon involved. Repeated “alerts” therefore relate to different series of operations, ensuring that the review process is never repeated for the same case.
CUSUM identifies “outliers” quickly. The analysis does not rely on historical data and, because the alerts are contemporaneous, the surgeon is easily able to identify the reasons for being contacted. This speed advantage allows for the rapid identification of failing implants or poor practices and allows implants to be withdrawn or practices to be changed in a timely manner.
When CUSUM is displayed graphically, each blue dot represents a successful arthroplasty event by an individual consultant. The blue tracking line increases if there is a complication after the operation. A complication is indicated with a red diamond. The increase in CUSUM resulting from a complication is counteracted by many small decreases in CUSUM resulting from operations with no complications. As complications are rare events, they cause a large increase in CUSUM. The CUSUM is not allowed to fall below zero so that periods in which the complication rate was higher are not masked by previous periods when the complication rate was very low.
The purpose of reviewing outliers is to emphasize quality improvement and not to attribute blame. The aim of the process is to continue to encourage local review of clinical practice and data quality, both of which contribute to the continual improvement of patient care.
When outliers are identified, the orthopaedic unit and the surgeon are contacted and are asked to undertake a local review to investigate the reasons for their results and to report back to a Scottish Arthroplasty Project analyst (Fig. 6). The surgeon reviews his or her own complications and compiles an Action Plan, which is then cosigned by a responsible local colleague. Guidance is given as to the clinical governance process and what a good response entails. The Scottish Arthroplasty Project Steering Committee members then independently grade the Action Plan anonymously according to three domains (Table I). The grades are taken to a committee meeting, and a final score (Table II) and appropriate feedback are decided on in an open committee discussion. If the Action Plan is deemed unsatisfactory, then a resubmission is requested and any discrepancies must be addressed. This review process is administered by the Scottish Arthroplasty Project analysts throughout so that outliers are not identifiable by members of the Scottish Arthroplasty Project Steering Committee. The Scottish Arthroplasty Project continues to monitor the outlier's performance following a response, and the surgeon will be contacted again if complication rates remain unusually high. If it is appropriate, then the Scottish Arthroplasty Project Steering Committee feedback may positively encourage the continuation of a complex practice.
Local cosignatories ensure that the investigation that has taken place is genuine. The professional registration of the cosignatory is at risk if the clinical governance process is found to be fraudulent. Our probity responsibilities as stated in the General Medical Council (GMC) Good Medical Practice guide6 clearly demand honesty and integrity in the publishing of results, thus ensuring that “sign-off” is a serious matter requiring substantial deliberation. The Scottish Arthroplasty Project Steering Committee provides feedback to the cosignatory in relation to the Action Plan.
We recognized that the introduction of this system would not be an instant solution. The process the orthopaedic surgeons of Scotland and the Scottish Arthroplasty Project has gone through is a good example of the four stages of facing reality as described in the “journey to Jiseki”7 (Table III). Initially, surgeons denied that there was a problem and assumed that the data were incorrect. Next, surgeons accepted that the data were accurate but were ambivalent toward the findings. Finally, the surgeons accepted that the process was valuable and that the data reflected a true problem that could be rectified. There was then a commitment to the process and a value attached to the Scottish Arthroplasty Project findings.
As surgeons have become more aware of the feedback system, particularly with the introduction of CUSUM, their responses have become more rapid and more comprehensive. It has only been necessary to report one outlying surgeon to their regional chief executive for not participating in the review process, and this problem was resolved locally.
Within the Scottish orthopaedic community, there has been a general acceptance of the role of Scottish Arthroplasty Project as an independent clinical governance process. From surgeons’ feedback, we know that notification of an outlying position presents a good opportunity for self-review even if no obvious problems are identified. When local management has questioned individual practice, Scottish Arthroplasty Project data are made available to the surgeon to support the surgeon's practice. This type of data has also been valuable in appraisal processes that will feed into the future professional revalidation system. Data also can be useful to the surgeon in medical negligence cases. Although there initially were concerns about lack of engagement from the orthopaedic surgeons, our methodology has resulted in enthusiasm from the surgeons and 100% compliance. We have found that the process has nurtured innovation, education, and appropriate risk aversion. For the relationship between the Scottish Arthroplasty Project and the surgeon to be reciprocal, the Scottish Arthroplasty Project offers surgeons the opportunity to access more extensive Scottish Arthroplasty Project data for research and audit projects. This has led to a number of publications and presentations (www.arthro.scot.nhs.uk/publications.html).
We collect only limited data, but we have achieved a demonstrable positive change and an overall reduction in our complication rates. The process has been so successful that the anesthetists have joined us in reviewing data on anesthetic complications, and we are considering expanding our data collection and moving into new areas. Focusing on the specifics of particular implant choice or thromboprophylaxis at a national level can be used as a powerful tool to engage the surgical community but may struggle to achieve local change in practice. By using CUSUM analysis of individual surgeons’ unadjusted results, our primary goal of improving global patient care in the arthroplasty setting appears to have been achieved and is demonstrated by a fall in the overall complication rates. We appreciate that surgeons respond positively to timely and accurate data, and the promotion of individual change in practice has been more effective and immediate than imposing top-down reform.
Note: The authors thank the orthopaedic surgeons of Scotland for their participation in the Scottish Arthroplasty Project. They also thank Jan Kerssens and Santiago Nieva of the Information Services Division Quality Improvement Program. Finally, they thank the Information Services Division Publication Department for the production of images.