Adjudication of outcomes is commonly conducted in large-scale, multicenter clinical trials1,2. The process typically involves the formation of an adjudication committee comprising physicians who are experts in their field of study1-4. The role of adjudication committee members is to provide a systematic, unbiased, and independent assessment of study outcomes using a set of predefined criteria developed prior to the initiation of the trial. The primary purpose of adjudication is to minimize bias, as end-point definitions are standardized a priori, and it reduces the variance that would exist between different site investigators1,3,5-10. While adjudication of outcomes is not mandatory in all clinical trials, it has become the “gold standard” in well-regarded clinical trials11. Furthermore, it is highly recommended for studies that include subjective outcomes that depend on a composite of diagnostic tests and when blinding is not possible2,7,9,10. When outcomes are subjective, adjudication reduces the likelihood that the study will provide inaccurate results due to misclassification of outcomes7. It is critical that the results of any research be as accurate as possible, since high-quality data are required to make evidence-based decisions and improve patient care7-9. As nearly all clinical outcomes have some inherent subjectivity, adjudication committees are an integral part of a well-designed clinical trial. Despite this, the use or reporting of adjudication committees remains scarce in the literature. Dechartres et al. found that only 33.4% of randomized controlled trials published in five high-impact-factor medical journals over two years (2004 and 2005) were reported as having an adjudication committee. This rate was even lower in trials that assessed an outcome with a high risk for misclassification12. Similar results were indentified in a search of eight high-impact-factor surgical journals from 2005 to 2009, with only three of forty-seven randomized controlled trials including an adjudication committee in their outcome assessment process13. These findings indicate a clear gap in data quality that needs to be addressed.
The assessment of fracture-healing is considered one of the most difficult diagnostic challenges in orthopaedics2,9,10,14,15. This process usually entails an evaluation of both radiographic and clinical criteria, which can be inconsistent. To further complicate the process, fracture union must be determined, despite the lack of a consistent definition as to what is considered healed and what is considered a nonunion2,9,10,14,15. The severity of infections and the need for a reoperation are examples of other patient-important outcomes that involve some degree of subjectivity in their assessment9,10. Such outcomes impact a patient’s quality of life, both socially and financially9,14,15, and incorporating the adjudication process in future orthopaedic clinical trials is likely to provide quality evidence that will help physicians make decisions regarding the best course of treatment for their patients.
The purpose of this article is to provide a detailed description of the adjudication process and how to efficiently incorporate it into an orthopaedic clinical trial. The topics discussed have been derived from our practice and the relevant literature and should provide a succinct guide for preparing for potential situations and making informed decisions. Because of the paucity of adjudication content in the orthopaedic literature, most of the reports discussed are from other fields of medicine, as the general concepts are the same.
Required Infrastructure
Prior to the initiation of any clinical trial, it is important to have the proper infrastructure in place. A project management team and a central methods center are required to facilitate the management of data and communication among the clinical sites participating in the trial, the trial’s sponsor and steering committee (if applicable), and the adjudication committee. It is essential for this research team to have adequate staffing and support and to be trained prior to the start of the study. A set of standard operating procedures should be developed to assist with the training process and to standardize all processes and procedures8-10. It is also important to ensure that a sufficient budget has been set aside for adjudication. Poor budgeting can result in reliance on shortcuts that will decrease the quality of the evidence produced16,17. A final recommendation prior to embarking on a clinical trial involving an adjudication committee is to consult an expert—i.e., to identify an individual with experience in adjudication, specifically in the field of study, who can provide input on some of the critical aspects of adjudication. Although an adjudication committee may be a necessary component of a well-designed trial, the reality is that, if it is ill-conceived, it will bear little fruit and will result in a regrettable use of valuable resources.
Adjudication Charter and Adjudicator Responsibilities
An adjudication charter should be developed to clearly outline the adjudication committee membership and their roles and responsibilities. It is important to make sure that the adjudication committee members are comfortable with their responsibilities and to request their input when the adjudication charter is being developed. The adjudication charter also describes which items will be adjudicated and provides the decision rules for standardizing the assessments. For orthopaedic trials, items that are commonly adjudicated include participant eligibility, radiographic characteristics of the fracture (e.g., assessment of the fracture gap or the quality of the surgery), fracture-healing, secondary procedures, complications, and major protocol deviations.
The adjudication charter should also indicate when outcomes will be assessed by the adjudicators7,18,19. For a large percentage of cardiovascular research and orthopaedic research with outcomes such as reoperations and fracture-related adverse events, adjudicators are often involved only in the confirmation of events that are reported by site investigators, whereas, in other orthopaedic trials, the primary outcome (often fracture-healing) is assessed at every visit by not only the clinical site investigator but also the adjudication committee. This method decreases the risk of false-negative and false-positive results, major pitfalls that can be addressed with adjudication. Despite its benefits, this method is applicable to only some outcomes and generally to studies with smaller sample sizes. When disagreements between the adjudication committee and site investigators arise, the adjudication committee’s decision should be final to minimize bias, although additional information should be collected when required11.
The timing of adjudication meetings should be clearly outlined18. Since the rate of enrollment is difficult to predict, rather than adjudicating at certain enrollment points, it is recommended that adjudication meetings be held at consistent intervals to ensure that important findings regarding the safety of the patients involved, among other things, are readily available for ethics and data safety and monitoring boards11. Adjudication at the completion of the study data collection is not sufficient. Although end-of-study adjudication may be less expensive, the ability to provide feedback to the site to avoid unnecessary visits and to identify potential sources of error in data collection is lost.
Another important item to consider is how much agreement is required to confirm the assessment of an outcome. In most trials, adjudication is performed on the principle of achieving complete consensus among the adjudicators; however, as committee sizes increase, this may be difficult and a majority consensus may be used20,21. This should be clearly outlined and reported in the adjudication charter.
Selection of the Committee
The composition of adjudication committees has varied. In previous orthopaedic trials, adjudication committees consisted of radiologists, orthopaedic surgeons, or both; however, it is important to remember that orthopaedic surgeons are responsible for making the final decisions about possible treatment options2,14,15,22,23. The adjudication committee usually comprises three or more individuals with both clinical and research expertise. The number of adjudicators required to successfully determine the outcomes varies depending on the level of agreement among adjudicators17,24. Of particular importance, an odd number of adjudication members is ideal for achieving consensus17. Walter et al. suggested that no more than three adjudicators are required when a high level of agreement is present24. The investigators in the SPRINT trial came to a similar conclusion, finding that reducing the number of adjudicators from six to four resulted in no change in the effect observed; however, when the committee was reduced to three members, the significance of the effect was lost17. Most adjudication panels in studies published in the literature have included at least three members. It should be noted that with each additional member the expected variability in the results is reduced, but there is an associated cost increase and increased difficulty in scheduling meetings or teleconferences. The SPRINT investigators found that the adjudication-related expenses would have decreased by 16% whenever the number of adjudication members was reduced by one. These reductions were coupled with minimal changes to the outcome assessments17.
The trial’s sponsor or steering committee may need to approve all members of the adjudication committee prior to their involvement with the study2,10. Each member should have no vested interest in the results of the study7. Despite the emphasis on having experts involved in the adjudication process, experts may have a dominant personality that could impact the reliability of the findings. For example, in the SPRINT trial, in which fracture-healing was adjudicated in a comparison of reamed and unreamed intramedullary nailing for tibial shaft fracture fixation, the original minority assessment offered by one dominant personality on a six-member committee resulted in the consensus decision >40% of the time17. Although the effect on the overall study findings was uncertain, it is important to consider the dynamic interaction among the adjudication committee members17.
In our experience, it has been helpful to assign one of the adjudication committee members as the chair of the committee, who has the responsibility of leading the consensus meetings and ensuring that the decision rules are followed and timelines are met. The chair should ensure that all adjudication committee members have the opportunity to express their thoughts and opinions not only during the development of the adjudication questions and end point definitions, but also during the consensus meetings when final consensus decisions on study outcomes are being reached.
Training Adjudication Members
After the adjudication committee members, study definitions, and trial case report forms have all been finalized, the adjudication committee members should be trained with use of a set of sample cases11,17,18. Generally, sample data that are identical to what will be presented to the adjudication committee members once the trial begins are provided to the adjudicators for assessment of the primary outcome. The adjudication process is then carried out until consensus is achieved, and it should involve multiple cases. This “pilot” is likely the most important step to calibrating the adjudication process. It allows familiarization with the trial protocol and procedures and helps to identify and resolve any issues that are encountered in the adjudication process2,7,9. It also establishes how complex events should be adjudicated and allows the sponsor or steering committee to assess the level of agreement among the adjudicators. The importance of training is something that is often overlooked and is a potential pitfall that can easily be corrected.
Data Collection for Adjudication
Once the trial begins, it is the responsibility of the central methods center to collect and process all adjudication materials (radiographic and clinical data) that they receive from the clinical sites. As technology continues to advance, electronic case report forms and electronic data capture systems are essential for timely input and management of data8. These systems allow the use of rigorous standards of procedure to ensure that the highest-quality data are collected from clinical sites1,25. Furthermore, any discrepancies can be dealt with promptly so that there are as few data queries as possible8. These systems also allow the development of an audit trail, which is essential in the event of an internal and external review. Finally, site visits and investigator meetings are an effective way of ensuring that adjudication materials are properly collected and minimizing source data queries1.
It is important to understand that adjudication committee members will only receive a snapshot of the information observed by site clinicians. Thus, it is necessary to ensure that the data presented to them are complete and without error. The data collected should include full clinical notes rather than portions of data that can be collected in case report forms, which may not provide adjudicators with sufficient information for proper adjudication. From an objectivity perspective, research staff must ensure that all identifiers are properly removed and that the adjudication materials are blinded before they are submitted to the adjudication committee members for review.
Adjudication of Outcomes
After the adjudication materials are available for review, each member of the adjudication committee independently reviews each case, reaches a decision on the study outcomes, and submits his or her response to the central methods center. The central methods center collates all of their responses and identifies any disagreements. The central methods center then schedules an adjudication committee meeting to discuss and resolve all disagreements via consensus. Consensus is preferred over voting as it provides an opportunity for open discussion among adjudication committee members, which leads to reduced subjectivity in the assessment and reliable results. This method ensures that all adjudicators develop similar decision-making processes, as one individual may have consistently missed something when reviewing the radiographic and clinical data had they not been made aware of it through this open discussion. This meeting should include all members of the adjudication committee and two individuals from the project management team who will record the final consensus decisions. Having two individuals recording the consensus findings is important for ensuring reliability. If the adjudication committee members believe that there was insufficient information to provide a final decision, it is the responsibility of the central methods center to follow up with the clinical site to request the required information. The final consensus decisions are then used in the statistical analyses for the trial1,7,8,10,15.
Web-Based Adjudication
Over the years, the adjudication process has moved from a paper-based system to a web-based approach. Using a web-based platform for adjudication saves time and is more efficient in terms of facilitating the coordination of adjudication materials between the clinical sites and the adjudication committee2,8,9. The Global Adjudicator system is one validated and regulatory-compliant system that we have used routinely to meet our needs of web-based adjudication26. It is both an image management and electronic data capture system with which clinical sites can upload adjudication materials and adjudication committee members can provide answers to study questions globally that will then be stored on the system, preventing the need to mail or fax study data to the central methods center9,10.
A potential pearl for future adjudication committees is the use of computer algorithms to identify potential events that should be reviewed by the adjudication committee1. Rather than relying on site investigators to identify events or on the adjudication committee to review data from each follow-up visit, an algorithm can be designed to conservatively identify potential events by screening data entered into an electronic data capture system. The parameters are selected from the patient data collected and are set at a conservative level, whereby a potential event identified would likely not be a true event. This reduces the number of missed events while subsequently reducing the number of visits adjudicated when an event has clearly not occurred. Despite this, an algorithm may not be applicable to all studies, depending on the outcome of interest, and it requires additional evidence to confirm its feasibility and effectiveness.