Randomized controlled trials (RCTs) are widely accepted as the most scientifically valid design for clinical research and offer the highest level of evidence-based medicine. Although drug trials can be blinded and medications can be easily administered with similar expertise to patients, a number of concerns apply to the conduct of surgical trials1. Rigorous orthopaedic clinical research is on the rise and the expert-based culture of orthopaedics is slowly changing to an evidence-based approach. The percentage of RCTs published in The Journal of Bone and Joint Surgery (American Volume) has increased from 4% to 21% and the percentage of case series has commensurately decreased from 81% to 48%2. The National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS) funded only eight trials in 2009 at an approximate annual cost of $28 million3. Trials go through a stepwise progression over a long period of four to six years from interest in a trial up to the dissemination of the knowledge gained through presentation and publication4.
Successful clinical trials must overcome multiple challenges, which include identifying a question of clinical relevance; having experienced and committed investigators to oversee the trial; developing sufficient infrastructures, personnel, and data management teams; navigating a rigorous internal review process; and finding sustainable sources of funding over long periods of time. The sample size must generate sufficient power for detection of differences between treatment groups. This necessitates that efficient and effective patient recruitment and retention strategies are developed5. Although rarely reported in most published articles, recruitment rates in surgical RCTs are thought generally to be <50% of eligible patients6 and some authors have even reported a recruitment rate of <3%7. The external validity or generalizability of the findings of such RCTs may be compromised severely by selective participation if participating subjects differ systematically from the target patient population8.
For example, at our institution, in an attempt to randomize 300 eligible patients with unstable distal radial fractures over a period of four years, only thirteen patients (4%) were eventually randomized. Similarly, we attempted to randomize patients with stress positive supination external rotation ankle fractures to operative or nonoperative management. None of the forty-five eligible patients consented to randomization. Our experience is not abnormal. Thus, there is a need for a review of reported reasons for nonentry of eligible patients into surgical trials. The present review article discusses the hidden challenges of RCTs in terms of cultural (surgeon and patient-related), methodological, and practical barriers to recruiting and retaining patients.
Orthopaedic Culture and Surgeon Preference
Orthopaedics is an expert-based field, but it is gradually changing to an evidence-based practice as endorsed by recent efforts by the American Academy of Orthopaedic Surgeons and specialty societies as well as by the increase in RCTs being published in orthopaedic journals. When compared with medical physicians, surgeons may be less tolerant of uncertainty, which affects their participation in RCTs9, and may dislike having open discussions involving uncertainty with their patients10. Previous negative experiences and perceived threat of litigation may make some surgeons reluctant to submit elements of their practice to evaluation and scrutiny and hence they do not become involved in RCTs. Surgeon preferences for particular treatment modalities and techniques may often not be based on robust existing evidence11. Additionally, surgeons may not be adept at multiple surgical techniques, which would render them unable to ethically participate and recruit patients in a randomized trial.
Equipoise and Consenting Methods
Although equipoise is at the foundation of conducting a clinical trial and the surgical community can believe that several treatments may be equal, individual surgeons may have strong preferences, particularly when patient factors are taken into account. However, more recently, this concept has been replaced with the term “uncertainty,” whereby the surgeon has an idea that a certain treatment is probably superior to the current standard but is uncertain whether he or she is absolutely correct. Discrepancy and/or incomplete disclosure of risk and benefits in consenting methods among surgeons and institutions involved in the trial can also contribute to low recruitment rates. Excessive flexibility in following consent procedures or inadequate informed consent poses another problem12. One potential solution is to standardize consent methods by videos and/or written protocols.
Advanced Statistical Training
The proportion of surgeons who have completed advanced training in biostatistics, epidemiology, and evaluative sciences is low4 and/or the training that they received is insufficient1. Investigators are expected to use and understand concepts as appropriate sample size estimates, type-I and II errors, the effect size, and the minimum clinical difference, regardless of significance13. Therefore, it becomes imperative for surgeons who are interested and involved in RCTs to partner with a PhD epidemiologist, research methodologist, or statistician before the trial is designed. Important outcome measures that allow the trialist to determine whether a study can be adequately powered may be lacking, and a pilot study to help determine the sample size necessary to detect a statistically and clinically important difference may be needed before embarking on a larger pivotal study14. In fact, the American Orthopaedic Association has established a Clinical Trials Curriculum as a promising resource to be potentially integrated into the formal residency training programs15.
Expert-Based Trials
Surgeons may be more proficient and comfortable with one surgical technique being evaluated in a trial compared with another. Therefore, a surgeon may be unlikely to participate for fear of not providing the best care for his or her patient. To circumvent this problem in larger trials, the use of surgeon and/or center expertise-based trials has been proposed as an alternative design as described in detail by Scholtes et al.16.
Loss of Follow-up and Increasing Retention
Different strategies can be used to improve follow-up and retention of patients. Trials in which ≥80% of patients were followed up are considered more robust17. It is routine that some attrition is planned, such that more patients are enrolled than are needed to complete the trial. It has been demonstrated that study coordinators need detailed demographic information such as age, sex, home address, distance to travel, job, travel method, and several contacts to determine if there are systematic reasons why a patient will be unlikely to be available for follow-up5. A strong personal relationship between participants and the research coordinator and the principal investigator increases the chances of participant return for follow-up8 and has been cited as a reason for agreeing to participate in trials18. Most coordinators will exclude patients who live too far away, are itinerant, or plan to move. Most orthopaedic journals request two-year follow-up for any reconstructive and many acute care procedures. This length of follow-up, which, in many cases, is after the patient is feeling well or has reached maximal improvement, makes patient return for evaluation more difficult.
Other strategies to increase patient retention and locate patients who are labeled as “lost to follow-up” include confirming patient contact information and details at each visit, creating a welcoming environment for interviews, scheduling appointment times around patient preferences, providing patients with reminders for upcoming visits, closely monitoring data for missed and overdue follow-up visits, minimizing waiting times, and encouraging patients to complete questionnaires while waiting. Other options include having study staff negotiate the reduction of the demands of study participation, continuing attempts to contact lost patients and all alternate contacts18, and aiming for a high recruitment rate early in the trial19.
Incentives to motivate patients and aid in recruitment can be grouped as either tangible or intangible. Self-motivation and the greater benefit to society are among the most common and powerful intangible incentives5. Other intangible incentives include continuity of the participant-clinician relationship and perception of a greater level of care than would otherwise occur with standard follow-up. Tangible incentives can have different forms; reimbursement for travel costs or time involved in participation is generally considered an acceptable and ethical practice. However, the cost incurred in providing these benefits cannot be underestimated and should not outweigh the benefit gained by having additional participants enrolled in the study. Although reimbursement of individuals is acceptable, it may lead to inclusion of patients with less financial security participating at a higher rate than those with greater security. Ideally, one would like any trial to create an atmosphere that does not require payment to ensure enrollment.
Unwillingness to Participate and Irreversibility of Surgery
Achieving the necessary sample size can be difficult when few patients are eligible and/or willing to participate. Many patients may not want to enroll because of the invasiveness and irreversibility associated with surgery. Unlike drug trials, surgical trials, particularly orthopaedic trials, may be quite invasive and, more importantly, irreversible. Thus, patients approach these trials with greater trepidation. Because of the magnitude of the intervention, even mild differences may cause patients to be reluctant to enter a trial. This reluctance is intensified when trials compare surgical with nonsurgical treatments. Patients may opt for the nonsurgical treatment, anticipating that failure of that method may later be treated surgically, therefore getting the best of both methods. Finding patients who will consent to participate in these trials is exceptionally difficult. The surgeon must provide all relevant information and advocate for the patient while at the same time staying true to the study design and the listed risks and benefits of all treatments. In such trials, a standardized script or video explaining the rationale of the study may eliminate variation or unintended bias introduced by the surgeon who is obtaining the consent.
Strength of Patient Preferences
One of the most important and prevalent barriers to surgical trials is patient preferences. With a strong shift to shared decision-making, surgeons are trained to review options, risks, and benefits with the patient. For patients who have a condition that can potentially be treated surgically, the consenting process is far more involved than it was even twenty years ago. Oftentimes, the same condition in individual patients will be treated differently if a careful explanation is provided. Patient personalities vary substantially, and, in particular, their risk analysis differs. Many patients wish to have surgery only if it is necessary or very clearly in their best interest. Others may wish to take a chance to get the best possible result. In the senior author’s (P.T.) experience, when given full information about surgical and nonsurgical options, almost every patient wishes to make a choice rather than be randomized. Thus, it is often necessary to include an observational arm in these trials or to shift to a surgeon-based trial, as was reviewed earlier in this article.
Similarly, treatments that are distinctly different may generate strong patient preference based on internal beliefs, rather than on data. In the senior author’s (P.T.) attempts to recruit patients into an ongoing distal radial study, when patients were shown pictures of an external fixator and the internal plating hardware, many of them chose on the basis of “not wanting something sticking out of the skin,” or, alternatively, “not wanting an incision to the bone.” A similar example in our experience was from an ongoing study comparing posterolateral plating (associated with peroneal tendon irritation) with lateral plating (associated with local sensitivity) of ankle fractures. The recruitment challenge in this study was that patients had strong preferences against potential long-term complications and hence voluntarily selected one of these two treatment choices. If properly consented, the patient may wish to choose his or her treatment. Once a patient has such a preference, for whatever personal reason he or she may have, attempting to convince the patient to be randomized is not possible. Persistent attempts would likely deteriorate the surgeon-patient relationship as the patient may perceive that the surgeon is more interested in his or her trial than in the patient’s best interest.
Patients might also sometimes lack the understanding of the concept of equipoise20 or the fact that surgeons sometimes do not have clear-cut answers about surgical indication or even what the best treatment is. Hence, patients need to be empowered to help in the decision-making process and surgeons need to clarify the process of evidence-based medicine4. A feasibility study of patient willingness to participate in surgical and oncology trials showed a low level of willingness due to a stated dislike for randomization and a desire for autonomous selection of the intervention21. Patient preferences may not be based on the existing scientific evidence11 but may be based on the desire for the most novel treatment or on other commercial advertisements. The strength of patient preference also partly depends on the comparison of interest. Possibilities include a new procedure compared with a sham procedure, similar but distinguishable procedures, dramatically different procedures, or a surgical treatment compared with a nonsurgical treatment. Research to evaluate surgical innovations in pediatric patients is perhaps the most difficult and susceptible to preferences that preclude randomization11.
Language Barriers
Other barriers to patient participation in research include language barriers that lead to difficulty in obtaining consent and follow-up questionnaires. Such difficulties can be overcome if there is sufficient funding and if it is feasible to conduct the study in multiple languages. However, the challenges involved in accurately translating questionnaires or accessing interpreters should not be underestimated5.
Demographic Characteristics and Medical Conditions
In general, it is reported that patients who refused to participate generally tended to be older, single, either less well-educated or very highly educated, and more ill and to have comorbidities that interfere with activities of daily living22.
Long-Term Follow-up
To mitigate the need for long-term follow-up, outcome measures that necessitate shorter follow-up or earlier surrogate markers as the final end points should be used whenever possible. For example, the treatment of asymptomatic hip dysplasia in pediatric patients aims to prevent or delay the onset of hip arthritis in middle age. A surrogate or intermediate outcome such as the radiographic assessment of the appearance of the hip and shape of the femoral head in adolescence is related to the long-term risk of arthritis23.
Inclusion and Exclusion Criteria
Inadequate sample size and inadequate ability to recruit can be overcome by the participation of multiple investigators at multiple centers. Investigators can relax the eligibility criteria and narrow down the exclusion criteria judiciously, improve the consent procedure, or prolong the duration of the trial4. The use of outcomes that can be compared with the pre-intervention time frame allows for a broader patient group as the resultant change over time can be evaluated rather than just the difference in the outcome measure at a particular time.
Estimation of Sample Size
The researcher must also consider the required sample size that is influenced by the presumed effect size produced by the intervention. In general, one seeks to have an effect size of 0.8 or a relative risk reduction of >50%. Additionally, the clinically relevant difference in many scoring systems is often available from prior related research when available. When point estimates are not available, then an initial pilot study should be carried out to evaluate the ability of the study to determine a difference in outcome between study groups and whether enrollment numbers can be obtained.
Selection of a Representative Sample and an Appropriate Site
It is essential to consider which population and research setting might be appropriate to answer a particular research question when planning a study. For example, it is important to study hip fractures in community hospitals instead of level-I trauma centers as the prevalence is greater in local hospitals. Csimma and Swiontkowski24 reported on the importance of site selection from the BESTT (BMP-2 Evaluation in Surgery for Tibial Trauma) trial. Among the sixty centers in the study, >80% of the sample size was recruited across only twenty-six centers24. Contrary to intuitive approaches to site selection, having opinion leaders, experience with previous randomized trials, and publication records was not at all predictive of recruitment ability. The key factor for site recruitment was the motivation of the site principal investigator, often a junior faculty member or an individual with sufficient protected time and interest in participation24. Another method that has been shown to help in selecting sites is a sham enrollment period. In this method, each potential site fills out a screening form for all patients who would be eligible for the trial and asks the patients if they would have participated if such a trial were to exist. True enrollment will be far less, but this method will screen out ineligible patients far better than Current Procedural Terminology (CPT) coding. Rather than looking only at the condition or surgery, sham enrollment will seek out other issues such as language barriers, location, likelihood of follow-up, and other factors in the inclusion and exclusion criteria.
“Surgical” Equivalent to the Food and Drug Administration
Directives of the European Union and the U.S. Food and Drug Administration (FDA) have been developed to evaluate drugs. In surgery, no true parallel exists that requires high-quality evidence as a prerequisite for full adoption of certain implants or surgical procedures. Establishing nationwide controlling bodies that govern and control the testing of surgical implants and procedures can make outcomes more predictable and reassures patients about the increased safety and scrutiny of novel devices and procedures. This type of evaluation through assessment bodies has begun to appear in Australia and New Zealand25.
Funding
In contrast to pharmaceutical and drug treatments, there is less incentive for commercial entities to fund randomized surgical trials. There is a need to obtain national funding for surgical trials and to explicitly recognize the challenges in randomized surgical trials. Morshed and Bhandari noted that funding for clinical trials has been extremely limited, with a range of up to $100,000 across smaller orthopaedic societies and rarely greater than a few million dollars across national funders such as the National Institutes of Health and the Canadian Institutes of Health26. Other reports in the literature have simply concluded that the reality of orthopaedic research funding is that it is currently not feasible to run multiple, large, appropriately funded trials with peer-reviewed funding alone and that strong collaborations among industry partners, other researchers, and other surgeons and/or physicians are paramount.
Gatekeepers
During the recruitment process, it is preferable if the research personnel are directly responsible for identifying and screening potential participants to reduce the role that any gatekeepers play27. The largest potential barrier is that some insurance carriers will not support care given to patients who are in a surgical trial, particularly if one of the arms is not the standard of care.
Experience with a Randomized Trial for Treatment of Unstable Distal Radial Fractures
As an example of the challenges in enrolling patients in a surgical randomized trial, we present our experience with a randomized trial of unstable distal radial fractures. Over a four-year period at our institution, we evaluated all unstable distal radial fractures for possible randomization into treatment with either external fixation or volar plating. Three hundred (53%) of the 568 patients who were screened met radiographic criteria. Of these 300 patients, only thirty-nine (13%) were entered into the study. Of the 300 patients eligible for randomization, only thirteen (4%) were willing to be randomized and twenty-six (9%) chose the management that they wanted, entering the observational arm. The reasons why the remaining 261 patients were excluded are shown in Figure 1.
Although there is no consensus as to what would constitute an acceptable recruitment threshold of eligible patients, such a figure would probably vary for individual trials and would depend on sample size and difference in magnitude of measured effects between the two assessed treatment modalities. In this article, we have attempted to present the most common and prevalent reasons for the failure to recruit and retain patients in RCTs. It is also obvious that the order of importance and prevalence of these challenges depend on what the study question is, the institution or country where the trial is being conducted, and the patient population who is being studied. Our review also focused on factors that can be controlled by the surgeons before the study is commenced and is intended to avoid the most common pitfalls in recruitment and methodology. For example, easy-to-read consents have been associated with less anxiety and higher satisfaction rates. The data from our distal radial experience strengthen the need to develop other mechanisms to compare outcomes such as expert-based trials and raise the question of why other patient populations agree to consent. The consent process itself may require revisiting.
Finally, in the current environment of increased accountability, orthopaedic surgeons will likely in the near future be called on to justify their treatment decisions. This justification can be achieved by demonstrating sound clinical evidence mainly from clinical trials in support of the treatment selected4. Although orthopaedic trials have historically been single-surgeon or single-center, small, underpowered studies, the recent society and journal recognition of the importance of evidence-based orthopaedic practice will support the new generation of multicenter clinical trials. Above all, it is important that researchers ensure respect, confidentiality, and privacy for participants at all times, which remain the foundation of recruitment and retention of patients in any study. Clinical trials in the U.S. are conducted according to Good Clinical Practice guidelines provided by the FDA, which ensures the protection of human research subjects, and the integrity of data collected in clinical trials and the familiarity with these guidelines are important for investigators in any RCT4.