Abstract
Background: The Consolidated Standards of Reporting Trials statement
for the reporting of randomized controlled trials has been limited by its
applicability to surgical trials. In response, a Checklist to Evaluate a
Report of a Nonpharmacological Trial was recently developed by the
Consolidated Standards of Reporting Trials group to address reporting issues
in surgical trials. We aimed (1) to apply the checklist for nonpharmacological
therapies to orthopaedic randomized controlled trials across multiple journals
from 2004 through 2005, and (2) to survey authors when methodological
safeguards itemized in the checklist were not reported to determine whether
they actually had been performed. We hypothesized that lack of reporting of a
methodological safeguard did not necessarily mean it had not been
conducted.
Methods: We searched for relevant orthopaedic randomized controlled
trials across eight journals in the period from January 2004 through December
2005. We applied the Checklist to Evaluate a Report of a Nonpharmacological
Trial to all eligible studies. We contacted authors to determine what
methodological safeguards were actually used, especially when details remained
unclear from the publication.
Results: We included eighty-seven randomized controlled trials from
eighty-five scientific reports. In assessing the randomized controlled trials
with the checklist for nonpharmacological therapies, seventy-three studies
(84%) had unclear reporting of treatment allocation concealment. Only
seventeen studies (20%) mentioned surgeon skill or experience. The blinding of
patients, ward staff, rehabilitation staff, clinical outcome assessors, and
nonclinical outcome assessors was unclear in forty-eight (55%), sixty-three
(72%), sixty-four (74%), forty (46%), and thirty-three studies (38%),
respectively. Authors from forty-three randomized controlled trials responded
to our survey. The results of the survey showed that 41% (95% confidence
interval, 25% to 58%) of the trials had adequate allocation concealment when
this had been unclear from the report. Although the surgical experience of the
investigators was rarely reported, most authors (70%) acknowledged that they
had defined "surgical expertise criteria" such as minimum case
criteria, specialized training, and clinical performance. The survey also
showed that 28% to 40% of the trials had blinding of relevant groups despite
the fact that the reporting of such blinding had been unclear in the
publications.
Conclusions: The quality of reporting in the orthopaedic literature
was highly variable. Readers should not assume that bias-reducing safeguards
that were not reported in a randomized controlled trial did not occur. Our
study reinforces the need for the consistent use of a tool like the Checklist
to Evaluate a Report of a Nonpharmacological Trial to assess the methodology
of surgical trials.
When making treatment recommendations, healthcare practitioners and
practice-guideline developers look to evidence that allows them to make causal
inferences about the effects of treatments. The most secure source for such
evidence can be found in properly conducted randomized controlled trials
(RCTs), i.e., level-I evidence. However, the strength of this evidence hinges
on the methodological rigor with which the study was conducted. As an example,
Altman et al. reported that a lack of allocation concealment (i.e., the
assurance that investigators in a randomized trial cannot determine beforehand
the treatment allocation of the next patient to be enrolled in their study)
leads to an overestimation of treatment
effects1.
The Consolidated Standards of Reporting Trials (CONSORT)
statement2 was
developed in the 1990s in an effort to improve the quality of reporting of
published randomized trials in the medical literature. The CONSORT statement
has been supported by a growing number of health-care journals and editorial
groups since its introduction, but it is not optimized for use in evaluating
surgical trials3.
Recently, CONSORT developers recognized the limitations of the checklist in
the evaluation of surgical interventions.
Surgical trials present unique methodological issues. It is often
impossible to perform sham interventions or to blind patients and care
providers4,5.
The skill of the care providers can play a crucial role in the success of the
treatment6. To
evaluate these types of studies, a Checklist to Evaluate a Report of a
Nonpharmacological Trial (CLEAR NPT) was recently developed by the CONSORT
development
team7.
Given the introduction and endorsement of the CLEAR NPT checklist for
surgery, we aimed (1) to apply the checklist to published orthopaedic
randomized trials across multiple journals from 2004 through 2005, and (2) to
survey the authors of these randomized trials when items in the checklist were
not reported to determine whether they actually had been conducted. In our
secondary question, we hoped to determine whether "not reported"
was consistent with "not conducted." We hypothesized that lack of
reporting of a methodological safeguard on the CLEAR NPT checklist did not
necessarily mean it had not been conducted.
We conducted an observational study of published trials in the orthopaedic
literature and a subsequent cross-sectional survey of authors to evaluate the
quality of reporting. This study was approved by our university ethics
board.
Eligibility Criteria
To qualify for inclusion in this investigation, a study had to meet the
following criteria: (1) It had to be a nonpharmaceutical randomized trial
dealing with an intervention for an orthopaedic problem performed on patients
in an operating room, and (2) it had to have been published between January 1,
2004, and December 31, 2005, in the English language. Studies were excluded if
they involved physiotherapy, manipulation, or acupuncture. Also excluded were
animal, cadaver, and basic-science studies, as well as papers summarizing the
results of a previously published study.
Study Identification
We searched for relevant trials across eight journals (four orthopaedic and
four general medicine publications). These journals included The Journal
of Bone and Joint Surgery (American Volume), The Journal of Bone and
Joint Surgery (British Volume), Clinical Orthopaedics and Related
Research, Acta Orthopaedica, British Medical Journal, Lancet, the
Journal of the American Medical Association, and the New England
Journal of Medicine, and they were searched for relevant studies
published from January 2004 through December 2005. One investigator conducted
an independent hand search of the issues of the eight journals published in
2004 and 2005. Any potentially relevant studies were retrieved for further
review. Another hand search was independently conducted by three investigators
who divided the eight journals among themselves. In this search, the scope was
limited to all orthopaedic-related nonpharmaceutical randomized trials, as
identified by review of the Materials and Methods section of each study.
To further ensure that all randomized trials published during the study
period were identified, a professional librarian performed searches for
orthopaedic-related randomized trials in all eight journals using both MEDLINE
and EMBASE databases. The final potentially eligible list of trials from the
four independent reviewers and a librarian were combined and reviewed for
final inclusion in the study. Two investigators independently evaluated the
abstracts of the combined search results for eligibility. Discrepancies were
resolved by a consensus of the two reviewers. Consensus was achieved in all
cases—those that were included and those that were excluded.
Data Abstraction
The details abstracted from each study included the following: journal
name, category of trial, region of body, number of centers, and funding
received.
The CLEAR NPT Checklist
A series of five papers were scored by both investigators using the CLEAR
NPT checklist. The scoring of this test series was discussed, and one
investigator subsequently reviewed all eligible studies for quality of
reporting. The accuracy of the data abstraction was confirmed by a second
investigator. Consensus was achieved on all papers. The papers were reviewed
in random order as dictated by a computer-generated sequence of numbers. The
checklist7 was
utilized to score the quality of reporting (see Appendix). Briefly, this
ten-item checklist contains questions designed to assess the reporting in
nonpharmacological trials in areas crucial to internal validity, such as
standardization, care-provider skill, blinding, compliance, cointerventions,
and subject attrition. These questions are typically answered as yes, no, or
unclear. While we adhered to the CLEAR NPT scoring guidelines as described by
the authors, we also extended the scoring guidelines in attempts to achieve as
much objectivity and reliability as possible.
In scoring the papers, we found it necessary to expand two questions found
in the checklist. The question about outcome assessor blinding was made into
two questions with regard to the blinding of the clinical outcome assessor and
the nonclinical outcome assessor. The question on caregiver blinding was
expanded into three questions with regard to the blinding of the surgeon, ward
staff, and rehabilitation staff.
Author Survey to Resolve Uncertainties About Study Methodology
We developed a twenty-six-item questionnaire to be completed by the study
authors to clarify the methodological issues probed by the CLEAR NPT (see
Appendix). The questions focused on the assessment of investigator expertise
and clarification of the methods of randomization, concealment, and blinding.
An online version of the questionnaire was created with use of the
SurveyMonkey online service
().
The answers provided by the authors were used to provide details that were not
reported or were unclear in the published study.
Questionnaires were e-mailed to the corresponding author of each included
study. Our e-mails provided a description of our survey, as well as an
Internet link to the online version of the questionnaire. If authors did not
respond within three weeks, we sent follow-up e-mails at three and six weeks.
Approximately eight weeks after our first e-mail, we attempted to make
telephone contact with nonresponders. If we were unable to confirm receipt of
our e-mail messages during a telephone call, we faxed the authors a copy of
our questionnaire with an accompanying cover letter explaining the study. We
were able to obtain a Japanese translation of our cover letter to fax to
authors located in Japan.
Data Analysis
We summarized data descriptively. We used proportions with 95% confidence
intervals to summarize categorical variables, and we used means and standard
deviations to summarize continuous variables. For each aspect of
methodological reporting, we determined the proportion of randomized
controlled trials for which authors directly informed us that they had used
the methodological safeguard, despite not explicitly reporting this in their
study. Exact 95% confidence intervals were calculated for these
proportions.
Study Identification
Areview of the articles published from January 2004 through December 2005
revealed 613 potentially relevant reports. Application of the eligibility
criteria eliminated 528 studies (Fig.
1). Eighty-five reports (13.9%) met the criteria as randomized
trials concerning orthopaedic procedures that took place in the operating
room. Agreement between reviewers with regard to the eligibility of the
studies was moderate (? = 0.60). Although the inclusion of orthopaedic
trials seemed self-explanatory, our criteria of including only surgical
interventions performed in an operating room led to some discrepancies among
reviewers. Discrepancies were resolved by discussion and consensus.
Characteristics of the Randomized Trials
Two papers contained data from a total of four different randomized trials.
Thus, we included eighty-seven randomized trials from eighty-five scientific
reports. Seventy-seven authors were contacted in total. Four of them had
authored two reports each, and two had authored three randomized controlled
trial reports. Thirty randomized trials (34%) were found in the American
Volume of The Journal of Bone and Joint Surgery. No eligible studies
were found in either the New England Journal of Medicine or the
Journal of the American Medical Association. A total of 9844 patients
were included, with sample sizes ranging from eighteen to 552 participants.
The average sample size (and standard deviation) across studies was 113
± 102 participants. Sixty-seven studies (77%) were single center,
forty-one studies (47%) focused on the treatment of degenerative disease, and
the most studies (63%) involved problems affecting the hip or knee. The
funding status was unclear in thirty-nine studies (45%)
(Table I).
Quality of Reporting Assessed with the CLEAR NPT
The quality of reporting was assessed with the CLEAR NPT
(Table II). Forty-one studies
(47%) were unclear in describing the method of generating an allocation
sequence. Of the seventy-three studies (84%) with unclear allocation
concealment, thirty-three (45%) used envelopes as a means of allocation
concealment, but the studies were inadequate with regard to the description of
the envelopes. Only seventeen studies (20%) made any mention of surgeon skill
or experience. Of those that did, most made simple statements of skill or
training, leaving only four studies (5%) that provided objective measures of
the skill of the surgeon (Table
III). Blinding of the participants, ward staff, rehabilitation
staff, clinical outcome assessors, and nonclinical outcome assessors was found
to be unclear in forty-eight (55%), sixty-three (72%), sixty-four (74%), forty
(46%), and thirty-three studies (38%), respectively. Forty-five studies (52%)
made use of specific methods to avoid assessor bias. Due to the nature of our
eligibility criteria, surgeon blinding was judged to be impossible in the vast
majority of reports (eighty-six; 99%)
(Table IV). Intention-to-treat
analysis was judged to be used in thirty-eight studies (44%).
Although not technically part of the CLEAR NPT, data analyst blinding and
whether a sample size calculation was done a priori were also assessed. Three
studies (3%) had data analyst blinding, while eighty-four (97%) were unclear.
In regard to sample size calculation, twenty-eight studies (32%) performed a
priori sample size calculations and one (1%) did not, while fifty-eight (67%)
were unclear.
What Authors Did Not Tell Readers
We received responses regarding forty-three (49%) of the eighty-seven
randomized controlled trials. Of seventy-seven authors, thirty-nine (51%)
completed our questionnaire and six (8%) declined participation, leaving
thirty-two (42%) who did not respond at all.
Table V presents the
proportion of studies that had used a methodological safeguard, according to
the authors who completed the survey, when the report itself had been unclear
or ambiguous regarding that safeguard. Often, what the authors reported in our
survey differed from what they had reported in the primary publication
(Table VI). When concealment of
allocation was unclear in the published report, almost half (41%) of the
authors responded that they had in fact used sufficient methods to conceal the
randomization. Although most of the authors did not report the experience
(skill) of the surgeons in their reports, 70% of them acknowledged that they
had set objective measures such as minimum case criteria and/or comparison
with good clinical outcomes (Table
III). In 28% to 40% of the studies, the authors specified that
they had blinded relevant groups, despite the fact that they had not
explicitly stated so in their articles. The majority of the authors used
intention-to-treat analysis when it was unclear whether this was conducted in
their studies. Data analysts had been blinded in seventeen studies (40%; 95%
confidence interval, 22% to 53%), and a priori sample size calculations had
been done in fourteen studies (67%; 95% confidence interval, 33% to 73%) when
the randomized controlled trial reports had been unclear with regard to these
methodological parameters.
Principal Findings
In a review of eighty-seven randomized trials evaluating orthopaedic
surgical therapies, we found that (1) the quality of reporting as measured by
the CLEAR NPT was highly variable, (2) readers should not assume that
investigators who do not describe concealment of randomization, blinding
outcomes, or investigator skill in their reports did not actually use these
safeguards to protect against bias (i.e., readers should not assume that
"not reported" means "not performed"), and (3) the
CLEAR NPT, a surgery-specific checklist, is a useful measure of surgical
reporting.
The randomized controlled trial, a level-I study, can have a powerful and
immediate impact on patient care. Randomization provides one single and
uniformly important advantage over all other methodologies—it balances
known and unknown prognostic factors between treatment groups. No degree of
statistical manipulation in a nonrandomized study of two alternative
treatments can correct an imbalance of patient prognostic factors. However,
inadequate reporting of a randomized trial may leave readers unclear about the
so-called validity of its methodology and thus may call into question the
applicability of the results to patient care.
Strengths and Limitations
Our study has a number of strengths. Multiple investigators, including a
librarian, were independently involved in the literature search. We achieved
acceptable agreement in our eligibility decisions. We also used a
surgery-specific checklist (CLEAR NPT) developed by the CONSORT development
group. We were comprehensive in our attempts to contact authors. A Japanese
cover letter was created for the benefit of the authors located in Japan.
Lack of clarity with regard to the reporting of blinding in the randomized
controlled trial literature was reported by Montori et al., who found eight
combinations of blinded groups among eighty-three randomized controlled trials
reported as
"double-blind."8
The problem with the use of conventional terms such as "double
blinding" was further demonstrated by Devereaux et al., who reported
that physicians and textbooks varied greatly in their interpretations and
definitions of single, double, and triple
blinding9. For this
reason, two items on the checklist were divided into multiple questions. The
question on blinding of the care provider was subdivided into three questions
concerning blinding of the surgeon, rehabilitation staff, and ward staff. The
question on blinding of the outcome assessor was subdivided into blinding of
the clinical outcome assessor and blinding of the nonclinical outcome
assessor.
Although the CLEAR NPT checklist has been designed as an adjunct to the
CONSORT statement for surgical therapies, its reliability has yet to be
assessed. To improve the transparency and relative objectivity of the
questions, we developed scoring guidelines to supplement those provided by the
original CLEAR NPT paper (see Appendix). As an example, we created objective
guidelines to determine whether the number of patients who dropped out and
those lost to follow-up were the same between groups.
Although several attempts had been made to contact the authors, our survey
response rate was just over 50%. It is plausible that authors simply did not
understand the questions being posed and thus thought their study met all
methodological standards. Alternatively, authors may have felt compelled to
provide "socially desirable answers" (i.e., that all
methodological safeguards were used in their randomized controlled trial).
However, this concern is mitigated by the finding that authors acknowledged
many methodological shortcomings with their studies.
Review of Relevant Literature
The CONSORT statement was developed to address the issue of quality of
reporting in randomized controlled
trials2. While the
use of the CONSORT statement has been associated with improvements in the
quality of reports of randomized controlled
trials10,11,
it is a tool that is not optimized for nonpharmacological trials, which
present their own unique methodological
issues4,5.
Foley et al., in a review of eighty-one randomized controlled trials from the
physiotherapy literature, found that the methodological quality for
pharmacological interventions was substantially higher than that for
nonpharmacological
interventions12. A
similar result was found in a review of the literature evaluating treatments
for hip and knee
osteoarthritis4. The
CLEAR NPT was created with use of the Delphi consensus method in 2005 to
assess the reporting of nonpharmacological
studies7. Our
findings support those of Jacquier et al., who reviewed a total of 158
nonpharmacological trials and reported a low quality of reporting as assessed
by the CLEAR
NPT13.
Devereaux et al. conducted a survey of 105 authors of published randomized
controlled trials to identify whether those who failed to report concealment
of randomization or blinding actually used these safeguards in their
studies14. They
reported that "readers should not assume that biasreducing procedures
not reported in an RCT did not occur." The authors in the full-text
publications failed to report the presence or absence of concealment of
randomization in 55% of the trials and the blinding of outcome assessors in
83%. When the authors were contacted directly, they frequently reported that
they had used concealment of randomization (96%; 95% confidence interval, 87%
to 100%) and blinding of outcome assessors (79%; 95% confidence interval, 69%
to 87%), despite not having reported the use of these methodological
safeguards in their
studies14. We also
found high variability in the reporting of concealment and blinding status in
surgical trials (with unclear reporting in 84% and 46%, respectively).
However, when the authors were asked, fewer reported that they had actually
used these safeguards (41% and 35%, respectively).
Our study demonstrated that the proportion of orthopaedic randomized trials
that failed to meet the criteria on a surgery-specific checklist ranged from
1% to 99%. Information was often inadequately reported, leaving assessments
about the study methodology unclear in up to 84% of the articles. The quality
of reporting in the orthopaedic literature has been previously
evaluated15-17.
Bhandari et al. reviewed 2468 studies, and seventy-two (2.9%) were identified
as randomized
trials17. More than
half of the trials were limited by a lack of concealed randomization, lack of
blinding of outcome assessors, or failure to report reasons for excluding
patients. Those authors called for the application of standardized guidelines
for the reporting of clinical trials in orthopaedics to improve quality. In a
subsequent study, investigators applied the CONSORT criteria to a wide
spectrum of surgical
trials16. The
studies adhered to an average (and standard deviation) of 36% ± 34% of
critical CONSORT items, and the proportion of CONSORT items met by each trial
was an average of 32% ± 29%. At the very least, authors of surgical
randomized trials should endeavor to include sufficient detail to allow a
careful review of the methodology.
The CLEAR NPT is an important advancement in the assessment of surgical
trials over the CONSORT statement. Its focus on the reporting of interventions
that require some degree of technical skill in their administration places a
new value on the unique methodological considerations in conducting and
reporting surgical trials. Our experience in applying the CLEAR NPT to the
orthopaedic literature suggests the need for better surgical reporting. We
also found that modifications to this checklist may be considered to enhance
its reliability. Our work also suggests that readers of orthopaedic trials
should not assume that if it was not reported, it was not done. In many
circumstances, authors did use safeguards but simply did not report them.
Authors, journal editors, and reviewers may consider adopting the CLEAR NPT as
a standard checklist to improve the reporting of these trials.
The annotated CLEAR NPT and the questionnaire sent to authors are available
with the electronic versions of this article, on our web site at
(go to
the article citation and click on "Supplementary Material") and on
our quarterly CD-ROM (call our subscription department, at 781-449-9780, to
order the CD-ROM). ?
Altman DG, Schulz KF, Moher D, Egger M,
Davidoff F, Elbourne D, Gotzsche PC, Lang T; CONSORT GROUP (Consolidated
Standards of Reporting Trials). The revised CONSORT statement for reporting
randomized trials: explanation and elaboration. Ann Intern Med.
2001;134:
663-94.134663
2001
[PubMed]
Begg C, Cho M, Eastwood S, Horton R,
Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF. Improving
the quality of reporting of randomized controlled trials. The CONSORT
statement. JAMA. 1996;276:
637-9.276637
1996
[PubMed][CrossRef]
Moher D, Schulz KF, Altman D; CONSORT
Group (Consolidated Standards of Reporting Trials). The CONSORT statement:
revised recommendations for improving the quality of reports of parallel-group
randomized trials. JAMA.
2001;285:
1987-91.2851987
2001
[PubMed][CrossRef]
Boutron I, Tubach F, Giraudeau B, Ravaud
P. Methodological differences in clinical trials evaluating nonpharmacological
and pharmacological treatments of hip and knee osteoarthritis.
JAMA. 2003;290:
1062-70.2901062
2003
[PubMed][CrossRef]
Boutron I, Tubach F, Giraudeau B, Ravaud
P. Blinding was judged more difficult to achieve and maintain in
nonpharmacologic than pharmacologic trials. J Clin Epidemiol.
2004;57:
543-50.57543
2004
[PubMed][CrossRef]
Tu JV, Hannan EL, Anderson GM, Iron K,
Wu K, Vranizan K, Popp AJ, Grumbach K. The fall and rise of carotid
endarterectomy in the United States and Canada. N Engl J Med.
1998;339:
1441-7.3391441
1998
[PubMed][CrossRef]
Boutron I, Moher D, Tugwell P, Giraudeau
B, Poiraudeau S, Nizard R, Ravaud P. A checklist to evaluate a report of a
nonpharmacological trial (CLEAR NPT) was developed using consensus. J
Clin Epidemiol. 2005;58:
1233-40.581233
2005
[CrossRef]
Montori VM, Bhandari M, Devereaux PJ,
Manns BJ, Ghali WA, Guyatt GH. In the dark: the reporting of blinding status
in randomized controlled trials. J Clin Epidemiol.
2002;55:
787-90.55787
2002
[PubMed][CrossRef]
Devereaux PJ, Manns BJ, Ghali WA, Quan
H, Lacchetti C, Montori VM, Bhandari M, Guyatt GH. Physician interpretations
and textbook definitions of blinding terminology in randomized controlled
trials. JAMA. 2001;285:
2000-3.2852000
2001
[PubMed][CrossRef]
Moher D, Jones A, Lepage L; CONSORT
Group (Consolidated Standards for Reporting of Trials). Use of the CONSORT
statement and quality of reports of randomized trials: a comparative
before-and-after evaluation. JAMA.
2001;285:
1992-5.2851992
2001
[PubMed][CrossRef]
Plint AC, Moher D, Morrison A, Schulz K,
Altman DG, Hill C, Gabaury I. Does the CONSORT checklist improve the quality
of reports of randomised controlled trials? A systematic review. Med J
Aust. 2006;185:
263-7.185263
2006
Foley NC, Bhogal SK, Teasell RW, Bureau
Y, Speechley MR. Estimates of quality and reliability with the physiotherapy
evidence-based database scale to assess the methodology of randomized
controlled trials of pharmacological and nonpharmacological interventions.
Phys Ther. 2006;86:
817-24.86817
2006
[PubMed]
Jacquier I, Boutron I, Moher D, Roy C,
Ravaud P. The reporting of randomized clinical trials using a surgical
intervention is in need of immediate improvement: a systematic review.
Ann Surg. 2006;244:
677-83.244677
2006
[PubMed][CrossRef]
Devereaux PJ, Choi PT, El-Dika S,
Bhandari M, Montori VM, Schunemann HJ, Garg AX, Busse JW, Heels-Ansdell D,
Ghali WA, Manns BJ, Guyatt GH. An observational study found that authors of
randomized controlled trials frequently use concealment of randomization and
blinding, despite the failure to report these methods. J Clin
Epidemiol. 2004;57:
1232-6.571232
2004
[CrossRef]
Li P, Mah D, Lim K, Sprague S, Bhandari
M. Randomization and concealment in surgical trials: a comparison between
orthopaedic and non-orthopaedic randomized trials. Arch Orthop Trauma
Surg. 2005;125:
70-2.12570
2005
[CrossRef]
Bhandari M, Guyatt GH, Lochner H,
Sprague S, Tornetta P 3rd. Application of the Consolidated Standards of
Reporting Trials (CONSORT) in the Fracture Care Literature. J Bone
Joint Surg Am. 2002;84:
485-9.84485
2002
Bhandari M, Richards RR, Sprague S,
Schemitsch EH. The quality of reporting of randomized trials in the Journal of
Bone and Joint Surgery from 1988 through 2000. J Bone Joint Surg
Am. 2002;84:
388-96.84388
2002
Pildal J, Chan AW, Hrobjartsson A,
Forfang E, Altman DG, Gotzsche PC. Comparison of descriptions of allocation
concealment in trial protocols and the published reports: cohort study.
BMJ. 2005;330:
1049.3301049
2005
[PubMed][CrossRef]