0
Scientific Articles   |    
Quality of Prospective Controlled Randomized TrialsAnalysis of Trials of Treatment for Lateral Epicondylitis as an Example
James Cowan, BA1; Santiago Lozano-Calderón, MD1; David Ring, MD, PhD1
1 Orthopaedic Hand and Upper Extremity Service, Massachusetts General Hospital, Yawkey Center 2100, 55 Fruit Street, Boston, MA 02114. E-mail address for D. Ring: dring@partners.org
View Disclosures and Other Information
Disclosure: In support of their research for or preparation of this work, one or more of the authors received, in any one year, outside funding or grants in excess of $10,000 from the AO Foundation, Small Bone Innovations, Wright Medical, Joint Active Systems, and Smith and Nephew. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the authors, or a member of their immediate families, are affiliated or associated.
A commentary is available with the electronic versions of this article, on our web site (www.jbjs.org) and on our quarterly CD-ROM (call our subscription department, at 781-449-9780, to order the CD-ROM).
Investigation performed at the Orthopaedic Hand and Upper Extremity Service, Massachusetts General Hospital, Boston, Massachusetts

The Journal of Bone and Joint Surgery, Incorporated
J Bone Joint Surg Am, 2007 Aug 01;89(8):1693-1699. doi: 10.2106/JBJS.F.00858
5 Recommendations (Recommend) | 3 Comments | Saved by 3 Users Save Case

Abstract

Background: The Oxford Levels of Evidence are now routinely assigned at many orthopaedic journals. One disadvantage of this approach is that study designs with a higher level of evidence may be given greater weight than the overall quality of the study merits. In other words, there is no guarantee that research is scientifically valid simply because a more sophisticated study design was employed. The aim of this study was to review Level-I and II therapeutic studies on lateral epicondylitis to measure variation in quality among the highest-level study designs.

Methods: Fifty-four prospective randomized therapeutic trials involving patients with lateral epicondylitis were evaluated by two independent reviewers according to the Oxford Levels of Evidence, a modification of the Coleman Methodology Score (a 0 to 100-point scale), and the revised CONSORT (Consolidated Standards of Reporting Trials) score.

Results: The two reviewers were consistent in their use of the Oxford Levels of Evidence (? = 0.73, p < 0.01), the modified Coleman Methodology Score (? = 0.73; p < 0.01), and the CONSORT score (? = 0.53; p < 0.01). Both reviewers rated the majority of studies as Level II (91% and 94%) and as unsatisfactory according to the Coleman Methodology Score (87% and 89%) and the CONSORT score (62% and 63%). Areas of deficiency included poor descriptions of recruitment (>90% of the trials), power-level calculations (73%), randomization (58%), blinding (90%), and participant flow (50%) as well as inadequate follow-up, sample size, and blinding.

Conclusions: The use of the gold-standard trial design, the prospective randomized therapeutic study (Level-I or II evidence), does not ensure quality research or reporting. Critical analysis of scientific work is important regardless of the study design. Clinical scientists should be familiar with the CONSORT criteria and adhere to them when reporting clinical trials.

Figures in this Article
    Sign In to Your Personal ProfileSign In To Access Full Content
    Not a Subscriber?
    Get online access for 30 days for $35
    New to JBJS?
    Sign up for a full subscription to both the print and online editions
    Register for a FREE limited account to get full access to all CME activities, to comment on public articles, or to sign up for alerts.
    Register for a FREE limited account to get full access to all CME activities
    Have a subscription to the print edition?
    Current subscribers to The Journal of Bone & Joint Surgery in either the print or quarterly DVD formats receive free online access to JBJS.org.
    Forgot your password?
    Enter your username and email address. We'll send you a reminder to the email address on record.

     
    Forgot your username or need assistance? Please contact customer service at subs@jbjs.org. If your access is provided
    by your institution, please contact you librarian or administrator for username and password information. Institutional
    administrators, to reset your institution's master username or password, please contact subs@jbjs.org

    References

    Accreditation Statement
    These activities have been planned and implemented in accordance with the Essential Areas and policies of the Accreditation Council for Continuing Medical Education (ACCME) through the joint sponsorship of the American Academy of Orthopaedic Surgeons and The Journal of Bone and Joint Surgery, Inc. The American Academy of Orthopaedic Surgeons is accredited by the ACCME to provide continuing medical education for physicians.
    CME Activities Associated with This Article
    Submit a Comment
    Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
    Comments are moderated and will appear on the site at the discretion of JBJS editorial staff.

    * = Required Field
    (if multiple authors, separate names by comma)
    Example: John Doe





    David C. Ring, M.D.
    Posted on September 28, 2007
    Dr. Ring et al. respond to Dr. Leahy
    Massachusetts General Hospital, Boston, MA

    We apologize to Dr. Leahy and the readers of THE Surgeon, and to the editors and readers of the other journals mentioned for our unfortunate choice of words. We did not mean to offend.

    Austin L. Leahy, MCH, FRCS, FRCSI
    Posted on September 27, 2007
    Unititled
    The Surgeon: Journal of the Royal College of Surgeons of Edinburgh and Ireland

    To The Editor:

    Your authors, in their excellent article(1), refer to trials which were reported in relatively "obscure" journals. On behalf of the journals mentioned, may I, as the Editor of THE Surgeon: Journal of the Royal College of Surgeons of Edinburgh and Ireland, respond. The use of the adjective "obscure" is an interesting one. Presumably the authors did not mean to imply that our journals were not prominent or famous, were relatively unknown, or not clearly seen or easily distinguished.

    THE Surgeon circulates to over twenty thousand surgeons internationally and is a fully indexed journal with an impact factor 0.99. Obscure, I rather think not!

    The author did not receive any outside funding or grants in support of his research for or preparation of this work. Neither he nor a member of his immediate family received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the author, or a member of his immediate family, is affiliated or associated .

    Reference:

    1. Cowan J, Lozano-Calderon S, Ring D. Quality of prospective controlled randomized trials. Analysis of trials of treatment for lateral epicondylitis as an example. J Bone Joint Surg Am. 2007;89:1693-1699.

    David Ring, M.D.
    Posted on September 25, 2007
    Dr. Ring responds to Drs. Poolman and Bhandari
    Massachusetts General Hospital, Boston, MA

    The welcome comments of Drs. Poolman and Bhandari further illustrate the passion, enthusiasm, and detail that is increasingly placed not only in performing good science, but also in evaluating it. If science can be characterized as "organized skepticism", then the time when seemingly scientific pronouncements could go without challenge is clearly fading.

    Rudolf W. Poolman MD PhD
    Posted on September 13, 2007
    Quality assessment of RCT reports
    Onze Lieve Vrouwe Gasthuis, Amsterdam, The Netherlands

    To The Editor:

    With interest we read the manuscript by Cowan et al.(1). This manuscript is another example of the increased awareness among orthopaedic surgeons of the quality of reporting in trials. We further applaud the authors’ effort to highlight importance of individual validity criteria in assessing the quality of a randomized trial. We wish to provide two suggestions: 1) Use of overall summary scores may not be as helpful in interpreting the data and 2) newer checklists that focus on non- pharmacological therapies may be of interest in future research in this area.

    Summarizing scores of checklists has limitations. Juni et al. have elucidated this issue(2). When a mixture of pharmaceutical and non- pharmaceutical trials are evaluated, as Cowan and co-workers(1) did, a highest possible quality non-pharmaceutical RCT will score lower than a highest possible quality pharmaceutical RCT. Therefore, we and others recommend reporting individual checklist items rather than overall summary scores(3). The use of thresholds may also skew the direction of results and may lead to false conclusions in a meta-analyses(2). Furthermore, Juni et al. discouraged the use of individual scales as absolute and objective measures of trial quality and noted "relevant methodological aspects should be identified, ideally a priori, and assessed individually" (2). The authors did provide some data on the proportion of trials that met individual criteria and we suspect space constraints for journal publication may have limited complete tabular breakdowns of the proportion of RCTs that met each criterion of the Coleman and CONSORT checklists. This would be very helpful information to further interpret their findings. Perhaps the Journal could publish these findings online?

    Ideally, scales that are used to measure the quality of reporting of surgical trials should be tailored to the maximal possible quality, rather than to a unique gold-standard quality(2,3). Therefore, the Cochrane Collaboration's handbook advises to describe aspects of critical appraisal separately and to avoid summarizing results(4). Our previous study confirmed the variability of scores across each item of the Cochrane reporting quality assessment tool reviewing RCTs published in the Journal (3).

    We would also like to suggest checklists that have been developed to evaluate non-pharmaceutical studies(5,6). Although, frequently used, these checklists themselves lack a validation process. The Coleman Methodology Score is one example, especially if this score was modified. Utilization of modified scores without revalidation can skew results(7). Using a standardized checklist tailored to non-pharmaceutical trials, the CLEAR NPT(5) may facilitate comparison of different studies clarifying the quality of reporting in surgical trials including orthopaedics. This checklist was developed to overcome the limitations of the CONSORT checklist(5). Currently this checklist is under ongoing evaluation.

    The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the authors, or a member of their immediate families, are affiliated or associated.

    References:

    1. Cowan, J., Lozano-Calderon, S., and Ring, D.: Quality of prospective controlled randomized trials. Analysis of trials of treatment for lateral epicondylitis as an example. J Bone Joint Surg Am. 89:1693- 1699, 2007.

    2. Juni, P., Witschi, A., Bloch, R., and Egger, M.: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 282:1054- 1060, 1999.

    3. Poolman, R. W., Struijs, P. A., Krips, R., Sierevelt, I. N., Lutz, K. H., and Bhandari, M.: Does a "Level I Evidence" rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC Medical Research Methodology. 6:44, 2006.

    4. Higgins, J. P. T., Green, S., and Editors: Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. 2006.

    5. Boutron, I., Moher, D., Tugwell, P., Giraudeau, B., Poiraudeau, S., Nizard, R., and Ravaud, P.: A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. J Clin. Epidemiol. 58:1233-1240, 2005.

    6. Jacquier, I., Boutron, I., Moher, D., Roy, C., and Ravaud, P.: The Reporting of Randomized Clinical Trials Using a Surgical Intervention Is in Need of Immediate Improvement: A Systematic Review. Ann Surg. 244:677-683, 2006.

    7. Poolman, R. W., Struijs, P. A., Krips, R., Sierevelt, I. N., Marti, R. K., Farrokhyar, F., and Bhandari, M.: Reporting of outcomes in orthopaedic randomized trials: does blinding of outcome assessors matter? J Bone Joint Surg Am. 89:550-558, 2007.

    Related Content
    The Journal of Bone & Joint Surgery
    JBJS Case Connector
    Topic Collections
    Related Audio and Videos
    PubMed Articles
    Clinical Trials
    Readers of This Also Read...
    JBJS Jobs
    02/05/2014
    Oregon - The Center - Orthopedic and Neurosurgical Care and Research
    04/02/2014
    Illinois - Hinsdale Orthopaedics
    04/16/2014
    Connecticut - Yale University School of Medicine