Scientific Articles   |    
Doubling the Impact: Publication of Systematic Review Articles in Orthopaedic Journals
Mohit Bhandari, MD, MSc1; Victor M. Montori, MD, MSc1; Philip J. Devereaux, MD1; Nancy L. Wilczynski, MSc1; Douglas Morgan1; R. Brian Haynes, MD, PhD1
1 Department of Clinical Epidemiology and Biostatistics (M.B., V.M.M., N.L.W., D.M., and R.B.H.) and Health Research Methodology Program (P.J.D.), McMaster University Medical Centre, 1200 Main Street West, Room 2C9, Hamilton, ON L8N 3Z5, Canada. E-mail address for M. Bhandari: bhandari@simpatico.ca
View Disclosures and Other Information
In support of their research or preparation of this manuscript, one or more of the authors received grants or outside funding from the United States National Library of Medicine and a senior research fellowship from the Canadian Institutes of Health Research (P.J.D.). None of the authors received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, educational institution, or other charitable or nonprofit organization with which the authors are affiliated or associated.
Investigation performed at the Health Information Research Unit, McMaster University, Hamilton, Ontario, Canada

The Journal of Bone and Joint Surgery, Incorporated
J Bone Joint Surg Am, 2004 May 01;86(5):1012-1016
5 Recommendations (Recommend) | 3 Comments | Saved by 3 Users Save Case


Background: Investigators aim to publish their research papers in top journals to disseminate their findings to the widest possible audience. Systematic reviews of the literature occupy the highest position in currently proposed hierarchies of evidence. We hypothesized that the number of citations (a measure of scholarly interest) for systematic reviews (or meta-analyses) published in leading orthopaedic journals would be greater than the number of citations for narrative reviews published in the same journals.

Methods: We identified fifteen journals that had high Science Citation Index impact factors for the orthopaedic subspecialty and were believed to have a higher yield of studies and reviews of scientific merit and clinical relevance. For the year 2000, six research associates applied methodological criteria to each article in each issue of the fifteen journals to determine whether the article was scientifically sound (rigorous versus nonrigorous). Of the 3916 articles identified, 2331 were original or review articles. We queried the ISI (Institute for Scientific Information) Web of Science database to ascertain, as of March 2003, the number of subsequent citations to each one of the reviews after its original publication in all journals that published both narrative and systematic reviews.

Results: Of the 2331 articles published across the fifteen journals in the year 2000, 110 were review articles. Only seventeen (15%) of the 110 reviews met our criteria for systematic reviews with rigor. Rigorous systematic reviews received more than twice the mean number of citations compared with other systematic or narrative reviews (13.8 compared with 6.0, p = 0.008). The rigor of a review was a significant predictor of the number of citations in other orthopaedic journals (p = 0.01). In addition, rigor was significantly associated with the number of citations in nonorthopaedic journals (p = 0.03).

Conclusions: Our findings suggest that journal editors and authors can improve the relevance and scholarly interest in their reviews (as shown by the number of citations) by meeting standard guidelines for methodological rigor.

Figures in this Article
    Sign In to Your Personal ProfileSign In To Access Full Content
    Not a Subscriber?
    Get online access for 30 days for $35
    New to JBJS?
    Sign up for a full subscription to both the print and online editions
    Register for a FREE limited account to get full access to all CME activities, to comment on public articles, or to sign up for alerts.
    Register for a FREE limited account to get full access to all CME activities
    Have a subscription to the print edition?
    Current subscribers to The Journal of Bone & Joint Surgery in either the print or quarterly DVD formats receive free online access to JBJS.org.
    Forgot your password?
    Enter your username and email address. We'll send you a reminder to the email address on record.

    Forgot your username or need assistance? Please contact customer service at subs@jbjs.org. If your access is provided
    by your institution, please contact you librarian or administrator for username and password information. Institutional
    administrators, to reset your institution's master username or password, please contact subs@jbjs.org


    Accreditation Statement
    These activities have been planned and implemented in accordance with the Essential Areas and policies of the Accreditation Council for Continuing Medical Education (ACCME) through the joint sponsorship of the American Academy of Orthopaedic Surgeons and The Journal of Bone and Joint Surgery, Inc. The American Academy of Orthopaedic Surgeons is accredited by the ACCME to provide continuing medical education for physicians.
    CME Activities Associated with This Article
    Submit a Comment
    Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
    Comments are moderated and will appear on the site at the discretion of JBJS editorial staff.

    * = Required Field
    (if multiple authors, separate names by comma)
    Example: John Doe

    Helen H. Handoll
    Posted on June 21, 2004
    Focus on Impact Factors Misses Out an Important Source of Orthopaedic Systematic Reviews
    University of Teesside

    To the Editor:

    In “Doubling the Impact: Publication of Systematic Review Articles in Orthopaedic Journals” ,Bhandari et al. (1) reveal the unsatisfactory nature of the majority of reviews published in fifteen orthopaedic journals. They refer to their previous study of the methodological rigor of orthopaedic meta-analyses[1], where the findings of generally poor methodological quality applied to reviews published in journals but not to Cochrane reviews[2,3].

    Obviously, Cochrane reviews could not be included in Bhandari et al., despite the frequency with which they are cited, as these are not currently indexed in the citation indexes. However the authors did allude to the fact that “The Cochrane Collaboration… has published in excess of 1000 systematic reviews since its inception"[4].

    The number of rigorously conducted and maintained systematic reviews in the Cochrane Database of Systematic Reviews now approximates 2000[5]. At least 155 of these systematic reviews generated by members of the Back, Musculoskeletal, and Musculoskeletal Injuries Collaborative Review Groups are of direct relevance to orthopaedics. This contrasts to the seventeen systematic reviews identified by Bhandari et al. in the fifteen journals with high Science Citation Index impact factors for orthopaedics. It should be noted that four out of the eight systematic reviews identified in Spine are print versions of Cochrane reviews.

    As acknowledged by Bhandari et al. impact factors are an imperfect measure[6]. While we endorse Bhandari et al's call for improvements in the methodological rigor in the conduct and reporting of reviews in journals, we caution against the use of impact factors as a major determinant of quality of individual reviews/papers[6]. Finally, we would like to encourage authors and journal editors alike to improve not only the quality of the design and reporting of review articles[7] but also of the primary research underpinning these reviews.

    Helen H. Handoll, DPhil

    Lesley D. Gillespie, MMedSci

    William J. Gillespie, FRACS

    Rajan Madhok, FRCS

    Martyn J. Parker, MD

    We are all members of the Cochrane Musculoskeletal Injuries Collaborative Review Group

    Teesside Centre for Rehabilitation Sciences, The James Cook University Hospital, Marton Road, Middlesbrough, TS4 3BW, UK. E-mail address: h.handoll@ed.ac.uk

    1. Bhandari M, Morrow F, Kulkarni AV, Tornetta P 3rd. Meta-analyses in orthopaedic surgery. A systematic review of their methodologies. J Bone Joint Surg Am. 2001; 83:15-24.

    2. Parker M, Gillespie L, Gillespie W, Handoll H, Madhok R, Morton L et al. Systematic reviews, meta-analyses, and methodology. J Bone Joint Surg Am. 2001; 83:1433-4.

    3. Bhandari M, Morrow F, Kulkarni AV, Tornetta P 3rd. Systematic reviews, meta-analyses, and methodology. J Bone Joint Surg Am. 2001; 83:1434-5.

    4. Chalmers I. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews on the effects of health care. Ann NY Acad Sci. 1993;703:156-65.

    5. The Cochrane Library, Issue 2, 2004. Chichester, UK: John Wiley & Sons, Ltd.

    6. Kurmis AP. Understanding the limitations of the journal impact factor. J Bone Joint Surg Am. 2003; 85:2449-54.

    7. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 1999; 354:1896-1900.

    Related Content
    The Journal of Bone & Joint Surgery
    JBJS Case Connector
    Topic Collections
    Related Audio and Videos
    Clinical Trials
    Readers of This Also Read...
    JBJS Jobs
    IL - Hinsdale Orthopaedics
    LA - Louisiana State University Health Sciences Center-Shreveport
    PA - Thomas Jefferson University
    LA - Ochsner Health System