Scientific Articles   |    
Twenty Years of Meta-Analyses in Orthopaedic Surgery: Has Quality Kept Up with Quantity?
Bernadette G. Dijkman, BSc1; Jihad A.K. Abouali, MD1; Bauke W. Kooistra, BSc1; Henry J. Conter, MD1; Rudolf W. Poolman, MD, PhD2; Abhaya V. Kulkarni, MD, PhD3; Paul Tornetta, III, MD4; Mohit Bhandari, MD, MSc, FRCSC1
1 Division of Orthopaedic Surgery, Department of Surgery, McMaster University, 293 Wellington Street North, Suite 110, Hamilton, ON L8L 2X2, Canada. E-mail address for M. Bhandari: bhandam@mcmaster.ca
2 Department of Orthopaedic Surgery, Onze Lieve Vrouwe Gasthuis, Postbus 95500, 1090 HM Amsterdam, the Netherlands
3 Division of Neurosurgery, The Hospital for Sick Children, 555 University Avenue, Toronto, ON M5G 1X8, Canada
4 Department of Orthopaedic Surgery, Boston University Medical Center, 850 Harrison Avenue, Boston, MA 02118
View Disclosures and Other Information
Disclosure: The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. One of the authors receives salary support, in part, from a Canada Research Chair, McMaster University.

A commentary by Elena Losina, PhD, and Jeffrey N. Katz, MD, MSc, is available at www.jbjs.org/commentary and as supplemental material to the online version of this article.
Investigation performed at Division of Orthopaedic Surgery, McMaster University, Hamilton, Ontario, Canada

Copyright ©2010 American Society for Journal of Bone and Joint Surgery, Inc.
J Bone Joint Surg Am, 2010 Jan 01;92(1):48-57. doi: 10.2106/JBJS.I.00251
5 Recommendations (Recommend) | 3 Comments | Saved by 3 Users Save Case



As the number of studies in the literature is increasing, orthopaedic surgeons highly depend on meta-analyses as their primary source of scientific evidence. The objectives of this review were to assess the scientific quality and number of published meta-analyses on orthopaedics-related topics over time.


We conducted, in duplicate and independently, a systematic review of published meta-analyses in orthopaedics in the years 2005 and 2008 and compared them with a previous systematic review of meta-analyses from 1969 to 1999. A search of electronic databases (MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews) was performed to identify meta-analyses published in 2005 and 2008. We searched bibliographies and contacted content experts to identify additional relevant studies. Two investigators independently assessed the quality of the studies, using the Oxman and Guyatt index, and abstracted relevant data.


We included forty-five and forty-four meta-analyses from 2005 and 2008, respectively. While the number of meta-analyses increased fivefold from 1999 to 2008, the mean quality score did not change significantly over time (p = 0.067). In the later years, a significantly lower proportion of meta-analyses had methodological flaws (56% in 2005 and 68% in 2008) compared with meta-analyses published prior to 2000 (88%) (p = 0.006). In 2005 and 2008, respectively, 18% and 30% of the meta-analyses had major to extensive flaws in their methodology. Studies from 2008 with positive conclusions used and described appropriate criteria for the validity assessment less often than did those with negative results. The use of random-effects and fixed-effects models as pooling methods became more popular toward 2008.


Although the methodological quality of orthopaedic meta-analyses has increased in the past twenty years, a substantial proportion continues to show major to extensive flaws. As the number of published meta-analyses is increasing, a routine checklist for scientific quality should be used in the peer-review process to ensure methodological standards for publication.

Figures in this Article
    Sign In to Your Personal ProfileSign In To Access Full Content
    Not a Subscriber?
    Get online access for 30 days for $35
    New to JBJS?
    Sign up for a full subscription to both the print and online editions
    Register for a FREE limited account to get full access to all CME activities, to comment on public articles, or to sign up for alerts.
    Register for a FREE limited account to get full access to all CME activities
    Have a subscription to the print edition?
    Current subscribers to The Journal of Bone & Joint Surgery in either the print or quarterly DVD formats receive free online access to JBJS.org.
    Forgot your password?
    Enter your username and email address. We'll send you a reminder to the email address on record.

    Forgot your username or need assistance? Please contact customer service at subs@jbjs.org. If your access is provided
    by your institution, please contact you librarian or administrator for username and password information. Institutional
    administrators, to reset your institution's master username or password, please contact subs@jbjs.org


    Accreditation Statement
    These activities have been planned and implemented in accordance with the Essential Areas and policies of the Accreditation Council for Continuing Medical Education (ACCME) through the joint sponsorship of the American Academy of Orthopaedic Surgeons and The Journal of Bone and Joint Surgery, Inc. The American Academy of Orthopaedic Surgeons is accredited by the ACCME to provide continuing medical education for physicians.
    CME Activities Associated with This Article
    Submit a Comment
    Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
    Comments are moderated and will appear on the site at the discretion of JBJS editorial staff.

    * = Required Field
    (if multiple authors, separate names by comma)
    Example: John Doe

    Mohit Bhandari, MD, MSc, FRCSC
    Posted on February 09, 2010
    Dr. Bhandari responds to Ms. Hing and Mr. Smith
    McMaster University, Ontario, Canada

    We appreciate the concerns about reporting quality and the need for standardization in the orthopaedic literature. The PRISMA checklist is the latest iteration of a reporting guideline and many journals have indeed adopted it for all submitted systematic reviews.

    We tend to believe that, along with better reporting, we require more education about the scientific conduct of high quality reviews as well as a better understanding among investigators and journal editorial boards regarding the importance of systematic reviews in providing a barometer on the "best available" evidence. This would include negative meta-analyses as well as those with relatively poor quality primary studies.

    CB Hing, MBBS, BSc, MSc, MD, FRCS, FRCS(Tr&Orth)
    Posted on January 31, 2010
    Meta-Analyses Peer Review
    Watford General Hospital, Watford, United Kingdom

    To the Editor:

    We read with interest the article by Dijkman et al. (1) regarding the quality of orthopaedic meta-analysis and welcome the publication of research in this area. Whilst the authors have comprehensively investigated the methodology of published meta-analyses, they have not discussed the contribution of peer review in the publication process. If peer reviewers and editors are the ‘gatekeepers’ of the published literature, then the onus also lies on them to accept or reject submitted meta-analyses on the basis of their quality (2). Conversely, if knowledge is based on all research undertaken regardless of methodological quality, then a meta-analysis would not be flawed through publication bias (3), for example, by including such data when acknowledging its origin as less rigorous research. Furthermore, we agree that standardization of guidelines for meta-analyses should be encouraged and may be recommended as a universal requirement of author submission. The authors have correctly stated that this has been suggested through the QUORUM statement, but, of note, this has now been superceded by PRISMA (4).

    The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the authors, or a member of their immediate families, are affiliated or associated.


    1. Dijkman BG, Abouali JA, Kooistra BW, Conter HJ, Poolman RW, Kulkarni AV, Tornetta P 3rd, Bhandari M. Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity? J Bone Joint Surg Am. 2010;92:48-57.

    2. Opthof T, Coronel R, Janse MJ. The significance of the peer review process against the background of bias: priority ratings of reviewers and editors and the prediction of citation, the role of geographical bias. Cardiovasc Res. 2002:56;339-46.

    3. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Shing C, Pang C, Harvey I. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010. In press.

    4. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:e1-34.

    Related Content
    The Journal of Bone & Joint Surgery
    JBJS Case Connector
    Topic Collections
    Related Audio and Videos
    Clinical Trials
    Readers of This Also Read...
    JBJS Jobs
    S. Carolina - Department of Orthopaedic Surgery Medical Univerity of South Carlonina
    W. Virginia - Charleston Area Medical Center
    New York - Icahn School of Medicine at Mount Sinai
    District of Columbia (DC) - Children's National Medical Center