0
Scientific Articles   |    
Reference Accuracy in Peer-Reviewed Pediatric Orthopaedic Literature
Jon R. Davids, MD1; Daniel M. Weigl, MD2; Joye P. Edmonds, MLIS, AHIP1; Dawn W. Blackhurst, DrPH3
1 Shriners Hospital for Children, 950 West Faris Road, Greenville, SC 29605. E-mail address for J.R. Davids: jdavids@shrinenet.org
2 Schneider Children's Medical Center, 14 Kaplan Street, Petah Tikva 49202, Israel
3 Greenville Hospital System University Medical Center, 701 Grove Road, Greenville, SC 29605
View Disclosures and Other Information
Disclosure: The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity.

Investigation performed at the Shriners Hospital for Children, Greenville, South Carolina; Schneider Children's Medical Center, Petah Tikva, Israel; and Greenville Hospital System University Medical Center, Greenville, South Carolina

Copyright ©2010 American Society for Journal of Bone and Joint Surgery, Inc.
J Bone Joint Surg Am, 2010 May 01;92(5):1155-1161. doi: 10.2106/JBJS.I.00063
5 Recommendations (Recommend) | 3 Comments | Saved by 3 Users Save Case

Abstract

Background: 

Reference accuracy of articles published in the biomedical literature is determined by the presence of citation and quotation errors. A recent review demonstrated that the median citation error rate per biomedical journal was 39%, and the median quotation error rate per journal was 20%. Reference accuracy in pediatric orthopaedic articles has not been previously reported, to our knowledge.

Methods: 

Two hundred references from twenty articles published in four peer-reviewed orthopaedic journals were randomly selected for assessment of citation and quotation accuracy. Full-text copies of all original references were obtained by interlibrary loan methods and reviewed directly to establish citation accuracy. The presence of citation errors was determined by a single investigator. The relevance of citation errors was determined by assessing the ease of reference retrieval through PubMed. Quotation accuracy was determined by two examiners who reviewed each of the twenty articles and 200 references to compare the claims made for the references in the article against the data and opinions expressed in the actual reference.

Results: 

The total citation error rate across all of the journals was 26% (fifty-one of 200 references) with a 95% confidence interval of 16.5% to 37.3%. The median citation error rate per journal was 27% (range, 10% to 38%). Although citation errors were common, most were of minimal significance, as 196 of the 200 references could be retrieved with ease from PubMed. The total quotation error rate across all of the articles was 38% (152 of 398 reference citations) with a 95% confidence interval of 30.1% to 47.0%. The median quotation error rate per journal was 38% (range, 28% to 46%).

Conclusions: 

Citation and quotation errors are common in the pediatric orthopaedic literature. Reference accuracy continues to be a substantial problem in the biomedical literature despite recent technological advances such as online databases, easily accessible search engines, and widely available bibliographic software.

Figures in this Article
    Sign In to Your Personal ProfileSign In To Access Full Content
    Not a Subscriber?
    Get online access for 30 days for $35
    New to JBJS?
    Sign up for a full subscription to both the print and online editions
    Register for a FREE limited account to get full access to all CME activities, to comment on public articles, or to sign up for alerts.
    Register for a FREE limited account to get full access to all CME activities
    Have a subscription to the print edition?
    Current subscribers to The Journal of Bone & Joint Surgery in either the print or quarterly DVD formats receive free online access to JBJS.org.
    Forgot your password?
    Enter your username and email address. We'll send you a reminder to the email address on record.

     
    Forgot your username or need assistance? Please contact customer service at subs@jbjs.org. If your access is provided
    by your institution, please contact you librarian or administrator for username and password information. Institutional
    administrators, to reset your institution's master username or password, please contact subs@jbjs.org

    References

    Accreditation Statement
    These activities have been planned and implemented in accordance with the Essential Areas and policies of the Accreditation Council for Continuing Medical Education (ACCME) through the joint sponsorship of the American Academy of Orthopaedic Surgeons and The Journal of Bone and Joint Surgery, Inc. The American Academy of Orthopaedic Surgeons is accredited by the ACCME to provide continuing medical education for physicians.
    CME Activities Associated with This Article
    Submit a Comment
    Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
    Comments are moderated and will appear on the site at the discretion of JBJS editorial staff.

    * = Required Field
    (if multiple authors, separate names by comma)
    Example: John Doe





    Brendan D. Masini, MD
    Posted on June 30, 2010
    Post-Publication Peer-Review
    Brooke Army Medical Center, Fort Sam Houston, Texas

    To the Editor:

    We congratulate the authors of “Reference Accuracy in Peer-Reviewed Pediatric Orthopaedic Literature” (1) for a fine publication. Their results revealed a weakness in the current peer-reviewed literature and, perhaps more importantly, reemphasize the onus on all authors to be meticulous in their research of references and use of quotations. We owe it to ourselves and our profession to be as accurate as possible when submitting a manuscript for publication.

    The larger issue that they expose is one that is, in most cases, ignored or forgotten by the casual recipient of scientific journals. Beyond holding authors accountable for their work, this study demonstrates the importance of the post-publication review process. The peer review process, while designed specifically to assure pre-publication review by leaders in the respective fields addressed in the article, also involves the post-publication review of articles and techniques to ensure accuracy of the material and in turn perpetuate the legitimacy of the publication.

    A scientific review of material and trends in the literature can be presented in the form of scientific inquiry as the authors have done nicely. This has been done previously on other topics such as positive result publication bias (2,3) and nonscientific factors associated with publication (4). This form of post-publication review can identify trends in the literature that may be appropriate and reassuring or identify a call to the editorship for change.

    Another form of post-publication review is the specific indictment of research practices or results that present faulty data to the profession. This can lead to the often more scandalous result of a retraction or correction having to be published in the journal. This was seen recently with a paper published in a different journal (5). In this case, upon seeing the manuscript in print, the “alleged co-authors” contacted the editor, bringing to question the legitimacy of the study. These actions ultimately resulted in a formal withdrawal of the paper by the journal (6). This scenario can be looked at as failure by a journal to appropriately screen and perform a peer-review in the pre-publication setting, or alternatively, as a great success of the post-publication peer-review that restored truth to that particular scientific inquiry.

    The journal is not exposing weakness by admitting to, and correcting faulty information. Rather it is made stronger by the practice. The readership, in turn, should take pride in their involvement in the peer-review process as each individual member is a part of the advancement of the profession.

    The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity.

    References

    1. Davids JR, Weigl DM, Edmonds JP, Blackhurst DW. Reference accuracy in peer-reviewed pediatric orthopaedic literature. J Bone Joint Surg Am. 2010;92:1155-61.

    2. Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M. Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008;90:595-601.

    3. Lynch JR, Cunningham MR, Warme WJ, Schaad DC, Wolf FM, Leopold SS. Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. J Bone Joint Surg Am. 2007;89:1010-8.

    4. Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M. Nonscientific factors associated with acceptance for publication in The Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008;90:2432-7.

    5. Kuklo TR, Groth AT, Anderson RC, Frisch HM, Islinger RB. Recombinant human bone morphogenetic protein-2 for grade III open segmental tibial fractures from combat injuries in Iraq. J Bone Joint Surg Br. 2008;90:1068-72.

    6. Scott J. Withdrawal of a paper. J Bone Joint Surg Br. 2009;91:285-6.

    Jon R. Davids, MD
    Posted on June 02, 2010
    Drs. Davids and Weigl respond to Mr. Barnes
    Shriners Hospital for Children, Greenville, South Carolina

    We thank Mr. Barnes for his careful review of our study and his thoughtful comments. To our embarrassment, and despite our best efforts, our article on reference accuracy contained a citation error (1). In the classification scheme employed in our study, this would be considered a major citation error. The reference in question was itself a letter to the editor of the BMJ concerning reference accuracy. This letter was not accessible through PubMed (2), but was identified through our review of the literature and retrieved manually. The citation error is therefore attributable to human error on the part of the senior author.

    With respect to citation errors, it is our opinion that the impact of literature search engines and bibliographic software is generally positive. Recurrent manual collection and documentation of literature references creates repeated opportunity for new errors of citation. As noted in our study, the significance of citations errors is minimized by the strength of these technologies.

    Mr. Barnes also raises the issue of quotation errors, and we agree that they are more insidious and present a more difficult problem to solve. Stakeholders in addressing the problem of reference accuracy include the authors, the journals, and the readers. Direct verification of references for citation and quotation accuracy is certainly the responsibility of the authors. Responsible parties at the journal level include the editors and reviewers. The editors have the latitude to establish journal policies and practices. Our study confirmed that post-submission, pre-publication technical editing can improve citation accuracy. It is our opinion that clarification of the role of references, limiting the number and type of references permitted (e.g. primary sources only), and re-publication of classic articles (to promote review of primary sources, in individual or journal club settings) could have positive impact on quotation accuracy as well. Peer reviewers generally serve on a voluntary basis, and may be motivated to do so for a variety of reasons, including professionalism, prestige, and academic career advancement. To expect peer reviewers to monitor for reference accuracy seems unrealistic given the additional time that would be required. Finally, the readers should be aware of the issues and challenges related to reference accuracy. Residency training programs should support journal club activities (e.g. current, topical, and classic literature formats), where critical review of the literature by experienced mentors establishes a “caveat emptor” approach for the physicians in training.

    References

    1. Davids JR, Weigl DM, Edmonds JP, Blackhurst DW. Reference accuracy in peer-reviewed pediatric orthopaedic literature. J Bone Joint Surg Am. 2010;92:1155-61.

    2. PubMed. http://www.ncbi.nlm.nih.gov/sites/entrez. Accessed 2010 May 26.

    Michael R. Barnes
    Posted on May 14, 2010
    Accuracy of References
    Leicester General Hospital, Leicester, United Kingdom

    To the Editor:

    I was very interested to read the recent paper by Davids et al. (1) on the accuracy of references. It is a serious matter that needs to be resolved urgently, particularly in terms of quotation errors which can then permeate throughout the entire body of knowledge. I’m sure that we are all familiar with examples of this.

    What stimulated me to write was an obvious minor error in reference 26, BJM instead of BMJ, but this made me ask the question “Who else has noticed this error?” which leads on to the wider question “How many have actually read this paper?” There may now be a Journal Impact Factor, but are there any figures collected for readership of a particular article?

    As medicine and journals become ever more specialized but at the same time the number and frequency of journals increases, the readership of an individual article must inevitably decrease. As a general rule, with modern day pressures on time, readers will only read papers that are related to their particular areas of interest. Perhaps then it could be left up to the readers to check on the references as they are likely to be fairly familiar with the field. This will only work so long as corrections are published quickly and prominently.

    The current common practice would seem to be that it is the role of the authors to check the references. But should it not be the role of the reviewer? Surely peer review strongly implies that the reviewer has an intimate knowledge of the field and therefore should have an intimate knowledge of the relevant literature, and hence should easily spot any errors in the referencing. Again this is difficult because of pressures of time, but it would be helped by limiting the number of references allowed. This idea could be helped by the journals explaining clearly and frequently what purposes references actually serve. A long list often just seems to be intended to impress. With modern technology and systems, it is very easy to quickly generate dozens of references, without spending long hours in the library – but of these how many are ever read or indeed relevant? Perhaps it is now time for the journals to alter their methods to take account of modern day electronic search facilities.

    The author did not receive any outside funding or grants in support of his research for or preparation of this work. Neither he nor a member of his immediate family received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity.

    References

    1. Davids JR, Weigl DM, Edmonds JP, Blackhurst DW. Reference accuracy in peer-reviewed pediatric orthopaedic literature. J Bone Joint Surg Am. 2010;92:1155-61.

    Related Content
    The Journal of Bone & Joint Surgery
    JBJS Case Connector
    Topic Collections
    Related Audio and Videos
    PubMed Articles
    Clinical Trials
    Readers of This Also Read...
    JBJS Jobs
    12/04/2013
    New York - Icahn School of Medicine at Mount Sinai
    02/28/2014
    District of Columbia (DC) - Children's National Medical Center
    12/31/2013
    S. Carolina - Department of Orthopaedic Surgery Medical Univerity of South Carlonina
    04/02/2014
    W. Virginia - Charleston Area Medical Center