The Journal of Bone & Joint Surgery

The Journal of Bone & Joint Surgery, Volume 86, Issue 8

Scientific Articles | August 01, 2004

Interobserver Agreement in the Application of Levels of Evidence to Scientific Papers in the American Volume of The Journal of Bone and Joint Surgery

Mohit Bhandari, MD, MSc¹; Marc F. Swiontkowski, MD²; Thomas A. Einhorn, MD³; Paul TornettaIII, MD⁴; Emil H. Schemitsch, MD⁵; Pamela Leece, BSc¹; Sheila Sprague, BSc¹; James G. Wright, MD⁶

¹ Department of Clinical Epidemiology and Biostatistics, McMaster University Medical Centre, 1200 Main Street West, Room 2C9, Hamilton, ON L8N 3Z5, Canada. E-mail address for M. Bhandari: bhandari@sympatico.ca
² Department of Orthopaedic Surgery, University of Minnesota Medical School, 420 Delaware Street S.E., Box 492, Minneapolis, MN 55455
³ Department of Orthopaedic Surgery, Boston University Medical Center, 720 Harrison Avenue, Suite 808, Boston, MA 02118
⁴ Department of Orthopaedic Surgery, Boston University Medical Center, 818 Harrison Avenue, Boston, MA 02118
⁵ Division of Orthopaedic Surgery, Department of Surgery, St. Michael's Hospital, University of Toronto, 55 Queen Street East, Suite 800, Toronto, ON M5C 1R6, Canada
⁶ Division of Orthopaedic Surgery, The Hospital for Sick Children, 555 University Avenue, Suite 107, Toronto, ON M5G 1X8, Canada

View Disclosures and Other Information

The authors did not receive grants or outside funding in support of their research or preparation of this manuscript. They did not receive payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, educational institution, or other charitable or nonprofit organization with which the authors are affiliated or associated.

Investigation performed at the Department of Clinical Epidemiology and Biostatistics, McMaster University Medical Centre, Hamilton, Ontario, the Departments of Orthopaedic Surgery, University of Minnesota Medical School, Minneapolis, Minnesota, Boston University Medical Center, Boston, Massachusetts, and the Division of Orthopaedic Surgery, St. Michael's Hospital, University of Toronto, Toronto, Ontario, Canada

The Journal of Bone and Joint Surgery, Incorporated

J Bone Joint Surg Am, 2004 Aug 01;86(8):1717-1720

5 Recommendations (Recommend) | 3 Comments | Saved by 3 Users Save Case

Article

Comments (0)

text A A A

Abstract

Background: Since January 2003, all clinical scientific articles published in the American volume of The Journal of Bone and Joint Surgery (JBJS-A) have included a level-of-evidence rating. The aim of the current study was to evaluate the interobserver agreement among reviewers, with varying levels of epidemiology training, in categorizing the levels of evidence of these clinical studies.

Methods: Fifty-one consecutive clinical papers published in the American volume of JBJS were identified by a computerized search of the table of contents from January 2003 through June 2003. Each paper was blinded so that only the title, abstract (without the level of evidence designated), and methods section were provided to the reviewers. The papers were coded and were randomly organized in a binder. Six surgeons graded each blinded paper for (1) the type of study (therapeutic, prognostic, diagnostic test, or economic or decision analysis), (2) the level of evidence (on a scale of I through V), and (3) the subcategory within the particular level of evidence. Three surgeons were members of JBJS American Editorial Board, two surgeons were reviewers for JBJS-A, and one surgeon was an active researcher not formally associated with JBJS-A. The reviewers did not receive any formal training in the application of the classification system, but each was provided with a detailed description of the classification system used by JBJS-A. Intraclass correlation coefficients with 95% confidence intervals were determined for the reviewers' agreement regarding the type of study, level of evidence, and subcategory within the level of evidence.

Results: The majority (69%) of the fifty-one included articles were studies of therapy, and 57% of the studies constituted Level-IV evidence. The intraclass correlation coefficients for the agreement among all reviewers with regard to the study type, level of evidence, and subcategory within the level of evidence ranged from 0.61 to 0.75. Reviewers trained in epidemiology demonstrated greater agreement (range in intraclass correlation coefficients, 0.99 to 1.0), across all aspects of the classification system, than did reviewers who were not trained in epidemiology (range in intraclass correlation coefficients, 0.60 to 0.75).

Conclusions: These findings suggest that epidemiology and non-epidemiology-trained reviewers can apply the levels-of-evidence guide to published studies with acceptable interobserver agreement. The validity of this system remains a question for future research.

Figures in this Article

Since January 2003, all clinical scientific articles published in the American volume of The Journal of Bone and Joint Surgery (JBJS-A) have included a level-of-evidence rating¹. Levels of evidence are hierarchical rating systems for classifying study quality. On the basis of a review of several existing evidence rating systems (), JBJS-A uses five levels for each of four different study types (therapeutic, prognostic, diagnostic test, and economic or decision-modeling studies).

Currently, all clinical papers are assigned an evidence rating by the Associate Editor for Evidence-Based Orthopaedics of JBJS-A. This rating of scientific articles in JBJS-A has several goals. The first is for authors, reviewers, and readers to become familiar with the concept of levels of evidence. Second, it is expected that studies will be improved by an explicit articulation of the primary research question, which is an important aspect of evidence-based medicine and is used by many organizations to develop practice guidelines. Third and most important, the levels-of-evidence ratings place a clinical research study into context for the reader. However, to be useful to readers and investigators, this newly proposed system of classification should be reliable—that is, different reviewers should be able to independently apply the level-of-evidence definitions to scientific articles and obtain the same results. The agreement among various reviewers using this rating system has not been evaluated, to our knowledge. Thus, the aim of the current study was to evaluate the interobserver agreement among reviewers, with varying levels of epidemiology training, in categorizing levels of evidence of clinical studies published in JBJS-A.

Materials and Methods

Materials and Methods | Results | Discussion | References

Eligibility Criteria

Fifty-one consecutive clinical papers published in the first six months of American Volume 85 (2003) of The Journal of Bone and Joint Surgery were identified by a computerized search of the table of contents of these six issues. Case reports, nonscientific clinical papers, and basic research papers were not included in this review. Papers were blinded by two investigators who removed the authors, citation, level-of-evidence designation in the abstract, and introduction, results, discussion, and acknowledgment sections. Thus, only the title, abstract, and methods section were ultimately provided to reviewers. The papers were coded and were randomly distributed in a study binder. The investigators who blinded the papers were not involved in the assessment of the studies.

Assessment of Levels of Evidence

Six surgeons participated in the study: two surgeons with a master's level degree in epidemiology and four non-epidemiology-trained surgeons. These surgeons practiced in four different academic centers in North America (two in the United States and two in Canada). Three surgeons were members of the JBJS American Editorial Board, two were reviewers for JBJS-A, and one was an active researcher not formally associated with JBJS-A. Each independent and blinded reviewer graded clinical papers for (1) the type of study (therapeutic, prognostic, diagnostic test, or economic or decision analysis), (2) the level of evidence (on a scale of I through V), and (3) the subcategory within the particular level of evidence (see Instructions to Authors in this issue). Reviewers did not receive any formal training in the application of the classification system, and there was no discussion among reviewers prior to their ratings. Each reviewer was provided with the classification system from the JBJS-A web site () with their binder of papers. Our decision to use only the information available on the JBJS-A web site was based on our aim to evaluate the reliability of the classification system in its current application. Furthermore, we believed that the classification system was sufficiently clear to allow reviewers to apply it without extensive training.

Data Analysis

Data were analyzed with the SPSS statistical software package (version 11.2; SPSS, Chicago, Illinois). Briefly, intraclass correlation coefficients with 95% confidence intervals for the type of study, level of evidence, and subcategory within the level of evidence were determined for comparisons of (1) all six reviewers, (2) two epidemiology-trained reviewers (one of whom [J.G.W.] is the Associate Editor for Evidence-Based Orthopaedics of JBJS-A), and (3) four non-epidemiology-trained reviewers. The intraclass correlation coefficient is a measure of concordance ranging from 0 to 1. We based our sample size for this reliability study on the number of reviewers, with use of the formula²: 2R², where R = the number of reviewers. Finally, we measured the absolute agreement between the ratings of the Associate Editor for Evidence-Based Orthopaedics and the five other reviewers' ratings.

Results

Materials and Methods | Results | Discussion | References

Of the fifty-one articles, 69% were studies of therapy. We did not identify any economic analyses in the sample. Only seven (14%) of the fifty-one articles represented the highest level of evidence (Level I), and the majority (twentynine; 57%) represented Level-IV evidence (Table I).

View Larger

TABLE I Characteristics of Included Papers (N = 51)

Type of Study	No.
Therapeutic	35 (69%)
Prognostic	13 (25%)
Diagnostic test	3 (6%)
Economic analysis	0 (0%)
Level of evidence
I	7 (14%)
II	10 (20%)
III	5 (10%)
IV	29 (57%)
V	0 (0%)

Add to My JBJS

TABLE I Characteristics of Included Papers (N = 51)

Type of Study	No.
Therapeutic	35 (69%)
Prognostic	13 (25%)
Diagnostic test	3 (6%)
Economic analysis	0 (0%)
Level of evidence
I	7 (14%)
II	10 (20%)
III	5 (10%)
IV	29 (57%)
V	0 (0%)

The intraclass correlation coefficients for the agreement among the reviewers with regard to the study type, level of evidence, and subcategory within the level of evidence ranged from 0.61 to 0.75. Absolute agreement between the Associate Editor's rating and the ratings of the other five reviewers ranged from 67% to 82.4%. The two reviewers with epidemiology training achieved complete agreement in categorizing the type of study. The reviewers without higher degrees in clinical epidemiology also achieved substantial agreement in their assessments of study type (Table II). The epidemiology-trained reviewers achieved nearly perfect agreement in categorizing both the level of evidence and the subgroup within the level (intraclass correlation coefficient = 0.99; 95% confidence interval = 0.98 to 1.0). The non-epidemiology-trained reviewers agreed to a lesser degree in their assessment of the levels of evidence (intraclass correlation coefficient = 0.60; 95% confidence interval = 0.47 to 0.72) and of the subcategory within the level (intraclass correlation coefficient = 0.71; 95% confidence interval = 0.60 to 0.80) (Table II).

View Larger

TABLE II Interobserver Agreement for Levels of Evidence^*

	Intraclass Correlation Coefficient (95% Confidence Interval)	Absolute Agreement†
Type of study
All reviewers (n = 6)	0.75 (0.66-0.83)	82%
Epidemiology-trained (n = 2)	1.0
Non-epidemiology-trained (n = 4)	0.75 (0.66-0.84)
Overall level of evidence
All reviewers (n = 6)	0.61 (0.50-0.72)	67%
Epidemiology-trained (n = 2)	0.99 (0.98-1.0)
Non-epidemiology-trained (n = 4)	0.60 (0.47-0.72)
Subcategory within level of evidence
All reviewers (n = 6)	0.71 (0.62-0.80)	69%
Epidemiology-trained (n = 2)	0.99 (0.98-1.0)
Non-epidemiology-trained (n = 4)	0.71 (0.60-0.80)

Six reviewers rated fifty-one papersThe values represent the percentage of papers about which five reviewers had complete agreement with the JBJS-A Associate Editor (J.G.W.)

Add to My JBJS

TABLE II Interobserver Agreement for Levels of Evidence^*

	Intraclass Correlation Coefficient (95% Confidence Interval)	Absolute Agreement†
Type of study
All reviewers (n = 6)	0.75 (0.66-0.83)	82%
Epidemiology-trained (n = 2)	1.0
Non-epidemiology-trained (n = 4)	0.75 (0.66-0.84)
Overall level of evidence
All reviewers (n = 6)	0.61 (0.50-0.72)	67%
Epidemiology-trained (n = 2)	0.99 (0.98-1.0)
Non-epidemiology-trained (n = 4)	0.60 (0.47-0.72)
Subcategory within level of evidence
All reviewers (n = 6)	0.71 (0.62-0.80)	69%
Epidemiology-trained (n = 2)	0.99 (0.98-1.0)
Non-epidemiology-trained (n = 4)	0.71 (0.60-0.80)

Six reviewers rated fifty-one papersThe values represent the percentage of papers about which five reviewers had complete agreement with the JBJS-A Associate Editor (J.G.W.)

Discussion

Materials and Methods | Results | Discussion | References

In our study of the reliability of a level-of-evidence classification system as used by six reviewers to rate fifty-one clinical papers from JBJS-A, the epidemiology-trained reviewers achieved greater levels of agreement than did the nonepidemiology-trained reviewers in the assessment of study type, level of evidence, and subcategory within the level.

Our study was strengthened by the inclusion of reviewers with and without epidemiology training, by the blinded review of papers, and by the inclusion of consecutive clinical papers in one journal over a six-month period. The study had some limitations as well. First, our results may not be generalizable to reviewers with less research experience or to the assessment of a different sample of articles. Second, our study was powered to evaluate the overall reliability of the levels-of-evidence categories but was underpowered to examine the reliability of each individual level. Third, the sample of articles did not include any economic analyses and included only a few studies about a diagnostic test. However, the distribution of articles probably represents the types of studies published in JBJS-A. Thus, our results suggest that, for the studies currently being published in JBJS-A, the levels-of-evidence system meets acceptable levels of reliability.

Among study designs, there exists a hierarchy of evidence, with randomized controlled trials at the top, controlled observational studies in the middle, and uncontrolled studies and opinion at the bottom. Sackett proposed a grading system that categorizes the hierarchy of research designs as levels of evidence³. The current system adopted by JBJS-A requires three important steps in the classification of a clinical study. First, reviewers must decide whether a study is investigating the results of a treatment, the outcome of a disease, a diagnostic test, or an economic model (or decision analysis). Second, they must categorize the level of evidence of the clinical study on a scale ranging from I (results of a randomized controlled trial or systematic review of randomized controlled trials) to V (expert opinion). Finally, reviewers must subclassify a clinical study within each level of evidence. For example, a Level-II therapeutic study can be rated as a prospective cohort study, a poor-quality randomized controlled trial (e.g., one with <80% follow-up), or a systematic review of Level-II studies or nonhomogeneous Level-I studies).

Understanding the association between study design and level of evidence is important. Level-I studies may be deemed appropriate for application to patient care, whereas Level-IV studies should be interpreted with caution. For example, readers should be more confident about the results of a high-quality multicenter randomized trial comparing the effects of femoral intramedullary nailing with and without reaming on pulmonary function and union rates (a Level-I study) than they should about the results of two separate case series involving the effects of femoral nailing either with or without reaming on the same outcomes (Level-IV studies). However, it is not possible to perform randomized clinical trials to address all clinical questions, and Level-I evidence may not be available. The breadth of orthopaedics, with many low-incidence conditions, presents challenges with regard to the performance of randomized trials. Thus, in the absence of Level-I evidence, Level-II, III, or IV evidence may be the only available information to guide clinical decision-making. Arguably, well-designed observational studies can provide results consistent with those of randomized trials⁴. Ultimately, an answer to a clinical question must be based on a composite assessment of all evidence of all types. No single study necessarily provides a definitive answer.

In order to be most useful to readers and investigators, the level-of-evidence grade on a scientific paper must be both reliable and valid. Our study suggests that investigators with training in epidemiology can achieve nearly perfect agreement. Reviewers without epidemiologic training may also be able to reliably apply the definitions of the various levels of evidence to certain types of clinical studies. We were encouraged by these results because the reviewers had not received any formal training in the application of the grading system. This finding suggests that the current requirement that authors report the level of evidence in the abstract of all clinical papers submitted to JBJS-A is acceptable. Whatever the intraclass correlation coefficient value, the levels-of-evidence classification system should be viewed in the context of the importance of the clinical question, the quality of the study methods, and the biologic plausibility of the results. This classification is not error proof. Reviewers did not achieve complete consensus regarding one-third of the studies. Readers must not use the level of evidence as an absolute in lieu of reading the methods and results sections of a particular paper.

Although JBJS-A requires that authors submit the level-of-evidence rating, all manuscripts are reviewed by an editor with epidemiology training (J.G.W.). Thus, the agreement between the two epidemiology-trained individuals is most relevant to readers who want to be sure that the ratings are reliable.

The validity of this hierarchical approach to study design has been based largely on face and content validity—that is, the levels-of-evidence classification system appears to be comprehensive in its approach, with a sensible rationale for the hierarchy (i.e., randomized trials introduce less bias than do nonrandomized study designs). Sackett and colleagues have been developing a hierarchy of evidence for the past twenty years. The current version of their levels-of-evidence criteria has been through multiple iterations of revision and is currently available online at the Oxford-Centre for Evidence Based Medicine. The JBJS-A levels-of-evidence grading system was modified for surgical clinical studies with the aim of improving clarity without altering the comprehensiveness of Sackett's proposed system. However, underlying the concept of levels of evidence is the assumption that the data are sound and have been interpreted appropriately. Generally, assignment of levels is based only on the reported study design and results and not on the quality of the data or interpretation. The system fails if the quality of reporting is flawed.

The levels-of-evidence system used by JBJS-A is a new Journal initiative. The Editorial Board expects continued feedback on the process, comments on the usefulness for orthopaedic surgeons, and debates about the ratings applied to individual studies. While readers and investigators may continue to debate the utility of this hierarchical approach, our findings suggest that reviewers can apply the guidelines to articles with acceptable agreement. The validity of this system remains a question for future research.

References

Materials and Methods | Results | Discussion | References

Wright JG, Swiontkowski MF. Introducing a new Journal section: evidencebased orthopaedics. J Bone Joint Surg Am.2000;82: 759-60.82759 2000

Streiner DL, Norman GR, editors.Health measurement scales: a practical guide to their development and use. 3rd ed. New York: Oxford University Press; 2003. 2003

Sackett DL. Rules of evidence and clinical recommendations on the use of antithrombotic agents. Chest.1986;89(2 Suppl): 2-3S.892 1986 [PubMed][CrossRef]

Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med.2000;342: 1887-92.3421887 2000 [PubMed][CrossRef]

Topics

levels of evidence ; procedure on joint

Username

Password

Access for Patients

Forgot Password?

Have a subscription to the print edition?

Not a Subscriber?

Buy this Article

Get online access for 30 days for $35

New to JBJS?

Subscribe to the online edition for 30 days for $75

Register for a FREE limited account to get full access to all CME activities, to comment on public articles, or to sign up for alerts.

Have a subscription to the print edition?

Current subscribers to The Journal of Bone & Joint Surgery in either the print or quarterly DVD formats receive free online access to JBJS.org.

Activate your online account now

Forgot your password?

Enter your username and email address. We'll send you a reminder to the email address on record.

Username:*

Email Address:*

Forgot your username or need assistance? Please contact customer service at subs@jbjs.org. If your access is provided
by your institution, please contact you librarian or administrator for username and password information. Institutional
administrators, to reset your institution's master username or password, please contact subs@jbjs.org

References

Accreditation Statement

These activities have been planned and implemented in accordance with the Essential Areas and policies of the Accreditation Council for Continuing Medical Education (ACCME) through the joint sponsorship of the American Academy of Orthopaedic Surgeons and The Journal of Bone and Joint Surgery, Inc. The American Academy of Orthopaedic Surgeons is accredited by the ACCME to provide continuing medical education for physicians.

CME Activities Associated with This Article

Submit a Comment

Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discretion of JBJS editorial staff.

* = Required Field

Comment Author(s) * (if multiple authors, separate names by comma)

Example: John Doe

Affiliation & Institution*

Disclosure

Comment Title*

Comment*

Cancel

Print
PDF
Email

Recipient(s) will receive an email with a link (good for 5 days) to 'Interobserver Agreement in the Application of Levels of Evidence to Scientific Papers in the American Volume of The Journal of Bone and Joint Surgery'.
Your Name:*
Example: John Doe

Email Address:* CC Me:
Enter your valid email address. Example: jdoe@example.com

Recipient's Email Address:*

Separate multiple email address with semi-colons (up to 5).

Subject:* 's The Journal of Bone & Joint Surgery: 'Interobserver Agreement in the Application of Levels of Evidence to Scientific Papers in the American Volume of The Journal of Bone and Joint Surgery'
Subject for your email.

Message:

(Optional, message will truncate at 1000 characters)

Processing your request... Please Wait...

Copyright © in the material you requested is held by The Journal of Bone & Joint Surgery, Inc. (unless otherwise noted). This email ability is provided as a courtesy, and by using it you agree that that you are requesting the material solely for personal, non-commercial use, and that it is subject to Journal of Bone & Joint Surgery, Inc.'s Terms of Use. The information provided in order to email this topic will not be used to send unsolicited email, nor will it be furnished to third parties. Please refer to The Journal of Bone & Joint Surgery Inc.'s Privacy Policy for further information.

Copyright © The Journal of Bone & Joint Surgery Inc. All rights reserved.
Share
Get Citation

Mohit Bhandari, Marc F. Swiontkowski, Thomas A. Einhorn, Paul TornettaIII, Emil H. Schemitsch, Pamela Leece, Sheila Sprague, James G. Wright; Interobserver Agreement in the Application of Levels of Evidence to Scientific Papers in the American Volume of The Journal of Bone and Joint Surgery. The Journal of Bone & Joint Surgery. 2004 Aug;86(8):1717-1720.

Download citation file:

RIS (Zotero)

EndNote

BibTex

Medlars

ProCite

RefWorks

Reference Manager

Copyright © The Journal of Bone and Joint Surgery, Inc.
Rights & Permissions
Article Alerts

Processing your request... Please Wait.
Submit a Comment