Background: The Hirsch index (h-index), widely considered a valuable measure of assessing academic productivity, has been studied in various medical and surgical specialties and has shown strong associations between higher h-indices and academic promotion, as well as with National Institutes of Health (NIH) awards. Additionally, the m-index and e-index may complement the h-index in this assessment of merit. The purpose of this study was to investigate the relationship between the h, m, and e-indices and academic rank for 2,061 academic orthopaedic surgeons in the United States.
Methods: The h-indices of faculty members from 120 academic orthopaedic surgery residency programs were organized and calculated using the Scopus and Google Scholar databases. Additionally, m-index and e-index scores were calculated from Google Scholar. After application of exclusion criteria, 2,061 academic orthopaedic surgeons were included.
Results: Academic rank (assistant professor, associate professor, professor, and chair) increases as mean h-index, m-index, and e-index scores increase. Among 976 assistant professors, 504 associate professors, 461 professors, and 120 chairs, mean h, m, and e-indices increased with each academic rank. In the comparison of male and female surgeons, there was no significant difference in h, m, or e-index scores, with the exception of increased h-index scores among male assistant professors.
Conclusions: Scholarly impact, as defined by academic productivity and scientific relevance, can be classified by the h-index and supplemented by the m and e-indices. This study has revealed well-defined differences in h, m, and e-indices with regard to academic rank among orthopaedic surgeons. Although the h, m, and e-indices may be of value as adjunct assessment devices for scholarly merit, careful consideration of their limitations must be maintained.
Academic committees assigned to the service of faculty recruitment and advancement within medicine will consider a candidate’s total scholarly impact based on contributions to medical education, clinical performance, and scholarly activity1,2. However, evidence exists that a candidate’s published research activity is a factor of principal importance when being considered for such a promotion3. While the notion of quantifying the importance of an author’s academic contributions has been widely accepted, methods to quantify such measures have been debated.
The concept of analyzing bibliometric indices to evaluate an individual’s value to a department was first introduced in 1955 when Garfield proposed the journal impact factor as a metric to quantify the relative importance of published works4. A number of other subjective bibliometrics were later developed to analyze candidates seeking advancement within their fields. Such measures took into account the raw number of publications, the number of citations for each published manuscript, and in which journal such scholarly works were published. While providing general assessments of each candidate, these statistics did not allow for objective comparisons. Such methods of peer review for appraisal of merit have generated much concern and highlighted the need for more objective means to measure an individual’s scholarly impact upon his or her field5.
It was in this setting that physicist J.E. Hirsch first described the h-index in 2005 as an unbiased metric to mathematically adjust for both the quantity and quality of an author’s work6. Succinctly, the number of published papers (h) that have attained at least h citations each, excluding papers cited fewer than h times, is used to express an author’s h-index. For example, an author who has published 5 papers that attained at least 5 citations each would have an h-index of 5. In this capacity, the h-index serves as a single statistic to identify potential scholarly success in candidates seeking promotions within their department7. Additionally, the m-index, also proposed by Hirsch6, divides the h-index by the span of active publication years, which may provide a measure for comparing academic productivity among younger and older surgeons.
Although the h-index may have considerable value in assessing academic productivity, it lacks a mechanism to measure authors who may publish less frequently but have highly impactful articles. For example, an author with 5 publications with only 5 citations per publication has the same score as an author with 5 publications with over 100 citations per publication. The e-index, proposed by Zhang8, complements the h-index in measuring citations greater than h received by papers in the h-core, and is defined as all of an individual’s publications having at least h citations.
Since its introduction, the h-index has been widely studied in various medical and surgical specialties, including otolaryngology9, neurological surgery10, radiation oncology11, and ophthalmology12. The vast majority of these studies found strong associations between higher h-indices and academic promotions as well as National Institutes of Health (NIH) awards. To the best of our knowledge, there has not been such a study within academic orthopaedic surgery. The purpose of the study described herein was to investigate the relationship between the h, m, and e-indices and academic rank for 2,061 academic orthopaedic surgeons in the United States.
Materials and Methods
A list of orthopaedic surgery residency programs in the United States was generated using the American Medical Association’s FREIDA residency program database13. A search of the web site of each academic department was performed in October 2015 for information regarding faculty, including academic rank. Military programs, programs not associated with a medical school, and programs that were started <2 years prior to 2015 were excluded from this analysis. Furthermore, nonphysicians, nonacademic employees, and part-time or volunteer faculty were omitted from this analysis. After application of exclusion criteria, a total of 2,061 academic orthopaedic surgeons from 120 programs were included.
The calculation of an individual’s h-index as well as publication range (in years) was obtained from the Scopus abstract and citation database and from Google Scholar. The Scopus database, which requires an access fee, covers over 33 million publications recorded from almost 18,500 peer-reviewed journals from 1996 to 2015, and has been shown to cover a broader range of articles than other databases such as Web of Science. Although the Scopus database recently has begun a large effort to index articles prior to 1996, this process is not yet complete. In consideration of evaluating a database that is both free to use and contains a greater number of articles published prior to 1996, Google Scholar was used to calculate an individual author’s h-index, as well as the e-index and the m-index. Google Scholar was chosen over Scopus for the calculation of the m-index because of the potential for inaccurate calculation of publication-years using Scopus14. Similarly, Google Scholar was chosen over Scopus for the calculation of the e-index because of its comprehensive nature. Understandably, as a result of differences between databases, considerable variability in h-indices has been demonstrated previously15, and this variability will be analyzed in this study. In order to ensure the accuracy of the initial search for an individual surgeon within the Scopus and Google Scholar databases, departmental affiliations and the presence of orthopaedic surgery or orthopaedic surgery-related journals in each database were taken into account. Data collection was completed in October 2015.
The academic rank for each orthopaedic surgeon was categorized as assistant professor, associate professor, professor, or chair. Surgeons in the “chair” category were not counted in the “professor” category to avoid duplication. Mean h-index, m-index, and e-index values for each academic rank, as well as a comparison of mean scores between male and female surgeons within the 4 aforementioned academic rankings, were calculated, and significance was determined using a 2-sample t test and a 1-way analysis of variance (ANOVA). Significance was set to p < 0.05.
The h-indices of 2,061 academic orthopaedic surgeons from 120 programs were calculated. The study group included 976 (843 male, 133 female) assistant professors, 504 (459 male, 45 female) associate professors, 461 (442 male, 19 female) professors, and 120 (120 male) chairs. Mean (± standard deviation [SD]) h-indices calculated from Scopus were as follows: assistant professors, 3.6 ± 4.5; associate professors, 8.4 ± 6.7; professors, 15.1 ± 11.1; and chairs, 17.8 ± 13.7 (Fig. 1-A). Associate professors attained higher mean h-index values than assistant professors (p < 0.0001), professors attained higher mean h-index values than associate professors (p < 0.0001), and chairs attained higher mean h-index values than professors (p < 0.05). The mean (± SD) h-indices calculated from Google Scholar were as follows: assistant professors, 4.4 ± 4.3; associate professors, 10.5 ± 7.0; professors, 21.0 ± 13.6; and chairs, 25.7 ± 17.5. Similar to the data from Scopus, h-indices calculated from Google Scholar showed that associate professors attained higher mean h-index values than assistant professors (p < 0.0001), professors attained higher mean h-index values than associate professors (p < 0.0001), and chairs attained higher mean h-index values than professors (p < 0.01) (Fig. 1-B). When h-index values were compared according to academic rank between the 2 databases, Google Scholar values were significantly higher for all 4 academic ranks (assistant professors, p < 0.01; associate professors, p < 0.0001; professors, p < 0.0001; and chairs, p < 0.001).
In the comparison of male and female surgeons, male assistant professors had significantly greater mean h-index values than female assistant professors in both Scopus (3.8 ± 4.7 compared with 2.8 ± 3.5, respectively; p = 0.015) (Fig. 2-A) and Google Scholar (4.5 ± 4.3 compared with 3.4 ± 3.5, respectively; p = 0.006) (Fig. 2-B). There was no significant difference at the associate professor level in either Scopus (8.6 ± 6.9 compared with 6.5 ± 4.9, respectively; p = 0.055) or Google Scholar (10.6 ± 7.1 compared with 8.6 ± 6.2, respectively; p = 0.063) values. There was also no significant difference at the professor level in either Scopus (15.1 ± 11.2 compared with 14.6 ± 7.0, respectively; p = 0.821) or Google Scholar (21.3 ± 13.8 compared with 17.2 ± 9.2, respectively; p = 0.192) values. There were no female chairs, so we were unable to compare male and female surgeons at that level.
The m-indices were compared between academic levels (Fig. 3-A). Associate professors had higher m-index values than assistant professors (0.7 ± 0.6 compared with 0.5 ± 0.6, p < 0.0001), professors had higher m-indices than associate professors (0.8 ± 0.6 compared with 0.7 ± 0.6, p < 0.001), and chairs had higher m-indices than professors (1.0 ± 0.7 compared with 0.8 ± 0.6, p < 0.05). There were no significant differences in m-index values between male and female surgeons at the assistant professor (p = 0.698), associate professor (p = 0.347), or professor (p = 0.364) level.
Additionally, comparisons of e-indices were made between academic levels (Fig. 3-B). Associate professors had higher e-indices than assistant professors (18.3 ± 9.8 compared with 8.2 ± 7.9, p < 0.0001), professors had higher e-indices than associate professors (31.3 ± 17.1 compared with 18.3 ± 9.8, p < 0.0001), and chairs had higher e-indices than professors (37.4 ± 21.3 compared with 31.3 ± 17.1, p < 0.01). There was no significant difference in e-index values between male and female surgeons at the assistant professor (p = 0.051), associate professor (p = 0.052), or professor (p = 0.081) level.
Since first developed by Hirsch, the h-index has been used in numerous scientific disciplines and has gained widespread acceptance as a valid bibliometric for approximating an individual’s true research impact5,7. Its use in decisions regarding tenure and academic advancement, however, has only recently been documented in the field of medicine16. This study investigates the correlation between academic rank and the h-index, the e-index, and the m-index within the field of orthopaedic surgery. By using these statistics to objectively gauge productivity and relevance of published works, we hope to establish a foundation for use within the orthopaedic surgery field by which faculty members can assess the impact of their own publications relative to that of their contemporaries.
Citation analysis studies within the past decade have suggested that the h-index is able to account for the significance of an author’s work as a function of research productivity17,18. Several studies have further shown positive correlations between the h-index and academic standing19. The results of the present investigation demonstrate that the h-index as well as the m-index and the e-index are indicators of academic advancement and impact of publication within the field of academic orthopaedic surgery. This is demonstrated by significant increases in h-indices, m-indices, and e-indices among orthopaedic faculty (n = 2,061) from the ranks of assistant professor through chair. The value of this finding lies in the implication that fair and objective reasons, based on scholarly merit, have been considered in decisions regarding advancement within the field of academic orthopaedic surgery.
Although Martinez et al.20 showed no differences in h-indices between male and female musculoskeletal oncology surgeons, in this current study, female assistant professors had lower h-indices than their male counterparts. Studies examining disparities in academic productivity between the sexes across several medical and surgical specialties have documented that female physicians early in their career have lower h-indices than their male counterparts, but often match the level of academic productivity of their male colleagues later in their careers21-23. This may explain the disparity in h-indices between male and female assistant professors in orthopaedic surgery noted earlier in their careers. It is also important to note that there were no differences in the m-index (h-index divided by years of publishing) in this group, suggesting that female surgeons are similarly productive in terms of academic publications, but are so later in their careers.
Despite the value of the h-index as an objective measure in academia, its limitations should be considered carefully. For instance, the Matthew effect24, described as the tendency of renowned authors to receive a disparate share of citations as compared with less extensively known authors, has the potential to bias the h, m, and e-indices of the former but not the latter authors. In addition, a highly contested article receiving negative criticism may generate a large number of citations, potentially skewing the h, m, and e-indices. In this scenario, the inability of these indices to differentiate a highly lauded article from a highly controversial one has the potential to limit their value if measures are not taken to control for these confounders. Extensive work has been done in this area, leading to strict guidelines in an effort to recognize and resolve such limitations25. Rigorous authorship recommendations are now commonplace in peer-reviewed journals in order to decrease the practice of “gratuitous authorship” and the impact of several coauthors receiving equal credit on the h-index and other bibliometric analyses. A recent study examining neurosurgical faculty in the United States, however, suggested no significant differences in the h-indices of faculty when controlling for authorship positions10.
Another widely cited criticism regarding the h-index involves the problem of deliberate self-citation and the role it may play in falsely inflating the h-index26. Although authors with elevated h-indices may or may not have engaged in this practice, it is unsustainable, as these authors may already have extensively referenced articles, and thus the impact of self-citation on their ultimate h-index is minimal27. For authors with modest h-indices, however, preferentially citing papers on the verge of counting for their h-indices can more easily lead to biased results27. This dilemma was addressed in a study that excluded all self-citations when calculating the h-index, yet found no significant difference in the h-index of the authors28. We did not evaluate self-citation in our analyses.
While h, m, and e-indices have been shown to accurately measure an academic physician’s research productivity5,9-11,29, numerous other factors may contribute to academic advancement, including grant support and the educational activities of faculty. Boyer29 described a shift in emphasis to research over teaching duties with regard to academic advancement beginning in the 1950s, and defined 3 other important areas of scholarship, including scholarship of teaching, scholarship of application, and scholarship of integration. Simpson et al.30 discussed the challenges in overcoming a growing consensus that teaching took time away from research, which is considered to be the true pathway to academic promotion at their institution. Additionally, several articles have supported the concept that faculty members should be evaluated on more factors than just research productivity, and have highlighted areas for improvement to achieve these goals31,32. In a survey of 115 promotion committee chairpersons, respondents rated teaching skills, clinical skills, and mentoring as the 3 most important aspects of clinician-educator skills33.
Aside from the general limitations associated with the use of the h, m, and e-indices to determine the academic merit of an orthopaedic surgeon, there are several specific limitations to this study. First, while our methods section details the stringent process used to determine bibliometric data regarding a surgeon’s academic publications, there is a potential for inaccuracies if authors have changed their name after marriage. This might explain a trend of lower h, m, and e-indices among female surgeons compared with male surgeons at the same academic rank. Next, if an author has changed institutions 1 or several times, citation information could get lost; this is particularly important with the Scopus database, where the institution of an author is specifically listed in their user profile. Finally, if an author shares the same first, middle, and last name of another author in a similar field, this could lead to potential inaccuracies in their bibliometric data. In our study, however, no orthopaedic surgeons shared the same full name.
In conclusion, scholarly impact, as defined by academic productivity and scientific relevance, can be classified by the h-index and supplemented by the m and e-indices. This study has revealed well-defined differences in the h, m, and e-indices with regard to academic standing among orthopaedic surgeons. Although the h-index has been proven to be of value as an adjunct assessment device for scholarly merit, careful consideration of its limitations is important. Impartial decision-making regarding promotion and recruitment for academic orthopaedic surgery departments may benefit from the utilization of bibliometrics such as the h, m, and e-indices.
Investigation performed at the Departments of Orthopaedics and Otolaryngology-Head & Neck Surgery, Rutgers New Jersey Medical School, Newark, New Jersey
Disclosure: The Disclosure of Potential Conflicts of Interest forms are provided with the online version of the article.
- Copyright © 2017 by The Journal of Bone and Joint Surgery, Incorporated