Prevalence and Impact of Self-Citation in Academic Orthopedic Surgery
- Updated on: 2018-04-11
- Read original article here
The h-index is a commonly utilized metric for academic productivity. Previous studies have proposed that self-citation may limit the utility of the h-index. The purpose of this study is to evaluate the impact of self-citation on the h-index among orthopedic investigators. The study cohort consisted of program directors, chairpersons, and faculty at orthopedic surgery residency programs in the United States. The Scopus database was used to determine the h-index and number of citations ± self-citations. The total number of publications was correlated with the change in the h-index via self-citation. A total of 463 researchers were included (198 National Institutes of Health-funded faculty, 147 chairpersons, 118 program directors). Of these researchers, 83.8% cited previous work at least once (mean, 123.9 ± 277.6). Self-citations accounted for 5.8% of all citations. Including these citations increased the author h-index from 18.5 ± 14.9 to 19.2 ± 15.6 (P< .001). A minority of researchers (36.3%, P< .001) had increased their h-index via self-citation (range, 0-11). The proportional increase in the h-index via self-citation was positively related to the number of publications (P< .001). While the practice of self-citation is prevalent in orthopedics, its impact on the h-index is minimal for most researchers. With more publications, researchers can increase their h-index to a greater degree via self-citation.
The competitive nature of academic research requires objective metrics to define career end points, such as promotion and funding procurement. While various criteria are used to assess performance in academia, publications and research funding are particularly regarded. Quantifying research dollars is relatively straightforward, but measuring research productivity is more complex. Not all articles are created equal, and disparities exist regarding effort and the ultimate impact of articles. In 2005, a physicist created the h-index to measure both research impact and productivity. As a bibliometric, the h-index equals the number of publications h that have been cited at least h times. Given its simplicity, the h-index has gained wide popularity in diverse medical specialties, including orthopedic surgery. Other recent studies have applied the h-index to hand surgery and spine surgery.
Importantly, some authors have raised concerns regarding potential limitations of the h-index. One potentially significant limitation is the ability of authors to artificially inflate their h-index via self-citation. The impact of this practice is of particular interest as the h-index becomes widely adopted as a metric for promotion at many academic institutions. Furthermore, scholarly productivity has remained a critical component of successful grant funding procurement, and future grant funding applications may evaluate the h-index.
The purpose of this study is to determine the prevalence and impact of self-citation on the h-index in a large cohort of orthopedic investigators. Given their high level of investment in academic orthopedic surgery, we focused on program directors, chairpersons, and National Institutes of Health (NIH)-funded research faculty at orthopedic surgery residency programs.
This study qualified as non-human and non-animal research and received exemption per the standing policy of the Institutional Review Board. The Fellowship and Residency Electronic Interactive Database (FREIDA) was accessed to generate a list of orthopedic residency program directors. This database was also used to generate a list of allopathic orthopedic surgery residency programs. Official program websites were accessed to generate a list of orthopedic chairpersons. Lastly, the NIH RePORTER was used to generate a list of basic science orthopedic investigators who received funding anytime during 2011 to 2014. This methodology was used due to the lack of reporting of basic science investigators on program websites. A list of NIH-funded orthopedic investigators was cross-referenced via an online search to isolate a cohort of PhD investigators.
Orthopedic faculty were defined as chairpersons, program directors, or NIH-funded investigators. In cases of overlap, preference was given in that order. Orthopedic investigators who had not published an article after 1995 were excluded (6 chairpersons, 1 program director).
While several resources exist to calculate the h-index, the Scopus database (Elsevier) is one of the easiest programs to use. Author entries are created via institutional affiliations, thereby alleviating the need for manual reconciliations. Investigators were identified on Scopus via “author last name” and “first name, middle initial.” For each author, publications were screened for relevance to the field of orthopedics. Affiliated institutions were cross-referenced with information obtained from individual program websites. The “view h-graph” feature was used to calculate the number of publications, h-index, and number of citations. Then, the “Exclude self-citations” feature was used to calculate the number of corrected citations and the h-index excluding self-citations. Metrics were calculated over a 2-day period.
Bibliometric analyses were presented descriptively with means ± standard deviation. After testing for normality, differences in the h-index between groups were assessed via analysis of variance tests. The proportional increase in the number of citations and the h-index were calculated by dividing the difference between the before and after exclusion of self-citations by the total number of citations before exclusion. The relationship between the number of publications and the proportional change in the h-index was assessed via calculation of the Spearman correlation coefficient. The independent variable was the number of publications, and the proportional increase in the h-index via self-citation was the dependent variable. Statistical tests were performed on STATA 13 (StataCorp) and the results considered significant if P< .05. Figures were created using GraphPad Prism 6.02 Software.
A total of 463 orthopedic investigators were included (147 chairpersons, 118 program directors, and 198 NIH-funded faculty) (Table 1). On average, these researchers produced 72.3 ± 83.0 articles and referenced 2139 ± 3222 articles (mean, 29.6 references per article). The cumulative h-index was 19.2 ± 15.6, and was the highest among NIH-funded researchers (24.3 ± 17.0) (P< .001). In all, 83.8% of orthopedic investigators self-cited their previous work at least once, and the total number of self-citations was highest among NIH-funded investigators (221 ± 355) (P< .001). After these self-citations were excluded, the h-index changed by 0.6 ± 1.1 for all investigators, and this change was greatest among NIH-funded researchers (1.1 ± 1.3) (P< .001).
Most orthopedic investigators did not increase their h-index via self-citation (63.7%, P< .001). Table 2 categorizes investigators by changes in their h-index after excluding self-citations (range, 0-11). The maximal change in the h-index was seen in the most prolific group of investigators, who produced 261.0 ± 149.3 articles. In this group, the h-index increased by 11.1% ± 5.2%. The Figureinvestigates the relationship between the number of articles and the proportional increase in the h-index. The number of publications was positively correlated with the change in h-index after self-citations were excluded (r = 0.448, P< .001).
The practice of self-citation is widely prevalent among experienced orthopedic investigators. However, this practice seems to have minimal impact on the h-index for most investigators. Self-citation had a measurable impact on the h-index only after an investigator had many publications. At a mean of 87.9 ± 68.3 articles, investigators had a ∆h-index of 1. This represented a mean 5.9% increase. Overall, these findings underscore the utility of the h-index in assessing scholarly impact and ameliorate concerns over bibliometric manipulation.
Among a large group of experienced orthopedic investigators, self-citation has minimal effect on the h-index. Importantly, most investigators (63.7%) did not experience a full integer increase in their h-index. At a threshold of ∆ h-index increase of 1, investigators had impressive h-indices (24.0 ± 13.3), which eclipsed those of recent studies of hand surgeons (10.2 ± 9.9) and spine surgeons (13.6 ± 8.7). This finding suggests that committees for academic promotion in orthopedic surgery may disregard the impact of self-citation on the h-index. While the thresholds for promotion have not been defined in the orthopedic literature, a study in plastic surgery established an h-index threshold of 14.5 for promotion from associate to full professor. It may be, however, that h-indices are higher among orthopedic surgeons, as a previous study reported an h-index of 20 among full professors. Future research is needed to determine thresholds for promotion within orthopedic surgery, as the h-index varies by specialty according to unique citation patterns.
It is worth highlighting the academic performance of NIH-funded PhD researchers in orthopedics. Even including training grant awardees in this group, this cohort exceeded the academic productivity of their orthopedic chairpersons, as measured by the h-index. Previous studies in urology, neurosurgery, and otolaryngology have demonstrated the impact of NIH-funding on academic productivity. Ultimately, orthopedic departments could increase academic productivity by recruiting more PhD investigators with NIH funding.
In contrast to academic radiology, this study demonstrated a correlation between the number of publications and the increase in h-index via self-citation. Several reasons may help explain this disparity. The first reason is a technical one, as at the time of this study, the Scopus database had been updated to include citations before 1996. Considering that the h-index increases over time as older publications are cited, the exclusion of older articles is a significant limitation of previous h-index studies. Applying the same logic, the mean h-index for chairpersons of 19.9 quoted in this study contradicts a recent study, which quoted a mean h-index of 15.3. This previous study utilized citations that were limited to articles published after 1996.
Previous work on self-citation in the field of orthopedics has been limited to its influence on journal impact factors. Our results build on this literature in several important ways. Firstly, the calculation of a journal’s impact factor is a highly scrutinized process, and authors have criticized the mechanisms employed by editors to inflate impact factors. One study reported that 20% of authors have been encouraged to cite a journal during the revision process. Self-citation practices have been demonstrated in journals of cardiology, diabetes, anesthesia, and medicine. A study using a hypothetical model to assess the maximum potential for h-index increase by self-citation demonstrated an h-index inflation of 5 points over 20 years (5/14, 35.7%) by publishing 3 papers per year with 3 self-citations each. This study highlights a potential limitation of the h-index, but our study observed an h-index inflation of ≥4 in only 11 researchers (2.4%). Thus, results from our study ameliorate self-citation concerns in academic orthopedic surgery.
There are several limitations to this study that offer future areas of research. First, the validity of the h-index academic promotion in orthopedic surgery has not been evaluated. This was a motivation for the present study, and the authors have ongoing efforts to characterize the h-index in a larger cohort of orthopedic investigators. Importantly, an appropriate amount of self-citation was not established. It may be necessary for orthopedic researchers to cite their works as they become experts on a specific topic. Lastly, our analyses are prone to limitations inherent in the h-index, which does not account for author contribution or journal impact factors. Despite these limitations, we show that for most orthopedic researchers, the practice of self-citation does not impact the h-index.
In summary, self-citation is a widely prevalent practice among orthopedic investigators, but this practice has minimal impact on an author’s h-index. Approximately one third of orthopedic faculty in our study had a higher h-index due to self-citation. Greater h-index inflation through self-citation correlated with more publications. For the majority of orthopedic faculty, however, self-citation did not inflate the h-index, suggesting that promotional committees may disregard this concern when using the h-index as an adjunct measure for career advancement.
