Correlation between Altmetric Attention Score and Citation in the Urological Cancers Literature

Introduction

With tremendous advancements in life expectancy, the incidence of cancer-related deaths along with cancer treatment attempts has risen notably (1, 2). Urological cancers, mainly prostate, bladder, kidney, and testis cancers, are prevalent worldwide; however, their incidence and mortality vary among different countries (3, 4). Among men, apart from skin cancers, prostate cancer is the most prevalent cancer, with an annual incidence of 240,000 new cases in the United States (2). Likely, bladder cancer imposes a significant burden on the healthcare system; as shown in 2012, 14,880 deaths were attributed to bladder cancer in the United States (2, 5). Up to now, quite an amount of efforts have been made to address risk factors of urological cancers, and several of them, including, but not limited to, metabolic, behavioral, and environmental risk factors, have been identified (2).

Although traditionally, the impact of the journal that published the article and the number of citations were the main tools to measure the quality of the article, currently, due to progress and gaining popularity in new technology, new metrics have been introduced as means of evaluating the usage and the spread of scientific papers (6). Nowadays, with substantial improvements in new technologies such as wearable devices (7), big data (8), and cloud computing (9), desirable situations are provided for people to be aware, share, and analysis different aspects of cancer diseases. Moreover, medical staff can derive enormous benefits from evolving comprehensive information regarding, for instance, cancer pathogenesis and genetic features of cancer diseases (10). 

Altmetric, a supplement of bibliometric, traces the existence of a scientific paper on social platforms by calculating the mentions that a paper obtained through different media, including Twitter, Facebook, blogs, policy sources, news outlets, Wikipedia, Reddit, online videos, patents, and Google (11, 12). The Altmetric score calculates within an automated algorithm and indicates a weighted sum of attention for specific research output (11). To the best of our knowledge, there is no available study regarding the compassion between Altmetric score and citations in the urology cancer field. We found out that there is a knowledge gap concerning public attention to urological cancer articles. Hence, herein, we designed a study to evaluate whether there is an association between citation and Altmetric attention score (AAS) in the top 50 highly cited articles in the Cancer Urology Cancer field published in 2015. 

Methods

We aimed to define top-ranked articles in urology about the four most prevalent tumors, including prostate, bladder, renal/kidney, and testis. We assumed that each paper's five-year period was required to evaluate its penetration indexes into academic and public communities. Therefore, articles published in Scopus between January 2015 and December 2015 were included. Our search strategy was based on the following terms in the title or abstract: prostate, bladder, kidney or renal, or testis AND tumor or cancer or malignancy.

The search results were sorted according to citation count, and a full list of 50 top-ranked articles was obtained from Scopus. Then, the list was purified based on relevancy to the field of urology by two authors (M. AP and SS. T) independently. The study titles, abstracts, and full texts were checked and 23 irrelevant articles were excluded. Data extraction was performed by two authors (M. AP and SS. T). The first author, journal's name, article type, country, major topic, tumor type, AAS of paper, and citation in August 2020 were extracted. The two researchers identified article type (original, reviews, and note), major topic (epidemiology, treatment, molecular mechanism, and genetics, diagnosis and early detection, etiology and risk factors), tumor type (prostate, bladder, kidney, testis), and article access (non-open access vs. open access). The AAS was obtained from the Altmetric Bookmarklet tool downloaded from the website Altmetric.com. Information related to online attention to the publications (e.g., Twitter, Facebook, Mendeley, Etc.) was reported from this tool. The journals' impact factors (IF) and quartiles (Q) were collected from SJR (http://scimagojr.com). In addition, citation numbers were extracted from Scopus (HTTP: //www.Scopus. com).

Statistical analysis

We used the median and the ranges of values (minimum-maximum) to describe data. Study data that were not normally distributed and comparisons were made using a Kruskal-Wallis’s test to compare Altmetric scores among different categories. The Spearman rank correlation coefficient was used to describe the correlation between Altmetric attention scores and citation count or the number of mentions on Twitter. As well, the relationship between AAS and the journals' impact factors were assessed. Data analysis was performed in SPSS v.21.

Results

27 articles were included for analysis in the final list and were sorted in Table 1 according to their citation, and the articles' characteristics were reported. Among all, 77.7% were original articles, 18.5% were reviews, and one was a note. The source country of most of these top-ranked articles was North America (51.8%) and Europe (397%), and only three articles were from other geographic areas. The ten top most-cited manuscripts were original articles, and those focused on epidemiological issues or treatment advances had higher citation values. The mean value of citation scores for articles published originally from the USA, Europe, and other countries were 2029.21±4682.79, 710±313.48, and 434.33±66.42, respectively (data not shown).

Table 1. list of 27 most cited articles in Scopus, 2015 in relation to four urological cancers including prostate, kidney, bladder, and testis

Table 2 shows the AAS for the included articles. The highest AAS was for a note that was published "In Lancet Oncology" with the topic of "etiology and risk factors of prostate cancer". The second and third were two epidemiological articles about "global surveillance of cancer survival" and "The global burden of cancer" that were published in "The Lancet" and "JAMA Oncology", respectively. Table 2 shows the traditional science metrics in comparison with the online attention score. However, Figure 1a shows there was no correlation between citation and AAS (r=0.03, P-value=0.952). Figure 1 b shows that after eliminating two outlier articles from the study data, a statistically significant but weak correlation was observed between these indexes (r=0.56, P-value=0.003). Twitter had a significant role in online public attention, and there was a significant correlation between AAS and Twitter (r=0.79, P-value<0.001; Figure 2).

Table 2. Altmetric Attention Score and Citation for the most cited articles        

*ID: Citation rank; **AAS/C: Altmetric Attention Score (ASS)/ Citation

Figure1a. Relationship between citations and AAS for all papers

Figure1b. Relationship between citations and AAS for 26 papers (excluding the 2 outliers: 1 & 12)

Figure 2. Relationship between AAS and mentions on Tweeters in all papers

Table 3 shows the median value of citation, AAS, mentioned in Twitter, and almost all other evaluated indexes were higher in open-access journals. However, the estimated values did not show a statistically significant difference (P-values > 0.05). All article sources were Q1 journals, except for JAMA Oncology which was not in the SJR Q ranking in 2015. In addition, the two Chinese and Indian journals were Q3. The journal's impact factor in publication date ranged from 0 to 100.1, shifting towards 1.1 to 223.6 in 2019 (Table 4). The correlation between AAS and journals' impact factor in 2015 vs. 2019 is demonstrated in Figures 3a and 3b. There was only a very weak correlation between these factors. However, these findings were not statistically significant (respectively; r=.11, P-value=.557; r=.08, P-value=.727).

Figure 3a. Relationship between journals’ impact factor (in 2015) and AAS

Figure 3b. Relationship between journals’ impact factor (in 2019) and AAS

Table 3. Comparison of science metric and altimetric indexes according to journals’ accessibility status

Table 4. Comparison of article sources according to the type of journals

*Number of study articles published in each journal

Finally, we compared the median value of AAS in different categories. AAS in open-access journals (Median=413) was relatively higher than in non-open-access journals (Median=120.5). AAS in review articles was significantly lower than in other types of articles. The median value of AAS in North American countries was higher than in European and Asian countries. The AAS for Asian countries was higher than European ones. However, the reported ranges were too broad, and in one article were only ten. The ASS reported for each article in Table 2 was relatively higher in American publications. Considering both the median and range of values in Table 5, epidemiological articles had the highest online attention score.

Table 5. Factors associated with Altmetric Attention Score in 28 articles

Discussion

To our knowledge, no study so far has been conducted to explore the association between AAS and citation scores in the urological cancer field. Expectedly, in the current study, following removing two outlier articles, a weak positive association between AAS and Scopus citation was found. According to the findings of Eabhann et al.,'s study that was performed on 100 urological articles with the highest Altmetric scores published in 2014 and 2015, a significant correlation was found between Altmetric score and the number of citations per article and the impact factor of the journal (13). Likely, this positive correlation was replicated by Calopedose et al., among 22 urological articles (14). Currently, journals’ impact factor neither in 2015 nor in 2019 had a significant association with AAS. Other studies in oral cancer (15), nursing (16), radiology (17), and urology (6) fields reached the same result in this regard. This point highlights that the AAS of the article does not merely depend on the journal's impact factor, and other factors may have a role in gaining public attention.

Although the study of Torre et al. had the highest citation counts, it ranked 11th in terms of AAS in the present study. This original article which was published in "CA: A Cancer Journal for Clinicians" journal with an impact factor of 223.67, investigated global several malignancies including prostate, bladder, kidney, and testis, scale and profiles as well as preventive measures. At last, they concluded that the incidence of malignancies because of the increase in life expectancy and worldwide population and carcinogenic behaviors, including tobacco usage, is sustainably rising (18). The study had the highest AAS, whereas was discussed the relationship between red meat consumption and different malignancies (19). The article with the highest AAS was "Carcinogenicity of consumption of red and processed meat". This article made brief reports on the potential of red and processed meat as risk factors for malignancies, especially prostate cancer. Based on the article's topic, it is reasonable that the article had considerable popularity among people.

There were no significant associations between AAS and study type, topic, type of tumor, and geographic distribution in the present study. In the current study, most enrolled articles (approximately 78%) were original articles, and the review articles were placed in the second rank with a rate of 36%, which is similar to Mainwaring et al.'s study. In their study, 79% of bladder cancer articles with the highest citation were original, and 14% were reviewed, of which 8% were the review manuscript, and 6% were the meta-analyses manuscript (20). AAS and citation counts were higher in open-access journals than in non-open-access journals, but these differences were insignificant. Likewise, Hassona et al., claimed no significant difference between open and non-open access journals in terms of AAS (15). Maggio et al., demonstrated a positive association between online attention and public accessibility of the articles with health profession education content (21). Virtually all of the articles were published in Q1 journals. Highly discussed topics in the included study were molecular mechanisms and genetics (about 30%). Among four types of tumors, prostate cancer was of interest to the majority of studies. We thought this was not surprising because of the higher burden and prevalence of prostate cancer than other types.    

Despite the many benefits of Altmetric, it encompasses a variety of problems that should be paid attention to; First, Altmetric measures the quantitative influence of an article in social media and would not measure the quality of research outcomes or influences. The method for calculating the Altmetric score is designed based on the beliefs of the developer of this method for each online. Bornmann et al. demonstrated that Mendeley and Twitter mentions have a superior association with citations; thus, they may have different contributions to the Altmetric score. Likely, in the present study, the impact of Twitter on social attention was considerable. Second, Altmetric scores can be manipulated conveniently by using fake Twitter accounts or robots. Therefore, may Altmetric scores not be fully trustable (22-25). However, in the present study, Altmetric scores were calculated using Altmetric explorer which is less prone to manipulation among different means for measuring Altmetric. Third, due to discrepancies in penetration and usage of the Internet worldwide, Altmetric essentially does not indicate the online attention of the whole country (26). Finally, there is likely a bias in outcomes of Altmetric, which stems from higher rates of social media usage by younger individuals or authors than older ones. Nevertheless, Zhou et al. showed that the authors' social media size is conversely associated with the authors' popularity (27).

As substantiated progress in the use of social media in the urology field is occurring, several top journals of urology, mainly BJU international and European Urology, have selected editors of social media to expand their online existence (14). These activities confer apparent benefits, including intensifying the influence of social media in academia and leading young researchers and urologists to be aware of social media, which plays a crucial role in many training programs (28-30). 

We acknowledge that our study had some strengths:

1. As far as we know, no study has evaluated the association between Altmetric scores and citation counts in highly cited urology articles in the urology cancer field of one specific year.
2. In the present study, we examined articles that were published in 2015. It is not plausible to compare articles of different years because articles published earlier have a higher chance of gaining more citations and Altmetric scores than those published later [13].
3. We did not limit ourselves to journals that publish only urology subjects, and thus, we did not miss articles in the urology cancer field that were published in non-urology journals.
4. We performed the study on the articles published in 2015.

It is important to note that Altmetric can measure the impact of literature with higher velocity than traditional citation scores [6]. Since approximately five years passed from the included articles' publication date, the time for articles to gain enough citations was enough; therefore, this led to not overestimate the impact of Altmetric. 

Limitations:  

1. We only used keywords in English since the majority of the research was published in English.
2. For detecting articles with higher citation numbers, we only used the Scopus database. 

Conclusion

For the first time, the present study examined the online attention toward urology cancer field articles with the highest citation number utilizing Altmetric data. The article with the highest AAS was not highly cited, and the article with the highest citation number had not the highest AAS. Moreover, we demonstrated that while there is a weak positive association between citation number and AAS, no significant association was observed between AAS and journals' impact factor, study type, study topic, articles access, type of tumor, and geographic distribution. We think the article's impact will be assessed by a combination of altimetric and citation numbers in the future.

Declarations 

Author’s contributions

All authors contributed equally.

Acknowledgments

Thanks to the Urology Research Center, Tehran  University of Medical Sciences, Tehran, Iran.

Conflict of Interest

The authors declare that there is no conflict of interest.

Funding

No funding.

Ethical statement

Not applicable.

Data Availability

Data will be provided on request.

Abbreviations

AAS      Altmetric attention score

IF          Impact factors

Q          Quartiles