Evaluation based on scientific publishing: Other citation metrics

Other citation metrics

The Thomson Reuters Journal Impact Factor's greatest weaknesses are that it does not take into account the quality of the article being cited, nor does it take into account the different citing norms and practices of different disciplines. In order to address these weaknesses, various new journal citation indicators have been developed. The new indicators evaluate the quality of the citing, based on the quality of the journal that the citing article is published in. The evaluation is usually based on the PageRank algorithm that enables us to analyse the articles and the network that the citations form, and in this way analyse the relative value of the citation. Indicators based on the PageRank algorithm are not as straight forward and easy to understand as the IF, and they do not have the same well established position as the IF. However, new indicators are still being developed, and their status and usefulness in evaluation remain to be seen. On the other hand, they do draw attention to the many faults in evaluation, and they bring some competition to a field that has always been dominated by one commercial enterprise.

Article Influence, AI

The Article Influence score measures the average influence, per article, of the papers in a journal. As such, it is comparable to Thomson Scientific's Impact Factor. It is calculated by dividing the Eigenfactor score by the number of articles published. Article Influence scores are normalized so that the mean article in the entire Thomson Journal Citation Reports (JCR) database has an article influence of 1.00.

The Article Influence score is a journal specific Eigenfactor score. It is calculated by dividing the Eigenfactor score of the journal by the number of articles published. It tells the average impact of an article published in a journal from a five year period after the publication of the article. The Article Influence score is comparable with the IF of Thomson Reuters.

The Article Influence scores for journals are available at http://www.eigenfactor.org/. Thomson Reuter's Journal Citation Reports also publishes Article Influence scores in edition 2007 and later at http://www.isiknowledge.com/JCR.

Eigenfactor, EF

The Eigenfactor (EF) is an overall rating of the importance of a scientific journal whereby all articles published in a journal during a year are taken in to consideration when making the calculation. What counts in the Eigenfactor is the size of the journal, or how many articles a journal publishes. With all else equal, a journal's Eigenfactor score doubles when it doubles in size. The Eigenfactor is scaled so that all the publications that feature in the Journal Citation Reports (JCR) have a summed EF of one hundred. The calculation of the Eigenfactor for journals is based on the PageRank algorithm, which enables determining the value of the journal that is citing the article, and citations are given different weight according to how high the EF score of the journal is. The EF score is calculated by counting all citations from five years after the article was published, and not just citations from a specific category of journals. In addition to scientific journals, the EF takes into consideration citations in newspaper articles, doctoral theses and books etc. In this way, one can get a fairer estimation for journals situated in the middle ground between scientific disciplines. It also takes into account different citation norms which enhances the comparability of journals from different scientific areas.

The scholarly literature forms a vast network of academic papers connected to one another by citations in bibliographies and footnotes. The structure of this network reflects millions of decisions by individual scholars about which papers are important and relevant to their own work. Therefore within the structure of this network is a wealth of information about the relative influence of individual journals, and also about the patterns of relations among academic disciplines.

Borrowing methods from network theory, the Eigenfactor score ranks the influence of journals much as Google's PageRank algorithm ranks the influence of web pages. By this approach, journals are considered to be influential if they are cited often by other influential journals. Iterative ranking schemes of this type, known as eigenvector centrality methods, are notoriously sensitive to "dangling nodes" and "dangling clusters": nodes or groups of nodes which link seldom if at all to other parts of the network. Eigenfactor algorithm modifies the basic eigenvector centrality algorithm to overcome these problems and to better handle certain peculiarities of journal citation data.

The Eigenfactor ranking system accounts for difference in prestige among citing journals, such that citations from Nature or Cell are valued highly relative to citations from third-tier journals with narrower readership. The Eigenfactor score also adjusts for differences in citation patterns among disciplines.

The Eigenfactor score is additive: to find the Eigenfactor of a group of journals, simply sum the Eigenfactors of each journal in the group.

The Eigenfactor scores for journals are available at http://www.eigenfactor.org/. Thomson Reuter's Journal Citation Reports also publishes Eigenfactor scores in edition 2007 and later at http://www.isiknowledge.com/JCR.

Details of the calculation of the Eigenfactor score are available at http://www.eigenfactor.org/methods.pdf

A diagram of Eigenfactor metrics calculation.

Eigenfactor compared to Journal Impact Factor

  • The Eigenfactor ranking system accounts for difference in prestige among citing journals.
  • Like Thomson Scientific's Impact Factor metric, the Eigenfactor metrics measure the number of times that articles published during a census period provide citations to papers published during an earlier target window. While Impact Factor has a one year census period and uses the two previous years for the target window, the Eigenfactor metrics have a one year census period and use the five previous years for the target window.
  • The Eigenfactor score adjusts for differences in citation patterns among disciplines.
  • Self-citations of journals are ignored in Eigenfactor calculation.
  • The citation data used at Eigenfactor ranking come from Thomson Scientific's Journal Citation Reports (JCR) and in addition more than 100,000 reference items that are cited by the JCR-listed journals.

H-index for journals

A journal has an h-index of h if there is at least h papers published in that journal that have received at least h citations each.

The calculation of the H-index is not dependent on the defined citation window. If the H-index is calculated for the whole lifetime of a journal, it favours journals that have been established a long time ago. For most uses such a journal H-index does not seem useful. Instead, one must decide on the most suitable citation window. An advantage of using the H-index instead of the Impact Factor for evaluating journals is that the impact gap between multidisciplinary journals, such as Nature and Science, and other journals is significantly reduced making comparisons meaningful between these journals. Additionally, the impact of journals specializing in review articles is inherently deflated because of the limited number of annual articles in such journals.

The h-index is an alternative to the IF when evaluating the impact of journals based on citation information. It gives a rough estimate on the productivity of a journal over a long time period. The good qualities of the h-index, compared with IF, are its stability and its balanced combining of quantity and quality, which reduces the over-valuing of journals that publish review articles. Because the h-index is not based on the average number of article citations, it is not influenced by the fact that a few articles might have massive amounts of citations. However, the amount of articles a journal has published will affect somewhat the h-index score. A journal that publishes many articles will most likely get a higher h-index score than a journal that does not publish as much, because each article is a potential subject for citing. A journal that does not publish many articles will probably not get a high h-index score even if most of its articles were to be widely cited. The h-index is not calculated for one specific fixed time period like the IF, but it can be calculated for whatever time period you want, for example, for one or five years. This allows comparisons of journals that differ from each other in terms of how fast the journals start getting citations. However, even with the h-index, it is still preferable to compare journals from within the same discipline.

Publish or Perish calculates the h-index of journals using citation data from Google Scholar. This opens up the possibility of evaluating the impact of a journal that is not part of the Thomson Reuters' Web of Science database, and therefore has no IF score. Since Google Scholar has wider ranging reference data than the Web of Science, it is able to give a fairer evaluation to journals that are cited mainly in sources outside the Web of Science, like books and journals that are not included in the Web of Science. In addition the h-index for journals is available on the SCImago Journal & Country Rank website.