Evaluation based on scientific publishing: Evaluating researchers and research groups

Evaluating researchers and research groups

When evaluating the level of researchers and/or research groups on the basis of publishing activity, the most common measures are: the amount of publications, the amount citations and the quality of the publications used as publishing channels for the research. There are several tools for this purpose and some are discipline specific.
The esteem of an individual researcher can be also evaluated on the basis of other factors: the networking of the researcher, being a member of an advisory board, acting as an expert in peer reviewing, presentations and being a keynote speaker, and awards and recognition etc. Finding this information is much harder than the information mentioned in the previous paragraph, and sometimes the information is only available from the researcher's or research group's own website.


Research productivity

The simplest way to find out the amount of publications of a researcher is to search for publications in the most important databases to the discipline. When trying to establish the research activity of an individual, you should use the author search in the database. In order to evaluate research activity, it is very important to use as many databases as possible that include research from the person's subject area. In addition, finding monographs is often problematic because often they cannot be found in most databases.

In the example, the middle column contains the numbers of publications of researchers in Information Sciences from only the Web of Science. When the number of publications for the same researchers was checked in several databases (right hand column , the standing in the ranking list changed significantly for a number of researchers. For example, Hernon climbed from being 13th to 5th, and Dimotroff dropped from 3rd place to 9th. Here we can see that only after using a number of databases can we get reliable information on the research activity of researchers.
Image source: Meho Lokman I. & Spurgin Kristina M. (2005). Ranking the research productivity of library and information science faculty and schools: An evaluation of data sources and research methods. Journal of the American society for information science and technology 56(12): 1314-1331. 8.7.2008

Researchers being cited

By looking at the citings that researchers have received, we can evaluate the impact of researchers in their own academic field. When assessing the amount of citations, you should eliminate the researchers' references to their own research. Most databases make this very easy for you when you are conducting a search.

There are many different databases where you can establish how often researchers have been cited. The most significant ones are; the Web of Science, Scopus and Google Scholar. In addition, the Publish or Perish application, which uses the content of Google Scholar, is a diverse tool for charting and analysing citations researchers have received.

Particularly with the Web of Science, one should note where the author is in the author list of joint publications. If the author is only marked as second, third or later, the citations received before 1988 cannot be found, if the publication in question is not part of the journals that Web of Science follows. The citations can only be found if you know the first author of the publication, and you perform a search using that name.

In the Web of Science database it is worth searching for citations that a particular researcher gets using a cited reference search. In case there may be different ways of presenting the name, you should start from the most common ones and use, for example, Lajunen, L. Using the cited author index can also be useful as the picture demonstrates. Lajuner LHJ is in the author list because the erroneous spelling in some bibliography of Lauri Lajunen's name has been saved in the database.
Image source: Thomson Web of Knowledge (Web of Science) <http://scientific.thomson.com> 8.7.2008

The h-index

The h-index is an indicator used in academic publishing to measure the publishing activity of researchers and their scientific influence. The higher the h-index score the more the researcher in question has received citations from their publications. The researcher's h-index can be searched at least from the Web of Science and Scopus databases, and with the Publish and Perish program.

Read more about the h-index.

The Web of Science Citation Report tool tells us the score of the h-index with a green line. In the example, the searched researcher has 152 citations from the Web of Science. When the results are organised according to the number of citation, the amount publications and the number of citation meet at the 41st reference. This means this researcher's h-index is 41.
Source image: Thomson Web of Knowledge (Web of Science) <http://scientific.thomson.com> 19.12.2008

The g-index

The g-index was developed by Leo Egghe in 2006 and is an improvement of the h-index. Egghe thinks that the biggest fault of the h-index is that it does not take into account the real number of citations of each of the most cited publications, which can be considerable more than h. The g-index has not yet come into wide-spread use. However, the g-index scores can be searched with the Publish and Perish program.

Read more about the g-index.

The researcher's background and networks

When establishing the academic level or influence of a researcher, it is an essential part of the evaluation to examine the researcher's background and his/her networking with other researchers in the same field. The organisations or institutions where he/she has worked also tell us a lot about the networking of the researcher. In addition, we can gain insights into networking by finding out with whom the researcher has co-authored publications.

Often background information on researchers can be found from their webpages, if they have them, but some information can also be found with the help of databases. For example, the Scopus database is not only an excellent tool for finding out about publishing productivity and citation information, but also for finding out about the connections and networking of the researcher. The Scopus database automatically collects from its data on publications such information as all the co-authors, and it builds up history data on the organisations with which the researcher has been affiliated according to the publications. However, the information is only based on the publications in the Scopus database.

In the example, we have information from the Scopus database on Jorge E. Hirch, the developer of the h-index. The database can find 35 co-authors from 188 publications that the researcher has produced. In addition, from the data in Scopus we can see with which organisations Hirch has been associated.
Image source: Scopus <http://www.scopus.com> 13.12.2008

The quality of publications

Usually in all disciplines, the most important and respected publications in the field are known. The prestige value of a publication is usually enhanced when peer reviewing affects the editorial choice of the articles that are published. There are several different measures for evaluating the quality of publications. The most common is the Impact Factor.

Read more about the journal evaluation.

Read more about the measures used to evaluate the quality of the publications:

H-index and g-index scores can also be found for publications. In the case of publications, these scores can be searched with the Publish and Perish program, or using the SCImago Journal Ranking service for example.

Top lists

Help in evaluating researchers can be found from different top lists that have been compiled by many different services. They are often based on the amount of citations the researcher's publications have have received. Only the most productive and respected scholars are to be found in the top lists, therefore only rare researchers can be evaluated on the basis of these lists.
The following services provide different top lists on researchers::

  • ScienceWatch.com: ScienceWatch.com: not only contains the most cited researchers top lists, but also other data that can aid the evaluation, such as information about researchers, the publications, the organisations and even the countries.
  • HighlyCited.com is not a traditional top list, but it does contain only the most cited researchers in different disciplines.
  • CiteSeerX: CiteSeerX contains rankings of the most cited researchers in computer and information sciences.

Things to take in to consideration when evaluating research groups

Evaluating research groups, especially in Biosciences and Medicine, is more important than evaluating individual researchers, because almost all research is conducted in groups. When evaluating research groups, the measures are the same as when evaluating individual researchers: publishing productivity, the amount of citations from the publications and the quality of the publication that cites the research. In addition, the groups are evaluated with expert panels; for example, when they are applying for funding, for membership in separate programs in university consortia, research organisations and graduate schools, and in applications for the Centres of Excellence of the Academy of Finland.

Evaluating research groups differs from evaluating individual researchers because the indicator scores are not available directly from the databases. Caution and criticism should also be used when evaluating research groups and examining indicator scores, because many mistakes and misleading issues can accumulate around any one group. With indicators that are based on citation information, the citations of researchers in the group from fellow group members should be erased because, especially in a big publishing group, the number of citations can be huge. For example, calculating the h-index for a research group differs from the evaluation of individual researchers, where the index score is often calculated for the whole of the researcher's life (sometimes from a more restricted period). However, the h-index of research groups is often calculated from the last three to five years, so the quality assessment of the latest research produced by the groups can be as accurate as possible. Different indicators can give totally opposite evaluation results, so it is very important when evaluating research groups to use several indicators and supplement them with peer reviews. Research groups can also only be compared to groups in the same discipline, because the group sizes and numbers of publications vary in different disciplines. It is also necessary to separate out those groups that are conducting basic research and those groups that are developing applications from research; the latter group rarely publishes their research results in the form of journal articles.