Responsible use of publication metrics at the University of Oulu

Analysis of researchers and research groups

In recent years, both national and international recommendations have emerged on how to evaluate researchers, prioritising qualitative evaluation. In line with the principles of responsible researcher evaluation, publication metrics can only be used to support qualitative evaluation.

When analysing the publication outputs of researchers and research groups, depending on the need, almost any of the available metrics can be used. Most indicators describe one characteristic of a researcher's or research group's publication outputs from a certain perspective. For this reason, the use of several indicators is recommended in order to obtain diversified information about publishing activitiy.

Especially in evaluation situations, it is recommended to use as varied a set of different indicators as possible, and even then only to support qualitative evaluation. The list of indicators used by bibliometric services to support recruitment can be used as an aid in the selection of metrics. If the intention is to produce publication metrics independently in the evaluation of individual researchers, it is recommended to consult bibliometric services before performing the analyses. Without a separate request, bibliometric services carry out analyses of all applicants for tenure track and professorship positions at the University of Oulu.

For more information on the subject, see the Finnish national guide to publication metrics

Keep in mind

Journal-based metrics, such as Journal impact factor or Publication Forum classification, should not be used as a substitute for measuring the quality of an individual research article when evaluating an individual researcher.

In fields of science where a lot of research is carried out in groups, it may make sense to examine the group rather than individual researchers.

Analysis of universities and research institutes

Universities and research institutes are analysed for many different reasons and by many different parties. Since the University Act alone obliges Finnish universities to evaluate their research regularly, and thus the international overall evaluation of the university's research (Research Assessment Exercise - RAE) is carried out every six years. The RAE evaluation also includes the evaluation of the university's publication outputs by means of bibliometrics. The latest RAE2020 final report is available on Patio.

For twenty years, universities have also been compared in international university rankings. The most well-known of these are the Academic Ranking of World Universities (ARWU), Times Higher Education (THE) and the QS ranking. University rankings have received a lot of criticism in recent years, for example due to the incompatibility of the indicators used in the evaluations and the lack of transparency of the methods. Despite the various problems with the rankings, universities monitor them closely and report on their own rankings. More information on the subject can be found on the University of Oulu's university rankings web page.

Analysing research organisations may also be necessary for other parties besides the university administration.Leaders of a research groups and even individual researchers may find it necessary to perform various collaboration or benchmarking reviews, often based on publication data. Extensive citation databases such as Scopus or its analysis tool SciVal, which offers ready-made tools for carrying out the analyses, are best suited for such analyses.

For more information on the subject, see the Finnish national guide to publication metrics

Analysis of fields of science and research topics

Publication metrics are used in the analyses of fields of science in several contexts. For example, publication metrics can be used to observe changes in science or to describe the state of a field of science at a certain moment. Many national evaluations of fields of science use metrics to evaluate, for example, the number of publications, citation impact and interdisciplinarity in the field, alongside peer review and various statistical sources. Examples of such use of publication metrics are the various reports and databases published by research funders or similar organisations for science policy purposes (in Finland, for example, the State of scientific research in Finland reports carried out by the Academy of Finland and the Vipunen service’s bibliometric reports based on the Scopus (only in Finnish) and Web of Science (only in Finnish) databases).

You can also analyse scientific fields and research topics yourself, and the easiest way to do this is with the SciVal analysis tool, which utilises data from the Scopus database. The 96,000 research topics and 1,500 topic clusters identified through citation analysis provide the opportunity to analyse subject areas in SciVal. With the help of the feature, it is possible to examine, for each research topic, e.g. which researchers or organisations have published on the topic or which publication channels have been used. For more information on the SciVal analysis tool, see the the Finnish national guide to publication metrics

When analysing fields of science and research topics it is important to consider the differences in publishing and citation practices across different fields of science. 

Key differences include variations in the prevalence of co-publications and the average number of authors per publication, as well as differences in average publication speed and citation numbers. There are also differences between fields of science in how well the citation databases used in the analyses cover the publication outputs.

Analysis of publication channels

There are multiple reasons for evaluating publication channels. In Finland, the analysis of publication channels is central even at the national level, as the Publication Forum classification implemented for the quality assessment of publication channels is used in connection with the distribution of funding by the Ministry of Education and Culture to universities. The key benefit of the Publication Forum is its coverage. The classification takes into account the special features of the publishing practices of different disciplines. The level classification of publication channels includes journals, book series, conferences, and book publishers. The publication forum portal aims to cover all peer-reviewed publication channels used in Finnish research organizations as well as scientific publishers. For more information on the Publication Forum classification in the Finnish national guide to publication metrics.

From the point of view of publication metrics, the analysis of publication channels focuses on scientific journals. This is partly due the extensive coverage of the metrics available on them, especially ones based on citation numbers. However, the Finnish national guide to publication metrics also briefly describes the analysis of articles, books and conference publications (see links below).

More information on the subject:
Analysing journals in the Finnish national guide to publication metrics.
Analysing articles in the Finnish national guide to publication metrics.
Analysing books in the Finnish national guide to publication metrics.
Analysing conferences in the Finnish national guide to publication metrics.
Indicators for evaluating journals in the Finnish national guide to publication metrics.
Tools for analysing publication channels

The methods used to evaluate journals provide an estimate of the average visibility of a journal. Indicators based on the number of citations should be used with caution, and it should be remembered that, due to the variation between fields of science and publication types, they are not suitable for evaluating a single article, researcher or research organisation.

When using the Publication Forum classification, it should be noted that the classification is intended for examining the average quality of large volumes of publications by universities.

The concerns about Impact Factor have already been raised in the San Francisco Declaration on Research Assessment in 2012. The declaration drew attention to the fact that Impact Factor was originally developed as a tool for deciding on journal acquisitions, not for assessing the scientific quality of journals or articles.