Evaluation based on scientific publishing: Subjective assessment methods

Expert surveys

Expert surveys are journal evaluating methods based on subjective assessments by panels of a group of expert scientists. This sort of evaluation is usually done by universities and institutes for different purposes, such as establishing the level of research, evaluating researchers and subscribing to journals. These evaluations are particularly useful for those journals that do not have Impact Factors available. There are several such journal rating lists available on the Internet.

Expert surveys are hard work, and even though many researchers can name a few leading journals within their discipline, evaluating hundreds of journals is very difficult. Journal ratings based on expert opinion are subjective, and reviews of the same journal can be different in different listings. Nevertheless, they do give you valuable information on the quality of the journal, and supplement quantitative measures based on statistics.

Publication Forum (Julkaisufoorumi, JUFO)

Publication forum system is based on quality classification of scientific publication channels. Rating takes place in field specific expert panels. Evaluations based on citation information should be complemented with further expert assessments. Finnish Publication Forum is following the Norwegian and Danish examples.

The Finnish Publication Forum is based on quality classification of scientific publication channels - journals, conferences, publication series and book publishers - in all research fields. Rating of publication channels takes place in field specific expert panels.
The rating has three levels: 1 = basic; 2 = leading; 3 = highest level.

The Publication Forum category of peer-reviewed articles in journals and compilations and monographs is determined principally on the basis of the level awarded to the journal/series.

The classification of the book publisher is used to rate articles in compilations or monographs for Publication Forum purposes only in cases where the publication does not come out in a series of publications or such series has not been classified.

The level of peer-reviewed conference papers is determined on the basis of the level of the journal/series, publisher or established name of the event.

The classification is suitable for macro-level discussions on the publication production of research organisations, research fields or the country as a whole. Since 2015, the classification has been used as a quality indicator of the research output produced by universities within the university funding model established by the Ministry of Education and Culture.

Peer Reviews

Journals that have peer reviews, carried out by experts, of the manuscripts that are offered for publication are usually seen to be of high quality. Peer reviewing guarantees that researchers uphold certain codes of conduct that are expected within their discipline, prevents groundless claims and false interpretations. For example, Thomson Scientific accepts only peer reviewed journals for their database. Different journals carry out peer reviews in different ways, and you can find out whether a journal uses peer review from Ulrichsweb.

Usually peer reviews are conducted anonymously so that the writer of the article does not  know the identity of the reviewer, but the reviewer knows who the writer is. Even though peer review is often seen as being useful, it is not without problems and has been criticised. Problems with peer reviewing are often related to the slow publishing process and the bias with the evaluating. The evaluator can, for example, discriminate against women writers and researchers from small universities, or favour interpretations that support their own opinions. In peer reviewing, serious mistakes and deficiencies are often not noticed, and they only emerge when the article has already been published. In addition, evaluating is subjective and answerability is missing from the evaluation process because it is done anonymously. Peer reviewing can also make plagiarism possible, since the reviewer may abuse unpublished material from the manuscript. Such problems with peer reviewing are fairly rare, and a more central question is how they affect the quality of the article. The influence of peer reviewing has not been well researched, and what research there is has been done mainly with biomedical journals.

In order to avoid the problems of peer reviewing, some journals, mainly new ones have started to use alternative peer reviewing methods.

  • In a "double blind" peer review, neither the authors nor the reviewers know each others' identities. This method is mainly used as a test method when trying to establish the functionality of peer reviewing and chart the possible problems related to it.
  • An open peer review is different from the traditional peer review because it is not conducted anonymously. Its aim is to avoid the traditional abuses and problems of peer reviewing. On the other hand, it also generates academic merit for the reviewer. Different journals have different ways of conducting open peer reviews, but the basic principle is that the author of the text knows the identity of the reviewer. Some journals only inform the writer of the article about the identity of the reviewers, but other journals publish the names of the reviewers in the journal too. In addition, some journals even publish the reviewer's report, and in that way the readers gain insight into the whole review procedure. A problem with open peer reviewing is that not all researchers are willing to be reviewers when the reviewing is not anonymous.


Biology Direct uses open peer-review. It is an open access journal and it's key aim is to provide authors and readers with an alternative to the traditional model of peer review. This includes making the author responsible for obtaining reviewers' reports via the journal's Editorial Board; making the peer review process open rather than anonymous; and making the reviewers' reports public, thus increasing the responsibility of the referees and eliminating sources of abuse in the refereeing process. Image source: Biology Direct <http://www.biology-direct.com/> 8.4.2009.

Nature undertook a trial on open peer reviewing in 2006. The results can be read at http://www.nature.com/nature/peerreview/.

  • In post-publication refereeing, all manuscripts are published electronically in an archive where they are free to be commented on. The writer will alter the script based on the comments, and after that the script will be peer reviewed normally, and finally it will be published in an academic journal. This model is used in physics and in related subjects, like mathematics and information technology. This sort of peer reviewing can work if distinguished researchers take part in commenting and the comments are honest and candid. It cannot however, replace traditional peer reviewing but could be used as an addition.

arXiv.org uses post-publication refereeing. Image source: Arxiv <http://arxiv.org/> 8.4.2009.

Some journals allow everyone to comment on the articles after they have been published, using Internet conversations in which the authors of the articles can participate; this is known as open peer commentary.

Prestige of journals

Some journals have a relatively permanent prestigious reputation and enjoy international recognition which extends further than just the circles of researchers of the specific discipline that the journal represents. Examples of such journals could be Nature, Science and JAMA. The reputation of a journal can be based on the reputation of the publishing community or society, the publisher or editor-in-chief, and it therefore does not always tell us anything about the quality or influence of the journal. The reputation of the publisher or publishing house is an important evaluation criterion, especially for new journals.

Even though the reputation of a journal is fairly permanent, the journal itself does change somewhat with time, and reputation is therefore not a very good evaluation basis. On the other hand, when examining the relationship between the reputation of a journal and how influential it is, one can find that both having a really good reputation and having a really bad reputation are exaggerated in terms measures of influence. This can be explained by the "Mathew effect" whereby journals that already have good reputations get more glory and publicity than they would actually deserve and vice versa. The good reputation of a journal is often taken for granted and it is not re-evaluated.

The preservation of academic journals and their significance in the electronic publishing age has divided opinion, but it seems that they have still maintained their status. The journal is still important because the name tells the reader about the content and the quality. However, what is no longer important is that the journal is published in separate issues, rather articles can be published individually when they are ready.