Evaluation based on scientific publishing: Subjective assessment methods

Expert surveys

Expert surveys are journal evaluating methods based on subjective assessments by panels of a group of expert scientists. This sort of evaluation is usually done by universities and institutes for different purposes, such as establishing the level of research, evaluating researchers and subscribing to journals. These evaluations are particularly useful for those journals that do not have Impact Factors available. There are several such journal rating lists available on the Internet.

Expert surveys are hard work, and even though many researchers can name a few leading journals within their discipline, evaluating hundreds of journals is very difficult. Journal ratings based on expert opinion are subjective, and reviews of the same journal can be different in different listings. Nevertheless, they do give you valuable information on the quality of the journal, and supplement quantitative measures based on statistics.

Publication Forum (Julkaisufoorumi, JUFO)

Publication Forum system is based on quality classification of scientific publication channels. Rating takes place in field specific expert panels. Evaluations based on bibliometric methods can be complemented with JuFo which is suitable for evaluation of publications of research institutes, disciplines or countries but not for evaluation of individual researchers. JuFo -levels of publications are used as one criteria in the funding model  for universities by the Ministry of  Culture and Education.

Information about classification criteria of JuFo.
Search publication channels in JuFo.
JuFo is updated by the Federation of Finnish Learned Societies.

Peer Reviews

Journals that have peer reviews, carried out by experts, of the manuscripts that are offered for publication are usually seen to be of high quality. Peer reviewing guarantees that researchers uphold certain codes of conduct that are expected within their discipline, prevents groundless claims and false interpretations.

Different journals carry out peer reviews in different ways, and you can find out whether a journal uses peer review from Ulrichsweb - Global serials directory. Check also the journal's home page.

Usually peer reviews are conducted anonymously so that the writer of the article does not  know the identity of the reviewer, but the reviewer knows who the writer is. Even though peer review is often seen as being useful, it is not without problems and has been criticised. Problems with peer reviewing are often related to the slow publishing process and the bias with the evaluating. The evaluator can, for example, discriminate against women writers and researchers from small universities, or favour interpretations that support their own opinions. In peer reviewing, serious mistakes and deficiencies are often not noticed, and they only emerge when the article has already been published. In addition, evaluating is subjective and answerability is missing from the evaluation process because it is done anonymously. Peer reviewing can also make plagiarism possible, since the reviewer may abuse unpublished material from the manuscript. Such problems with peer reviewing are fairly rare, and a more central question is how they affect the quality of the article. The influence of peer reviewing has not been well researched, and what research there is has been done mainly with biomedical journals.

In order to avoid the problems of peer reviewing, some journals, mainly new ones have started to use alternative peer reviewing methods.

  • In a "double blind" peer review, neither the authors nor the reviewers know each others' identities. This method is mainly used as a test method when trying to establish the functionality of peer reviewing and chart the possible problems related to it.
  • An open peer review is different from the traditional peer review because it is not conducted anonymously. Its aim is to avoid the traditional abuses and problems of peer reviewing. On the other hand, it also generates academic merit for the reviewer. Different journals have different ways of conducting open peer reviews, but the basic principle is that the author of the text knows the identity of the reviewer. Some journals only inform the writer of the article about the identity of the reviewers, but other journals publish the names of the reviewers in the journal too. In addition, some journals even publish the reviewer's report, and in that way the readers gain insight into the whole review procedure. A problem with open peer reviewing is that not all researchers are willing to be reviewers when the reviewing is not anonymous. An example of open peer review in Biology Direct
  • In post-publication refereeing, all manuscripts are published electronically in an archive where they are free to be commented on. The writer will alter the script based on the comments, and after that the script will be peer reviewed normally, and finally it will be published in an academic journal. This model is used in physics and in related subjects, like mathematics and information technology. This sort of peer reviewing can work if distinguished researchers take part in commenting and the comments are honest and candid. It cannot however, replace traditional peer reviewing but could be used as an addition. arXiv.org is an example of post publication peer review

Some journals allow everyone to comment on the articles after they have been published, using Internet conversations in which the authors of the articles can participate; this is known as open peer commentary.

Prestige of journals

Some journals have a relatively permanent prestigious reputation and enjoy international recognition which extends further than just the circles of researchers of the specific discipline that the journal represents. Examples of such journals could be Nature, Science and JAMA. The reputation of a journal can be based on the reputation of the publishing community or society, the publisher or editor-in-chief, and it therefore does not always tell us anything about the quality or influence of the journal. The reputation of the publisher or publishing house is an important evaluation criterion, especially for new journals.

Even though the reputation of a journal is fairly permanent, the journal itself does change somewhat with time, and reputation is therefore not a very good evaluation basis. On the other hand, when examining the relationship between the reputation of a journal and how influential it is, one can find that both having a really good reputation and having a really bad reputation are exaggerated in terms measures of influence. This can be explained by the "Mathew effect" whereby journals that already have good reputations get more glory and publicity than they would actually deserve and vice versa. The good reputation of a journal is often taken for granted and it is not re-evaluated.

The preservation of academic journals and their significance in the electronic publishing age has divided opinion, but it seems that they have still maintained their status. The journal is still important because the name tells the reader about the content and the quality. However, what is no longer important is that the journal is published in separate issues, rather articles can be published individually when they are ready.