Evaluation based on scientific publishing: Evaluating conferences

Evaluating conferences

In some fields such as Computer Science and Electric Engineering, peer-reviewed conferences are important channels for the fast dissemination of research results since the publication process in this type of venue is normally shorter than for journals.

The most common method for evaluating the reputation and quality of a publication venue is through citation analysis, where the quality of a venue is directly related to the citations received by the papers published by that venue. There are several citation-based metrics for assessing journals as publication venues. In the case of scientific conferences, there are no consolidated metrics or criteria for quality assessment.

Another approach for evaluating the quality of publication venues is to consult and analyze the opinion of a large number of specialists in a given scientific field. However, the cost for consulting and collecting the opinion of a large number of specialists may be extremely high, even impractical, particularly in very dynamic fields, in which venues are created, cease to exist, and change quality very often.

The main criteria used in evaluating the quality of conferences include analyzing the impact of papers through citation counting, submission and acceptance rates, tradition of the conference, relationship to industry (sponsors), convenience of conference locations, and analyzing the characteristics of the Program Committee members.

Publication Forum (Julkaisufoorumi, JUFO)

Publication forum system is based on quality classification of scientific publication channels. Rating takes place in field specific expert panels. Evaluations based on citation information should be complemented with further expert assessments. Finnish Publication Forum is following the Norwegian and Danish examples.

The Finnish Publication Forum is based on quality classification of scientific publication channels - journals, conferences, publication series and book publishers - in all research fields. Rating of publication channels takes place in field specific expert panels.
The rating has three levels: 1 = basic; 2 = leading; 3 = highest level.

The Publication Forum category of peer-reviewed articles in journals and compilations and monographs is determined principally on the basis of the level awarded to the journal/series.

The classification of the book publisher is used to rate articles in compilations or monographs for Publication Forum purposes only in cases where the publication does not come out in a series of publications or such series has not been classified.

The level of peer-reviewed conference papers is determined on the basis of the level of the journal/series, publisher or established name of the event.

The classification is suitable for macro-level discussions on the publication production of research organisations, research fields or the country as a whole. Since 2015, the classification has been used as a quality indicator of the research output produced by universities within the university funding model established by the Ministry of Education and Culture.

Examples of conference rankings

Perfil-CC ranking

The Perfil-CC ranking, accessible at http://www.latin.dcc.ufmg.br/perfilccranking/, provides a list of 1000 computer science conferences divided in three categories (A, B, and C) according to their quality, A being the highest quality category. The ranking list was created by a polling process, in which over 300 Brazilian specialists classified conferences into three categories. A procedure like this is very costly and time consuming.

The Perfil-CC ranking for Computer Science conferences. Conferences listed above belong to the highest category A.
Source: Perfil-CC ranking <http://www.latin.dcc.ufmg.br/perfilccranking/> 25.8.2009.


The conference ranking, accessible at http://www.conference-ranking.org/, is produced with collection of information from different Internet sources for the reference of research scientists. The listings of the quality conferences have been based on different academics sources but may not be yet complete..

An example of the rankings at conference-ranking.org. The first rank conferences of Artificial Intelligence.
Source: Conference-ranking.org <http://www.conference-ranking.org/index.html> 25.8.2009.

computer science conference ranking

Computer sciences conference ranking, accessible at http://www.cs-conference-ranking.org/conferencerankings.html, ranks computer science conferences based on EIC (Estimated Impact of Conference) values.

An example of the rankings at the computer sciences conference ranking. The ranking of Artificial Intelligence, Machine Learning, Robotics, and Human Computer Interaction conferences.
Source: Computer Science Conference Ranking <http://www.cs-conference-ranking.org/conferencerankings.html> 25.8.2009.


CiteSeerX ranks publication venues using Impact Factors. The venues include journals and conference proceedings included in DBLP database.

CiteSeerX provides a ranking list of venue impact factors. Above is the 2007 list.
Source: CiteseerX <http://citeseerx.ist.psu.edu/> 25.8.2009.