Who is the 'Journal Grand Master'? A new ranking based on the Elo rating system

Lehmann, Robert and Wohlrabe, Klaus (2017): Who is the 'Journal Grand Master'? A new ranking based on the Elo rating system.

Abstract

In this paper we transfer the Elo rating system, which is widely accepted in chess, sports and other disciplines, to rank scientific journals. The advantage of the Elo system is the explicit consideration of the factor time and the history of a journal's ranking performance. Most other rankings that are commonly applied neglect this fact. The Elo ranking methodology can easily be applied to any metric, published on a regular basis, to rank journals. We illustrate the approach using the SNIP indicator based on citation data from Scopus. Our data set consists of more than 20,000 journals from many scientific fields for the period from 1999 to 2015. We show that the Elo approach produces similar but by no means identical rankings compared to other rankings based on the SNIP alone or the Tournament Method. Especially the rank order for rather 'middle-class' journals can tremendously change.

• Item Type: MPRA Paper
• Language: English
• Keywords: Elo rating system, journal rankings, SNIP indicator
• Subjects: A - General Economics and Teaching > A1 - General Economics > A12 - Relation of Economics to Other Disciplines; A - General Economics and Teaching > A1 - General Economics > A14 - Sociology of Economics

References

Chen, K.-M., Jen, T.-H. and Wu, M. (2014). Estimating the accuracies of journal impact factor through bootstrap. Journal of Informetrics, 8 (1), 181–196.

Elo, A. E. (1978). The Rating of Chessplayers, Past & Present. Arco Publishing, New York NY.

Glickman, M. E. (1995). A Comprehensive Guide to Chess Ratings. American Chess Journal, 3, 59–102.

Glickman, M. E. and Jones, A. C. (1999). Rating the chess rating system. Chance, 12 (2), 21–28.

Greenwood, D. C. (2007). Reliability of journal impact factor rankings. BMC Medical Research Methodology, 7 (1), 48.

Joe, H. (1991). Rating systems based on paired comparison models. Statistics & Probability Letters, 11 (4), 343–347.

Kóczy, L. A. and Strobel, M. (2010). The World Cup of Economics Journals: A Ranking by a Tournament Method. IEHAS Discussion Papers No. MT-DP – 2010/18.

Laslier, J.-F. (1997). Tournament Solutions and Majority Voting. Studies in Economic Theory 7, Springer-Verlag Berlin Heidelberg.

Leydesdorff, L. and Opthof, T. (2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61 (11), 2365–2369.

Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4 (3), 265–277.

Moed, H. F., Colledge, L., Reedijk, J., Moya-Anegon, F., Guerrero-Bote, V., Plume, A. and Amin, M. (2012). Citation-based metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way. Scientometrics, 92 (2), 367–376.

Nieuwenhuysen, P. and Rousseau, R. (1988). A quick and easy method to estimate the random effect on citation measures. Scientometrics, 13 (1-2), 45–52.

Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research, 33 (1), 1–7.

Schubert, A. and Glänzel, W. (1983). Statistical reliability of comparisons based on the citation impact of scientific publications. Scientometrics, 5 (1), 59–73.

Stern, D. I. (2013). Uncertainty Measures for Economics Journal Impact Factors. Journal of Economic Literature, 51 (1), 173–189.

Vanclay, J. K. (2012). Impact factor: outdated artefact or stepping-stone to journal certification? Scientometrics, 92 (2), 211–238.

Veček, N., Črepinšek, M., Mernik, M. and Hrnčič, D. (2014). A Comparison between Different Chess Rating Systems for Ranking Evolutionary Algorithms. In Proceedings of the 2014 Federated Conference on Computer Science and Information System (FedCSIS), pp. 511–518.

Waltman, L., van Eck, N. J., van Leeuwen, T. N. and Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7 (2), 272–285.