Eco Notepad

Quality and Diversity of Research: a Ranking of Academic Journals

3 June 2022

Post n°272. In order to assess the quality of its research output, the Banque de France has constructed its own ranking of academic journals in economics and finance. The objective is to encourage the production of research on issues relevant to the Banque de France. This post explains the ten principles underlying the construction of this ranking.

Image Chart 1: Evolution of the number of authors of academic papers affiliated with one of the Eurosystem central banks on the REPEC website Source: Authors' calculations based on central bank ranking data from the REPEC website
Chart 1: Evolution of the number of authors of academic papers affiliated with one of the Eurosystem central banks on the REPEC website Source: Authors' calculations based on central bank ranking data from the REPEC website

The emphasis on research by central banks is growing steadily. According to the IDEAS-REPEC website, the number of researchers in euro area central banks increased threefold between 2012 and 2022 (Chart 1). Furthermore, more and more academic research papers are referred to in central bankers' speeches: this holds true both for the number of speeches and for the average number of references per speech (Chart 2).

Image Chart 2: Citations of academic research in central bankers’ speeches Source: Banque de France based on research papers referred to in speeches published online by members of the governing bodies of three central banks.
Graphique 2 : Citations des travaux des chercheurs des banques centrales dans des discours Source : Banque de France à partir des travaux cités dans les discours disponibles en ligne des membres des organes dirigeants de trois banques centrales.

Central banks rely on research for decision-making, for informing the economic public debate, and for contributing to academic production. These three objectives shape the metric for assessing the quality of research constructed by the Banque de France. Its aim is to provide a tool for assessing the quality of publications while preserving the academic freedom of researchers.

This journal ranking is designed to encourage the production of high-quality research while preserving the diversity and equality between different types of research (theory, empirics, history, experimental) and different research topics. While high-quality research contributes to high-quality decision-making, the need for diversity aims to respond to the situations described by Jean-Claude Trichet in 2010: "in the face of the crisis, we felt abandoned by conventional tools. ...[W]e were helped by one of the areas of economic literature: historical analysis".

In order to meet these objectives, the Banque de France has defined ten principles for constructing its ranking of academic journals. These principles were defined by a working group made up of representatives of the Bank's research-oriented directorates, whose members are the authors of this post, in consultation with researchers in order to take into account their reactions and suggestions.

Combining research quality and diversity

A ranking embodies a scientific policy that must take a long-term perspective. Principle 1 establishes the need to rank journals in accordance with the recommendations made in the “External evaluation of research at the Banque de France” report produced by Patrick Bolton, Stephen Cecchetti and Lucrezia Reichlin in 2016.

Research should aim for excellence, and should be checked and validated by anonymous external reviewers who guarantee the quality of the scientific work. Principle 2 therefore requires that the Banque de France ranks only peer-reviewed journals referenced on the REPEC website.

Each journal is characterised by a governance system monitoring the work of both the editors and the anonymous reviewers (referees). This governance and the quality of the referees’ work, as well as the reputational effects linked to the age of the journal, often account for the quality of the best journals. Principle 3 therefore concerns the determination of the academic journals with the highest ranking in each field with the restriction that, by definition of what is a top field journal, this group includes at most two best journal(s) per research field.

Chart 3 shows the diversity of the research fields by the Banque de France.

Image Chart 3: Breakdown of the topics of academic papers published by Banque de France researchers in 2020 Source: Banque de France
Chart 3: Breakdown of the topics of academic papers published by Banque de France researchers in 2020 Source: Banque de France

Principle 4 defines the types of academic journals ranked, with the aim of representing the diversity of expertise needs. In addition to the "general interest" journals, eleven fields typical of the usual fields of central banking research are listed. In view of the critical need for interdisciplinary research, a "social sciences" field is included, while the "climate and environment" field reflects the integration of these issues by central banks.

Image Source: Banque de France
Source: Banque de France

Encouraging researchers to produce high-quality research on a regular basis

In order for each researcher to take it into account and in order to use it for assessing research activity, the ranking system integrates signals that are useful to the institution. Principles 5 to 8 aim at ensuring that 1) short papers published in quality journals with a rapid validation process are taken into account, 2) the ranking and therefore individual evaluations are predictable; 3) co-authorship in research is encouraged and 4) researchers' careers between different institutions are fluid.

Principle 5, in particular, concerns papers published by some academic journals in a short form, i.e. as “Papers & Proceedings” or guest papers, which are downgraded one notch compared to the journal concerned.

Research is a collaborative activity par excellence that requires a wide range of skills. Principle 6 aims at encouraging collective work by weighting co-authorships favourably, while avoiding the juxtaposition of too many authors (less than seven).

Principle 7 defines the effective allocation of a number of points to researchers for a publication in each academic journal within a given field. In order to take into account the diversity of research needs, the fields are not ranked, regardless of the number of references or the popularity of the field in the academic community. The journals in each field are ranked according to the five layers of quality. Each publication in a journal grants to its authors a number of points ranging from 25 for the lowest-ranked journals to 400 for the best.

Researchers' careers rarely take place in a single institution. Research requires exchanges, and mobility should not be penalised. Principle 8 therefore ensures harmonisation with other journal rankings made by other central banks by stating that no journal may be ranked by the Banque de France below the ranks attributed to the same journal by similar institutions.

These different principles make it possible to define for each researcher a minimum expected publication activity, defined over a cycle of several years, which can thus be used in the same way as the number of citations to determine some bonuses or for career promotion.

A ranking of academic journals should be regularly updated. Principles 9 and 10 provide that the ranking can be completed during the year to add a journal not yet ranked. A group of researchers from the Banque de France is conveyed for this purpose.

To conclude

Research is a two-pronged activity, as it is based on both emulation and cooperation. By clarifying the principles for ranking academic journals, our aim was to establish a framework governing the evaluation of each researcher's contributions in order to allow for argued and informed discussion. Since the demand for one type of research or another is partly unpredictable, it was necessary to preserve the incentives to publish in all fields that are potentially useful for the Banque de France to fulfill its missions. As shown in Chart 4, this led to equalising the weighting of each field compared to the REPEC ranking.

Image Chart 4: Comparison between the value assigned to category 1 and 2 journals by the Banque de France and IDEAS-REPEC academic journal rankings as a % of a Top5 publication in economics or Top3 in finance Source: Authors' calculations based on the REPEC ranking "Recursive Discounted Impact Factors (Last 10 Years)" and the Banque de France journal ranking.
Chart 4: Comparison between the value assigned to category 1 and 2 journals by the Banque de France and IDEAS-REPEC academic journal rankings as a % of a Top5 publication in economics or Top3 in finance Source: Authors' calculations based on the REPEC ranking "Recursive Discounted Impact Factors (Last 10 Years)" and the Banque de France journal ranking.

Key: a researcher publishing two papers in the first two categories of econometrics journals obtains 14% of the value of these same two papers with the REPEC ranking if they were published in the eight best economics and finance journals, against 60% in the Banque de France ranking.

Note: each bar gives, for each of the fields defined by the Banque de France, the score obtained by a researcher with two research articles published in the first two categories of journals in his or her field. This metric is calculated relative to the value assigned by the REPEC and Banque de France rankings to the top eight economics and finance journals (i.e. the top 5 in economics and the top 3 in finance).