This review of the international literature on evaluation systems, evaluation practices, and metrics (mis)uses was written as part of a larger review commissioned by the Higher Education Funding Council for England (HEFCE) to inform their independent assessment of the role of metrics in research evaluation (2014–5). The literature on evaluation systems, practices, and effects of indicator uses is extremely heterogeneous: it comprises hundreds of sources published in different media, spread over disciplines, and with considerable variation in the nature of the evidence. A condensation of the state-of-the-art in relevant research is therefore highly timely. Our reviewpresents the main strands in the literature, with a focus on empirical materials about possible effects of evaluation exercises, ‘gaming’ of indicators, and strategic responses by scientific communities and others to requirements in research assessments. In order to increase visibility and availability, an adapted and updated review is presented here as a stand-alone—after authorizationby HEFCE.

Over the past decade, academic social networking sites, such as ResearchGate and Academia.edu, have become a common tool in academia for accessing publications and displaying metrics for research evaluation and self-monitoring. In this conceptual article, we discuss how these academic social networking sites, as devices of evaluation that build on both traditional values, objects, and metrics in academic publishing and on social media logics and algorithmic metrics, come to fulfil a need in the current academic (publishing) ecosystem. We approach this issue by identifying key affordances that arise in the interaction between platform and user. We then position these affordances in relation to potential needs of academics in today’s publishing landscape by drawing on Hafermalz’s metaphor of the ‘fear of exile’, which provides an alternative way of understanding the importance of visibility in the networked world, as a combination of competitive exposure and existential recognition. We end by considering the grounds on which the platforms may be attributed some level of legitimacy. This is done in order to understand the inherent contradiction between the broad use of the platforms and the fact that their integrity has been questioned repeatedly. We seek an answer to a legitimacy for the platforms in the fact that a pragmatic, mutual benefit exists between them and the research community; a benefit that is enhanced by the audit society influencing current academia. 

Given the increased role of bibliometric measures in research evaluation, it is striking that studiesof actual changes in research practice are rare. Most studies and comments on ‘a metric culture’in academia focus on the ideological and political level, and there is a clear shortage of empiricalstudies that analyze how researchers handle demands for accountability in context. In adopting amixed-methods approach involving both bibliometric data and answers form questionnaires, weprovide an in-depth study of how researchers at the faculty of Arts at Uppsala University (Sweden)respond to the implementation of performance-based research evaluation systems. Publicationpatterns from 2006 to 2013 show that journal publications, especially English-language ones, areincreasing, and the proportion of peer-reviewed publications has doubled. These changes are inline with the incentives of the evaluation systems under study. Answers to the survey confirm thatscholars are conscious about this development, and several respondents articulate a disagreementbetween disciplinary norms and external demands. However, disciplinary background aswell as career stage or academic age appears to have a significant influence on how individualresearchers react to the instigation of evaluation systems. Finally, responses to national andlocal evaluation regimes are complex, localized, and dependent on many factors. In-depthcontextualized studies of research practices are needed in order to understand how

Disciplines display field-specific ways of valuing research contributions, and these different ‘styles of valuation’ influence how academic careers are assessed and formed. Yet, differences in how research is evaluated are also prevalent between different levels of assessment: collegial and organizational. Consequently, we employ a multifaceted two-dimensional approach for studying styles of valuation where both horizontal (across domains) and vertical (organization levels) variations in assessment practices are examined. For this purpose, we make use of 16 faculty guidelines and 112 referee reports concerning candidates for becoming ‘docent’ (Habilitation) from four broad domains: the humanities, the social sciences, medicine and the natural sciences (including technology). By inductively identifying five broad dimensions used when assessing publication merits: (1) Attribution of work, (2) Qualities of content, (3) Publication channel, (4) Publication impact, and (5) Publication volume we can distinguish specific styles of valuation for each of our four domains. Moreover, by extending the analysis to an organizational level we detect opposing ways in which the evaluations are justified—what we call ‘disharmonic styles of valuation’. Thus, when developing insights on ‘quality understandings’—and their operationalization through styles of valuation—in academia we need to put less emphasis on their origins and rather focus on how they come to travel between and co-exist within specific evaluative contexts.

The use of bibliometric indicators on individual and national levels has gathered considerable interest in recent years, but the application of bibliometric models for allocating resources at the institutional level has so far gathered less attention. This article studies the implementation of bibliometric measures for allocating resources at Swedish universities. Several models and indicators based on publications, citations, and research grants are identified. The design of performance-based resource allocation across major universities is then analysed using a framework from the field of evaluation studies. The practical implementation, the incentives as well as the ‘ethics’ of models and indicators, are scrutinized in order to provide a theoretically informed assessment of evaluation systems. It is evident that the requirements, goals, possible consequences, and the costs of evaluation are scarcely discussed before these systems are implemented. We find that allocation models are implemented in response to a general trend of assessment across all types of activities and organizations, but the actual design of evaluation systems is dependent on size, orientation, and the overall organization of the institution in question.

A researcher’s number of publications has been a fundamental merit in the competition for academic positions since the late 18th century. Today, the simple counting of publications has been supplemented with a whole range of bibliometric indicators, which supposedly not only measures the volume of research but also its impact. In this study, we investigate how bibliometrics are used for evaluating the impact and quality of publications in two specific settings: biomedicine and economics. Our study exposes the various metrics used in external evaluations of candidates for academic positions at Swedish universities. Moreover, we show how different bibliometric indicators, both explicitly and implicitly, are employed to assess and rank candidates. Our findings contribute to a further understanding of bibliometric indicators as ‘judgment devices’ that are employed in evaluating individuals and their published works within specific fields. We also show how ‘expertise’ in using bibliometrics for evaluative purposes is negotiated at the interface between domain knowledge and skills in using indicators. In line with these results, we propose that the use of metrics we report is best described as a form of ‘citizen bibliometrics’—an underspecified term which we build upon in the article.

This article provides an overview of national bibliographic databases that include data on research output within social sciences and humanities (SSH) in Europe. We focus on the comprehensiveness of the database content. Compared to the data from commercial databases such as Web of Science and Scopus, data from national bibliographic databases (e.g. Flemish Academic Bibliographic Database for the SSH (VABB-SHW) in Belgium, Current Research Information System in Norway (CRISTIN)) are more comprehensive and may, therefore, be better fit for bibliometric analyses. Acknowledging this, several countries within Europe maintain national bibliographic databases; detailed and comparative information about their content, however, has been limited. In autumn 2016, we launched a survey to acquire an overview of national bibliographic databases for SSH in Europe and Israel. Surveying 41 countries (responses received from 39 countries), we identified 21 national bibliographic databases for SSH. Further, we acquired a more detailed description of 13 databases, with a focus on their comprehensiveness. Findings indicate that even though the content of national bibliographic databases is diverse, it is possible to delineate a subset that is similar across databases. At the same time, it is apparent that differences in national bibliographic databases are often bound to differences in country-specific arrangements. Considering this, we highlight implications to bibliometric analyses based on data from national bibliographic databases and outline several aspects that may be taken into account in the development of existing national bibliographic databases for SSH or the design of new ones.