Although established forms of peer review are often criticized for being slow, secretive, and even unfair, they are repeatedly mentioned by academics as the most important indicator of quality in scholarly publishing. In many countries, the peer review of books is a less codified practice than that of journal articles or conference papers, and the processes and actors involved are far from uniform. In Sweden, the review process of books has seldom been formalized. However, more formal peer review of books has been identified as a response to the increasing importance placed on streamlined peer-reviewed publishing of journal articles in English, which has been described as a direct challenge to more pluralistic publication patterns found particularly in the humanities. In this study, we focus on a novel approach to book review, Kriterium, where an independent portal maintained by academic institutions oversees the reviewing of academic books. The portal administers peer reviews, providing a mark of quality through a process which involves reviewers, an academic coordinator, and an editorial board. The paper studies how this process functions in practice by exploring materials concerning 24 scholarly books reviewed within Kriterium. Our analysis specifically targets tensions identified in the process of reviewing books with a focus on three main themes, namely the intended audience, the edited volume, and the novel role of the academic coordinator. Moreover, we find that the two main aims of the portal–quality enhancement (making research better) and certification (displaying that research is of high quality)–are recurrent in deliberations made in the peer review process. Consequently, we argue that reviewing procedures and criteria of quality are negotiated within a broader discussion where more traditional forms of publishing are challenged by new standards and evaluation practices.