Introduction to Interrater Agreement for Nominal Data

1. Front Matter

   Pages i-xii

   PDF

2. Theory and Practice

   1. Front Matter

      Pages 1-1

      PDF

   2. Introduction

      • Roel Popping

      Pages 3-9

   3. Reliability and Validity

      • Roel Popping

      Pages 11-20

   4. Interrater Agreement

      • Roel Popping

      Pages 21-78

3. Formulas

   1. Front Matter

      Pages 79-79

      PDF

   2. Indices

      • Roel Popping

      Pages 81-145

4. Back Matter

   Pages 147-150

   PDF

This introductory book enables researchers and students of all backgrounds to compute interrater agreements for nominal data. It presents an overview of available indices, requirements, and steps to be taken in a research project with regard to reliability, preceded by agreement. The book explains the importance of computing the interrater agreement and how to calculate the corresponding indices. Furthermore, it discusses current views on chance expected agreement and problems related to different research situations, so as to help the reader consider what must be taken into account in order to achieve a proper use of the indices.


The book offers a practical guide for researchers, Ph.D. and master students, including those without any previous training in statistics (such as in sociology, psychology or medicine), as well as policymakers who have to make decisions based on research outcomes in which these types of indices are used.

• Interrater agreement
• Nominal data
• Agreement
• Reliability
• Coding
• Raters
• Chance correction
• Agreement indices
• Inference
• Reporting reliability

• Department of Sociology, University of Groningen, Groningen, The Netherlands

  Roel Popping

Roel Popping taught statistics and methodology courses including text analysis at the Department of Sociology of the University of Groningen, the Netherlands, until 2013. Afterwards, he served as a Visiting Professor at the Department of Political Science of the Central European University, Budapest, Hungary, for four years. His research interests include methodology, with a specialty in text analysis and the study of agreement, their application in research on historical shifts in public opinion and values, primarily in the context of post-1989 Central and Eastern Europe, as well as decision-making for negotiation processes.

Policies and ethics