Intercoder reliability: a guide to accurate data

Intercoder reliability is an important aspect of content analysis. This article will explain what it is, why it is important, and when to use it.

Intercoder reliability is a measure of agreement between different coders on how to code the same data. This approach is used in content analysis when accuracy and consistency are key research objectives.

Intercoder reliability ensures you arrive at the same findings when multiple researchers code the same content.

Intercoder reliability is a crucial step in content analysis. In some studies, your analysis is only valid if you achieve a certain level of consistency in how you code the data. Coding requires some subjective judgment, and intercoder reliability helps this judgment to be shared among your researchers.

Using intercoder reliability is also an efficient way of getting the work done. If your team can code consistently, you can divide the work between them, so that each researcher handles a distinct part of the data.

You can also use intercoder reliability to prove your data's validity in the event of criticism or doubt.

Intercoder reliability is not the best tool for all research studies. The following factors determine when to use intercoder reliability:

• If you’re conducting a study that requires multiple researchers to interpret data the same way

• When you want your data coded consistently and uniformly

• When you're conducting qualitative content analysis with a group

• When a publication requires you to calculate intercoder reliability

Avoid using intercoder reliability in the following instances:

• When conducting an exploratory study

• If you want to use the perspectives of different researchers

• When looking to discover new things and find out how different people code similar data

There are three steps to calculating intercoder reliability.

Step 1: Select your preferred measure

There are dozens of measures for calculating intercoder reliability, including:

• Cohen's kappa (k)

• Holsti's method

• Krippendorff's alpha (a)

• Percent agreement

• Scott's pi (p)

Ideally, there would be a single, widely accepted index of intercoder reliability, but scholars, methodologists, and statisticians are yet to agree on the "best" one.

There are several recommendations for Cohen's kappa as "the measure of choice," widely used in behavioral coding research. However, Krippendorff has argued that Cohen's kappa is unsuitable as a measure of intercoder agreement.

Percent agreement is the most commonly used measure of intercoder reliability because it is easy to calculate and intuitive. However, critics say it overestimates true intercoder agreement for nominal-level variables.

Although Krippendorff's alpha is a popular and flexible measure, it requires tedious calculations, and automated (software) options are not widely available. You can use this measure with different coders, different sample-sized accounts, and missing data. It can also be used for the interval, ordinal, and ratio level variables.

Choose the right intercoder reliability index for you by considering the various indexes’ assumptions and characteristics. Consider your data properties too, such as the number of coders and the measurement level of each variable for which agreement will be calculated.

Researchers must explain why the assumptions and properties of their chosen index or indices are ideal for the characteristics of the data being analyzed. Stating the reasons for your choice can help to head off any criticism from data reviewers.

Step 2: Experiment with a sample data set

To determine intercoder reliability, ask your researchers to code the same portion of a transcript, then compare the results.

If the level of reliability is low, repeat the exercise until an adequate level of reliability is achieved.

Step 3: Code the data

During the qualitative coding phase, regularly check that your team members are coding consistently. If you find inconsistencies, make changes as needed.

You can increase intercoder reliability by:

• Regulating the quality and selection of sample papers 

• Selecting raters familiar with the constructs to be identified 

• Training the raters in systematic practice sessions

• Specifying the scoring task through clearly defined objective categories

Intracoder reliability entails a single coder's consistency over time, while intercoder reliability involves consistency between coders.

A score of 90 or higher is highly reliable. However, in most studies, a score of 80 or higher may be acceptable.