By Sagi Shaier · 24 October 2026 · 8 min read

Data annotation for NLP: crowdsourcing vs experts, and inter-annotator agreement

How to design labels people can apply consistently, when to pay a crowd and when to hire experts, and how to compute Cohen's kappa to check whether your labels can be trusted.

Annotation is attaching the labels a task needs to each raw example, such as a box around every car in a photo or a sentiment for every product review. A supervised model learns the mapping from text to label, so it can only be as consistent as the people who labeled its training data. Most of the effort goes into making those labels mean the same thing from one example, one day and one person to the next.

The labeling scheme

Annotation starts with a labeling scheme: the exact set of allowed labels and what each one means. "Positive, neutral, negative" is a scheme. So is a list of twelve support-ticket categories.

A workable scheme has to balance three demands that conflict:

• Consistency: hand the same text to two annotators (the people applying the labels) and they should come back with the same label. Otherwise the labels are noise.
• Coverage: the scheme has to record the distinctions your task depends on. A sentiment scheme with only "positive" and "negative" has nowhere to put "It arrived on time."
• Throughput: it has to be simple enough to label thousands of examples without the work stalling.

Push hard on one and the other two suffer. Adding labels improves coverage, and each judgment then takes longer and annotators split more often.

Annotation guidelines

Annotation guidelines are a written document with the rules for each label and worked examples of the hard cases. They help even when you label everything yourself, because without written rules you end up labeling the same kind of example differently from one day to the next. With several annotators the problem grows, since two reasonable people will label the same example differently.

Guidelines help most on edge cases. Take the review "oh great, another update that breaks everything." Is it positive because of the word "great", or negative because it is sarcastic? The guidelines have to make that call and show an example, so every case like it gets the same label.

Guidelines are also where coverage and throughput conflict. The finer the scheme, the longer each judgment takes. So teams usually trim the scheme down to one that anyone labeling can apply the same way every day, and they accept losing some detail.

Gold labels and the metadata shortcut

Gold labels are the trusted, agreed-upon labels you train and evaluate the model on. Every score you report is measured against them, so they need to be right.

A tempting shortcut is to reuse metadata that already exists as labels. A review's star rating stands in for its sentiment, or a forum post's tag stands in for its topic. The labels cost nothing to collect, and the catch is selection bias. The examples that happen to carry metadata may not look like the ones that do not. A model that scores well on the labeled slice can then fail on everything else.

Annotation is also iterative. Annotators keep finding cases nobody anticipated, so the guidelines get revised as the work goes on.

Sometimes you cannot share the text itself, because it is copyrighted or holds personal data. The workaround is stand-off annotation: you release only the labels, each keyed to a document identifier, and others fetch the original text themselves.

Crowdsourcing vs expert annotation

Small projects are often labeled by the team itself. Larger ones need more people, and choosing who labels comes down to cost against label quality.

Crowdsourcing puts many non-expert workers on a task. Platforms such as Amazon Mechanical Turk let requesters post labeling tasks that workers complete for pay. It is cheap and fast, which is why many labeled datasets are built this way.

Crowd labels are noisier, so each item usually goes to several workers and their answers get combined. A simple way to combine them is a majority vote:

1from collections import Counter
2 
3votes = ["negative", "negative", "neutral", "negative", "negative"]
4label, count = Counter(votes).most_common(1)[0]
5print(label, count, "of", len(votes))

1negative 4 of 5

Combining votes only helps when the workers judge independently and bring different perspectives. If they all copy the same cue or share the same blind spot, five labels repeat one error five times.

When a task needs domain knowledge, such as labeling clinical notes or legal filings, teams use expert annotation instead. Trained specialists produce cleaner labels, more slowly and at a higher price. Many teams mix the two, for example crowd labels checked by an expert, or guidelines written by experts and applied by the crowd.

Incentives and ethics

Some annotation is unpaid, for example citizen science, games or the small challenges a website shows to check that you are human. Most large-scale labeling is paid, per task or per hour. Pay alone does not make the labels good, so the workflow needs checks:

• Catch trials: items with a known answer mixed in to spot careless work.
• Double-labeling: the same item labeled by two people.
• Disagreement review: someone looks at the items where labels differ.
• Clear rejection rules, written down before the work starts.

The people doing this work often have little power in the arrangement. Pay can be low, the work unstable, and a requester can reject a submission. Treating workers decently means setting pay by how long the task takes and saying who is collecting the data and why. It also means warning people before they see disturbing material.

Inter-annotator agreement

If two people label the same items and disagree half the time, the labels are unreliable, and the task itself may be poorly defined. Inter-annotator agreement measures how consistently independent annotators give the same label to the same item.

Raw agreement

The simplest measure is raw agreement, the fraction of items where both annotators picked the same label:

1def raw_agreement(a, b):
2    matches = 0
3    for i in range(len(a)):
4        if a[i] == b[i]:
5            matches += 1
6    return matches / len(a)

a and b hold the two annotators' labels, one per item. The loop adds 1 to matches whenever both gave item i the same label, and dividing by the number of items turns the count into a fraction.

Raw agreement is easy to compute and easy to be fooled by. On a two-way choice, two people guessing at random already match about half the time. Here are two annotators who flip a coin for each of 1,000 sentences:

1import random
2 
3random.seed(0)
4guess_a = [random.choice(["pos", "neg"]) for _ in range(1000)]
5guess_b = [random.choice(["pos", "neg"]) for _ in range(1000)]
6print(raw_agreement(guess_a, guess_b))

10.507

They agree on 50.7% of the items without reading a single sentence, so you need a number that accounts for chance.

Correcting for chance

The fix is to subtract the agreement you would expect from chance alone and see how much is left. Say two annotators agree on 80% of items, and 60% agreement is what chance alone would give them. That leaves 40% of the range above chance, and the annotators fill 20 points of it, half. A chance-corrected score calls that 0.50.

Several scores apply this idea:

• Cohen's kappa is the standard chance-corrected score for two annotators. A kappa of 0 means agreement no better than chance, and 1 means perfect agreement.
• Fleiss' kappa extends the same idea to more than two annotators.
• For labels that are not categories, such as a 1-to-5 rating, an exact match is too strict. Agreement is measured by how far apart the ratings are instead.
• Krippendorff's alpha generalizes chance-corrected agreement to those other kinds of labels.

Computing Cohen's kappa

Cohen's kappa needs two numbers. $p_o$ is the observed agreement, the fraction of items the two annotators labeled the same. $p_e$ is the agreement expected by chance. To get it, multiply the two annotators' rates for each label and add up the products. The formula is

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

The top is how far the annotators got above chance, and the bottom is how far above chance they could have got.

For example, two annotators each sort 100 sentences into positive or negative. They agree on 80: both say positive on 50 and both say negative on 30. So $p_o = 0.80$.

Annotator A marked 60 positive and 40 negative, and annotator B did the same. By chance alone, they would both say positive with probability $0.6 \times 0.6 = 0.36$ and both say negative with probability $0.4 \times 0.4 = 0.16$. So $p_e = 0.36 + 0.16 = 0.52$, and

$$\kappa = \frac{0.80 - 0.52}{1 - 0.52} = \frac{0.28}{0.48} \approx 0.58$$

The annotators agree well above chance, and far from perfectly.

The same calculation in Python, on label lists built to match that table:

1def cohens_kappa(a, b):
2    n = len(a)
3    p_o = raw_agreement(a, b)
4    count_a, count_b = Counter(a), Counter(b)
5    p_e = 0.0
6    for label in set(a) | set(b):
7        p_e += (count_a[label] / n) * (count_b[label] / n)
8    return (p_o - p_e) / (1 - p_e), p_o, p_e
9 
10a = ["pos"] * 50 + ["pos"] * 10 + ["neg"] * 10 + ["neg"] * 30
11b = ["pos"] * 50 + ["neg"] * 10 + ["pos"] * 10 + ["neg"] * 30
12kappa, p_o, p_e = cohens_kappa(a, b)
13print(round(p_o, 2), round(p_e, 2), round(kappa, 2))
14 
15kappa, p_o, p_e = cohens_kappa(guess_a, guess_b)
16print(round(p_o, 3), round(kappa, 3))

10.8 0.52 0.58
20.507 0.014

At 50.7% raw agreement the coin-flipping annotators get a kappa of 0.014, close to the 0 that chance predicts.

Reading a kappa value

Kappa is a scale rather than a pass or fail mark. A rough, common reading:

Harder, more subjective tasks land lower even with trained annotators, so what counts as good depends on the task. A low score also says something about the setup, not only the people. The guidelines may be unclear, or the task may be hard to apply consistently. Items the annotators disagree on get settled by adjudication: a discussion, or a tie-breaking expert, turns each one into a single gold label.

Common mistakes

• Reporting raw agreement alone. On a two-label task, 50% comes from chance, so report kappa next to it.
• Labeling without written guidelines. The same person labels similar cases differently from one day to the next, and two people disagree even more.
• Using star ratings or tags as gold labels without checking them. Read a sample and compare it with what a person would label.
• Treating a low kappa as the annotators' fault. Look at the items they disagree on first, which often point to a gap in the guidelines.

Related concepts

Annotation is one step of building your own dataset, after choosing the text and extracting it from web pages or PDFs, and before documenting how it was made. When a public dataset already has labels, reading its dataset card tells you who labeled it and how. After labeling, the text still goes through preprocessing before training, and the label counts get checked for class imbalance.

QuiddityML teaches annotation, crowdsourcing and inter-annotator agreement as three concepts in the NLP track's data unit, where the exercises include filling in the blanks of the raw agreement function and working through Cohen's kappa one step at a time on a 100-sentence example (QuiddityML).