By Sagi Shaier · 26 October 2026 · 10 min read

Data quality problems in NLP: duplicates, contamination, bias, and representativeness

The problems to check for in a text dataset before training, how each one changes a model's scores, and short Python checks for duplicates, train/test overlap and test-set size.

A text dataset can look fine at a glance and still be broken. A model learns whatever is in its training data, including the mistakes, so problems in the data turn into problems in the model. Several of them also change the test score, which means a model can look better or worse than it is.

What to check for

The common problems in a text dataset are:

Noise and label errors: garbled text, and examples attached to the wrong label.
Class imbalance: one label far more common than the others.
Duplicates: the same text appearing more than once.
Contamination: test examples leaking into training.
Bias: systematic skews that push a model toward unfair or wrong predictions.
Representativeness: whether the data matches the text the model will meet after deployment.

The small table below shows several of them at once. Row 0002 says "I love this!" and is labeled negative. Rows 0004 and 0005 are the same ad twice. Row 0006 in training and row 9875 in test hold the same review. Row 9876 is legal text, a source the training rows never include.

A dataset table with eight rows next to six things to check: a label error in row 0002, positive labels nearly everywhere, duplicate rows 0004 and 0005, leakage between rows 0006 and 9875, bias in the source column and a mismatch in row 9876

Class imbalance has its own post on handling class imbalance. This post covers the rest.

Noise and label errors

Noise is corrupted or garbled text: encoding errors, cut-off documents and scraped page junk such as menus, ads and cookie banners mixed in with the real content. A model trained on it learns patterns that will not show up in clean text.

Label errors are examples attached to the wrong label, such as "great movie, loved it" marked negative. They do more damage than noise, because they mislead training and the score at the same time. If many test labels are wrong, a model can look accurate against the dataset and still be wrong about the real task.

Scraped HTML with encoding errors and ad text labeled as noise, and the review great movie, loved it labeled negative when it should be positive

Judging label quality automatically is hard. If a program could tell a good label from a bad one, nobody would have needed people to label the data in the first place. A few practical checks help:

Look at the model's confident mistakes. After training, sort the examples where the model disagrees with the label most strongly. Mislabeled examples often sit near the top.
Use catch trials. Mix items with known answers into the labeling work. An annotator who gets them wrong is probably guessing.
Write specific instructions. Give a correct and an incorrect example for each label, and say what to do with an item that does not fit.

Three ways to find bad data, confident errors, catch trials and clear instructions, above a small clean set and a larger noisy set, with the note that cleaning usually beats collecting more

Cleaning a modest amount of bad data usually beats collecting far more data of the same quality.

Duplicate data

Duplicate data is identical or near-identical examples repeated in a corpus. It gets in easily. The same article is republished on several sites, a scraper visits one page twice, or the same footer shows up on every page.

Duplicates cause two problems. You pay to train on the same content again and again. The model also gives repeated content more weight than it deserves, so a phrase from a footer that appears on every scraped page looks like one of the most common phrases in the language.

A grid of documents where the same document appears three times, which wastes compute and makes the model over-weight repeated content

The fix is deduplication. The simplest version uses two containers: a set named seen that answers "have I already kept this text?" quickly, and a list named unique that keeps the texts in the order they first appeared.

1texts = ["a", "b", "a", "c", "b"]
2seen = set()
3unique = []
4 
5for t in texts:
6    if t not in seen:
7        unique.append(t)
8        seen.add(t)
9 
10print(unique)  # ['a', 'b', 'c']

unique.append(t) keeps the first copy, and seen.add(t) makes every later copy of t get skipped.

A trace of the loop over a, b, a, c, b, showing the seen set and the unique list after each step, with the second a and the second b skipped

Exact matching misses near-duplicates, texts that differ only in casing, punctuation or spacing. A common first step is to compare a cleaned-up key instead of the raw string:

1import re
2 
3def dedup_key(text):
4    text = text.lower()
5    text = re.sub(r"[^\w\s]", "", text)
6    return re.sub(r"\s+", " ", text).strip()
7 
8reviews = [
9    "Great phone, battery lasts two days.",
10    "great phone battery lasts two days",
11    "Screen cracked after a week.",
12    "Great phone,  battery lasts two days!",
13]
14 
15seen = set()
16unique = []
17for r in reviews:
18    key = dedup_key(r)
19    if key not in seen:
20        unique.append(r)
21        seen.add(key)
22 
23print(len(reviews), len(unique))  # 4 2

The three phone reviews share one key, so only the first is kept. Texts that differ by a few words, like the same news story under a new headline, need near-duplicate detection that measures how much two texts overlap.

Data contamination

Data contamination is information getting into a part of the data where it does not belong. The common case in supervised learning is train/test leakage: information from the test set reaches training before the final evaluation. The test is no longer independent of training, so the score comes out higher than the model's real performance.

It usually gets in through ordinary mistakes:

Duplicates across splits. If the same review sits in both training and test, the model has seen that test example during training.
Preprocessing before splitting. Fitting a step on the whole dataset and then splitting lets statistics from the test examples shape how the training data was prepared.
Pretraining data. Text scraped from the web may already contain public benchmark examples, so a large model can score well on test text it partly memorized.

Splitting first and fitting preprocessing on the training data only, compared with preprocessing everything first, where test statistics shape the preprocessing and scores come out falsely high

A quick check for the first one is to look for test texts that also appear in training, using the same cleaned-up key as above:

1import re
2 
3def dedup_key(text):
4    text = text.lower()
5    text = re.sub(r"[^\w\s]", "", text)
6    return re.sub(r"\s+", " ", text).strip()
7 
8train = ["Arrived on time", "Loved it, five stars", "Buy now!!!"]
9test = ["Arrived on time", "Section 4 of the agreement applies"]
10 
11overlap = {dedup_key(t) for t in train} & {dedup_key(t) for t in test}
12print(overlap)  # {'arrived on time'}

Any text in that set should be removed from one side before trusting a result.

Protecting the test set

Contamination also happens without copying a single example. If you score many versions of a model on the same test set and keep the best one, each look tunes your choices to that particular set, and the score stops measuring how well the model works on new text.

The usual discipline is to touch the test set once. A separate development set takes all the trial and error, and you can check it as often as you like. It is also called a validation set. Three guardrails cover most cases:

Split the data first, before any preprocessing that mixes examples.
Keep the test set untouched until the final measurement.
Check for overlap between the splits before trusting any score.

A development set only helps if it comes from the same kind of text as the test set. Tuning against a development set that does not match means optimizing for the wrong target. The post on train, validation and test splits covers the splits in more detail.

A bar split into 70 percent train, 15 percent dev that you can look at as often as you like and 15 percent test that you touch once, above three guardrails

Bias

Bias in data means systematic skews that push a model toward unfair or wrong predictions. A model copies the patterns it sees in its training text, including social and historical ones that nobody wants it to learn.

The training text is only one way in. Bias also comes through the labels, when annotators bring their own assumptions. It comes through reused resources, such as sentiment word lists and pretrained word vectors that already carry a skew. It can also come from the training objective, meaning what the model is rewarded for.

Bias entering a model from four sources, the training text, the labels, the training objective and reused resources, with the predictions able to make the skew stronger

A model can also make a skew stronger than it was in the data. If a hiring corpus slightly over-associates one group with a role, a model can turn that mild pattern into a strong rule, and a resume screener ends up favoring patterns tied to gender or ethnicity that have nothing to do with the job.

One concrete harm is output that treats a group of people worse. A sentiment model might rate "Aisha had a great day" more negatively than "Emily had a great day" because it learned to treat certain first names as a negative signal. Toxicity detectors can make a related mistake and flag harmless text that mentions an identity or uses a dialect feature, which removes posts by the very people they misread.

This kind of bias is different from the statistical bias in the bias-variance tradeoff, even though the two share a name.

More data does not remove bias if the new data has the same skew. One partial tool is counterfactual data augmentation. It adds altered copies of examples, such as the same sentence with the gender swapped, so the model sees both versions with the same label:

1SWAPS = {"he": "she", "she": "he", "his": "her", "her": "his", "man": "woman", "woman": "man"}
2 
3def gender_swap(sentence):
4    return " ".join(SWAPS.get(w, w) for w in sentence.split())
5 
6print(gender_swap("she is a leader"))         # he is a leader
7print(gender_swap("he thanked his manager"))  # she thanked her manager

Real swapping takes more care than a word list. "her" can mean "his" or "him" depending on the sentence, and names need swapping too.

Two sentences that differ only in a first name scored differently, a toxicity detector flagging a harmless sentence, and counterfactual pairs such as she is a leader and he is a leader

No method removes bias in general. The responsible move is to state the known biases openly, for example in the model card that documents a model, instead of claiming they are solved.

Representativeness

Representativeness is whether your data reflects the population of text you will run the model on.

It counts most for the test set, which should look like the deployment target. A transcriber built for chemistry lectures should be tested on chemistry lectures, or its score says little about how it will do on the job. A general-purpose model is the opposite case, and its test set has to draw from many sources rather than one author or one website.

When the text after deployment differs from the training text, the model meets distribution shift, and accuracy drops without any error message. A model trained on formal news articles can test well and then do poorly on casual social media posts, because the language it meets is not the language it learned from.

Training and deployment data drawn as two distributions, overlapping when the data is representative and far apart when it is a mismatch

The gap grows with the distance between genres. A model trained on financial reports has little to go on with centuries-old novels, since the two share very little vocabulary or phrasing.

How big the test set needs to be

The test set also needs to be large enough to tell two models apart. On a small test set, the score moves a lot from one sample of test examples to the next. This simulation scores a model whose true accuracy is 85% on 1,000 random test sets of each size:

1import torch
2 
3torch.manual_seed(0)
4true_accuracy = 0.85
5for n in [100, 1000, 10000]:
6    correct = torch.bernoulli(torch.full((1000, n), true_accuracy))
7    scores = correct.mean(dim=1)
8    print(n, round(scores.std().item() * 100, 2))
9# 100 3.6
10# 1000 1.13
11# 10000 0.35

torch.bernoulli marks each test example correct with probability 0.85, and .mean(dim=1) turns each row into one test set's accuracy. With 100 test examples, the score typically lands 3.6 points away from 85%, so a 1-point gap between two models can be pure luck. With 10,000 examples, that spread falls to 0.35 points. The ability of a test set to tell two models apart is called its statistical power. It trades against training data, since every example held out for testing is one the model does not learn from.

Training on formal financial reporting and deploying on centuries-old literature share very little vocabulary, and accuracy across repeated runs swings widely on a small test set, less on a medium one and barely on a large one

How text varies

Checking representativeness means knowing which kinds of variation your data has to cover. The first is which language the text is in. Skills learned in one language rarely carry over cleanly to another, and a tool built and tested only on English leaves most of the world's several thousand languages uncovered. Code-switching, where a speaker mixes two languages in one sentence, breaks the assumption that each text has a single language.

Within one language, text still varies along several axes:

Register: how formal the text is. One person writes differently in a legal filing and in a group chat.
Genre: a news report, a novel, a source code file and a tweet are built differently and use different words.
Dialect: one language splits into regional spellings, words and grammar. A regional form like "iont" for "I don't" does not split into words the way the standard spelling does.
Time: language changes, and text from a century ago can read almost like a different dialect.
Demographics: a writer's age, community and background leave traces in how they write.

Five axes along which text varies, dialect, register, genre, time and demographics, each with an example

For each axis, ask whether your data covers the variation the model will face after deployment.

Common mistakes

Deduplicating after splitting. Copies that already landed in both train and test stay there. Deduplicate the whole dataset first, then split.
Fitting preprocessing on all the data. Fit any step that learns from the data, such as a vocabulary or normalization statistics, on the training split only.
Choosing a model by its test score. Pick on the development set and run the test set once at the end.
Comparing models on a small test set. On 100 examples, a 1-point difference is within the noise.
Testing on the wrong population. A test set drawn from news articles says little about a model meant for customer chat.

These checks come after collecting and labeling the data and before training. The post on building your own NLP dataset covers choosing and extracting the text, and the post on data annotation covers measuring how far annotators agree, which is often the first sign of label errors. The post on data leakage covers leakage outside NLP, such as features that give away the label. Once the data passes these checks, preprocessing turns the text into a consistent form for the model.

QuiddityML gives each problem in this post its own concept in the NLP track's data unit. The exercises on them include filling in and ordering a first-seen deduplication loop and putting the guardrails for protecting the test set in order (QuiddityML).