By Sagi Shaier · 23 October 2026 · 7 min read

Why text is hard data for machine learning

How raw text differs from tables and images, and the three decisions an NLP system has to make before a model can learn anything from it.

Text is hard data for machine learning because a model reads numbers, and raw text arrives as a sequence of symbols with no fixed length and no labeled structure. A sentence is not numbers yet. How words get split up, how they become numbers and how long inputs get handled all start from the way raw text behaves.

What counts as NLP data

NLP's raw material is human language. In most NLP systems it reaches the model as text, a sequence of characters. The source might be a tweet, a product review or a legal contract. Those sources differ in length, tone and reliability, and the model-facing form is still text.

Unlike clean table columns or pixel grids, that text arrives messy. Spelling varies, and slang and new names keep appearing. Formatting such as HTML tags and line breaks comes along with the words. The same words can also mean two different things: "I saw her duck" can be about a bird or about someone bending down.

Language also comes as speech. Speech is a separate modality, a different physical form of language with its own models. When spoken language reaches a text model, it is usually transcribed into text first.

A body of text gathered for training or study is called a corpus (plural: corpora). It is the NLP word for a dataset. A pile of news articles, a dump of forum posts and a collection of books are each a corpus.

Why text is unstructured

A spreadsheet tells you what each value means. Column three is age and column five is income, and the structure is right there in the layout.

A sentence has none of that. It carries no labeled fields, columns or rows. The subject, the sentiment and the topic are all in it, but none of them is marked in the raw string. This is what unstructured data means: the structure exists in the grammar and the meaning, and the model has to recover it instead of reading it off directly.

Three decisions before a model can use text

Tables and images reach a model already as numbers in a fixed shape. A table row has the same columns every time. An image is a grid of pixel values, and nearby pixels have a built-in relationship.

Text arrives as a sequence of discrete symbols instead. Before a model can use it, someone has to make several decisions, and three of the biggest come first:

1. What are the units? Whole words, pieces of words, or something else.
2. What numbers do the units get? And what those numbers should say about which units are close.
3. How is the sequence arranged? Texts vary in length, and the order of the units carries meaning.

Decision 1: what are the units?

A model that learns from raw text has to cut it into units it can count and predict. The obvious unit is the word, and a word is surprisingly hard to pin down. Is "don't" one word or two? Is "ice cream"? Is a hashtag a word? Different answers make sense for different purposes.

Whatever rule you pick decides what gets counted. Each occurrence of a word in running text is a token, and each distinct word is a type. The set of types is the vocabulary.

For example, "a cat saw a dog" has five tokens but four types, because "a" appears twice:

1sentence = "a cat saw a dog"
2tokens = sentence.split()
3types = set(tokens)
4print(tokens)
5print(len(tokens), "tokens,", len(types), "types")
6print(sorted(types))

1['a', 'cat', 'saw', 'a', 'dog']
25 tokens, 4 types
3['a', 'cat', 'dog', 'saw']

Splitting on spaces is one rule out of many. A different rule for cutting words gives a different vocabulary.

Words are built from pieces

Counting types puts "walk" and "walked" into the vocabulary as two separate entries, even though one is built from the other. "Walked" is "walk" plus a past-tense ending, and "dogs" is "dog" plus a plural ending. A model that treats every word as its own symbol never looks at those pieces, so nothing links "walk" to "walked".

Irregular forms are harder still. "Went" is the past of "go", and the two share no letters.

New forms keep appearing too. Turkish can stack several endings onto one root, so a single root shows up in many written forms. Many of those forms are rare, which adds to the long tail of words a model has barely seen. Splitting words into smaller pieces is one way NLP systems deal with this.

Decision 2: what numbers do the units get?

Raw symbols offer no help with the second decision. Numbers have closeness built in: 3 is near 4 and far from 900, and most models lean on that kind of distance.

Words have none of it. "cat" and "dog" name two animals a child would group together, but as raw symbols they carry no numeric value, so nothing says they are close. As symbols, "cat" is no nearer to "dog" than to "helicopter". Any sense of meaning or similarity has to be learned, because it is not there in the symbols themselves.

Similar words and related words

A model that knows "car" and "truck" are close can carry what it learns about one over to the other. Words can be close in two different ways.

Similar words can fill the same role in a sentence: car and truck, big and large, doctor and nurse. Related words belong to the same topic or situation but play different roles, such as doctor and hospital or coffee and mug.

Opposites are a tricky case. "Hot" and "cold" fill the same slot ("the tea is hot", "the tea is cold"), so they count as similar even though their meanings contrast.

The difference shows up in practice. A search for words similar to "doctor" should return "nurse", while a search for related words can also return "hospital".

Decision 3: how is the sequence arranged?

Text examples vary in length

A table gives every row the same number of columns, and text gives you nothing of the kind. A product review can be one word or four hundred, and both are valid examples of the same task.

Many models expect inputs of a fixed size. So a system that turns text into rows of numbers has to decide what to do about that difference in length.

Order carries meaning

Knowing every word in a sentence is not enough to know what it says. "The dog chased the cat" and "the cat chased the dog" use exactly the same words, and the order alone decides who did the chasing.

The rules for how words combine into sentences are called syntax. They are what make "the dog chased the ball" fine and "ball the chased dog the" nonsense.

A model that only counts which words appear cannot tell the two chasing sentences apart, because counting throws the order away:

1from collections import Counter
2 
3s1 = "the dog chased the cat"
4s2 = "the cat chased the dog"
5 
6print(Counter(s1.split()))
7print(Counter(s1.split()) == Counter(s2.split()))
8print(s1.split() == s2.split())

1Counter({'the': 2, 'dog': 1, 'chased': 1, 'cat': 1})
2True
3False

The word counts match, and the word sequences do not.

Linked words can sit far apart

Order also links words that sit far apart. Read this one through: "Maria filed the report on Tuesday after three weeks of interviews, a delayed lab result, and one round of legal review, and she still has not heard back."

The word "she" refers to Maria. Twenty-one words sit between them, and a reader connects the two anyway.

There is no ceiling on that distance. Another clause could add thirty more words before the pronoun turns up, and the name could have been introduced a page earlier. So a method that only looks at the last handful of words will sometimes have nothing to attach "she" to, because the word it depends on is already out of range.

Common mistakes when starting with text data

A common mistake is treating "a word" as a settled unit. Splitting on spaces counts "ice cream" as two words and "#NLProc" as one. A different splitting rule gives a different vocabulary, so it is worth choosing the rule on purpose and using the same rule everywhere the text is processed.

Another is counting words and expecting the counts to carry meaning they cannot hold. Counts throw away order, so "the dog chased the cat" and "the cat chased the dog" look identical. Counts also say nothing about whether "cat" is closer to "dog" than to "helicopter".

A third is checking only short examples. A method can look fine on short sentences and still fail on long ones, where the words that need to be linked sit far apart.

Related concepts

Underneath every character in a text file there is another layer of numbers. Unicode and UTF-8 explained covers how a computer stores each character as bytes and why collected text sometimes comes out garbled. For what NLP is and the tasks it covers, see What is NLP?.

QuiddityML teaches all of this in the text-as-data unit of its Intro to NLP track, and the exercises include labeling words in a sentence as tokens or types and picking out word pairs that are related but not similar.