By Sagi Shaier · 28 October 2026 · 16 min read

What is tokenization in NLP? Word, character, and subword tokens

Why a language model needs text cut into pieces first, the three questions that decide how big a piece should be, and why most models today use subword tokens instead of whole words or single characters.

Tokenization is splitting text into the pieces a model works with. Those pieces are called tokens, and the full list of tokens a tokenizer knows is its vocabulary. A language model sits behind its tokenizer, so the way text gets cut decides what the model can see, how long its inputs get and what happens to a word it has never met.

Why text has to become numbers

A neural network does arithmetic. It multiplies tables of floating-point numbers over and over. A string of characters is not something you can multiply, so before a model touches any text, the text has to become numbers.

That conversion has two steps. First the text is cut into pieces, then every piece is swapped for an integer id. The second step is a lookup in a table that pairs each piece with its id. Inside the model, each id then picks out a row of learned numbers, and those rows are what the model computes with. The model never receives the original string.

The first step is where the design decisions are, and it is what the rest of this post is about.

Tokens and the vocabulary

Run a tokenizer over The cat sat. and you get back something like ['The', 'cat', 'sat', '.']. Each token is swapped for an integer id, and that list of ids is what the model reads.

In linguistics a token is each occurrence of a word in running text. A tokenizer's tokens are the same idea made concrete: whichever pieces this particular tokenizer decided to cut. They are often not words.

The tokenizer sits in front of the model, so everything downstream works on its output. Whatever it threw away is gone for good, and whatever split it chose is the shape the model has to learn around.

There is no single right tokenization

Hand the same sentence to two tokenizers and they will often split it differently, with neither one wrong. Is don't one token or two? Does state-of-the-art stay whole or break at the hyphens?

One tokenizer might split don't into do and n't. The n't piece is a clitic, a short form that cannot stand alone and attaches to the word before it. That same tokenizer might keep state-of-the-art as one token. Another tokenizer might keep don't whole and break state-of-the-art at every hyphen.

Each tool commits to conventions that suit what it was built for. Token boundaries are a design choice, so when you compare two tokenizers or debug a pipeline, expect the splits to differ and check which conventions yours follows. It also means a token count from one tokenizer cannot be compared with a token count from another.

How big should a token be?

There is a range of sizes to pick from. At the coarse end each sentence is one unit. Below that come words, then single characters, then the bytes underneath the characters.

Three questions tell you what each choice costs:

1. How big is the vocabulary? The tokenizer keeps one list of every piece it is allowed to hand back, and each piece on that list gets a number. That list has to be fixed before the model trains, so the two things to ask are how long it is and whether you can ever finish writing it.
2. Do related units share anything? play and playing are the same word with an ending added, so the model should see that they go together. The model sees only the ids it is handed. If play and playing come back as two unrelated ids, nothing in the input says they belong together.
3. How long does the text get? Length is measured in ids, since ids are what the model reads. The cat sat. costs 4 ids as words and 12 as characters. Every extra id is more work for the model.

The first two questions conflict. The choices with small vocabularies you can finish writing are the ones whose units carry the least meaning on their own.

Sentences as tokens

Start at the coarse end and cut the text wherever a sentence ends, so each sentence is one token.

How big is the vocabulary? It has no limit. People write sentences that have never been written before all day long, so there is no list to write down. The entries you did write down would each show up about once in the whole corpus, which means the model would see each one a single time.

Do related units share anything? No. The cat sat on the mat. and The cat slept on the rug. are nearly the same sentence, and they get two unrelated ids. The model cannot even see the words the two have in common.

How long does the text get? Very short, which is the one question sentences win. A whole paragraph is a handful of ids, and that does not buy much when the ids carry nothing the model can use.

Practical NLP tokenizers are not built this way. Sentences mark the coarse end of the range, and every step down from here fixes something.

Finding where a sentence ends

Cutting at sentence ends assumes you can find them. That task has its own name, sentence segmentation.

A full stop looks like a reliable marker until you count how often it is something else. Dr. Chen paid 3.14 for it. Then she left. has four full stops and two sentences. The one after Dr ends an abbreviation and the one in 3.14 sits inside a number.

The usual approach is a list plus a few rules. Keep a list of known abbreviations. Then treat a full stop as a sentence boundary only when the word in front of it is not on that list and the next word starts with a capital letter:

1ABBREVIATIONS = {"Dr.", "Mr.", "Mrs.", "e.g.", "etc.", "U.S."}
2 
3def split_sentences(text):
4    words = text.split(" ")
5    sentences, current = [], []
6    for i, word in enumerate(words):
7        current.append(word)
8        is_last = i == len(words) - 1
9        next_is_capital = not is_last and words[i + 1][:1].isupper()
10        if word.endswith(".") and word not in ABBREVIATIONS and (is_last or next_is_capital):
11            sentences.append(" ".join(current))
12            current = []
13    if current:
14        sentences.append(" ".join(current))
15    return sentences
16 
17print(split_sentences("Dr. Chen paid 3.14 for it. Then she left."))
18# ['Dr. Chen paid 3.14 for it.', 'Then she left.']

The word Dr. is on the list, so its full stop is skipped. The word 3.14 does not end with a full stop, so it never counts as a boundary.

Trained splitters do the same job by learning from text where people marked the boundaries by hand. Few models tokenize at sentence level, so this step rarely feeds a tokenizer. It is still needed for anything that works one sentence at a time, such as translation systems and any corpus you want to count by sentence.

Words as tokens

One step down are words, which is where most people's intuition starts. Counting text in words is the ordinary way to measure it, and a word carries meaning on its own in a way a whole sentence does not.

The method is short. Break on whitespace and pull the punctuation off:

1import re
2 
3text = "The cat sat."
4words = re.findall(r"\w+|[^\w\s]", text)
5chars = list(text)
6print(words, len(words))
7print(chars, len(chars))
8# ['The', 'cat', 'sat', '.'] 4
9# ['T', 'h', 'e', ' ', 'c', 'a', 't', ' ', 's', 'a', 't', '.'] 12

The pattern \w+|[^\w\s] matches either a run of letters and digits or a single punctuation mark. The second line splits the same text into characters for comparison.

There are small calls to make along the way. Does don't stay whole or become do and n't? Is New York one token or two? Real word tokenizers answer these one case at a time, and different ones answer differently.

How long does the text get? About one id per word, roughly as few as you can get while still cutting on meaning. That is the question words win. The other two are where they lose.

The vocabulary never closes: Heaps' law

How big is the vocabulary? Large, and it keeps getting larger. Read a paragraph and you meet a few words you have not seen. Read a book and you meet more. The number of distinct words you have seen keeps climbing as you read.

Heaps' law describes that growth. Let $N$ be the number of words read so far, counting repeats, and $V$ the number of distinct words among them (the word types). Then

$$V = K \cdot N^{\beta}$$

where $K$ is a constant and $\beta$ sits between 0 and 1. The same relationship is also called Herdan's law.

As an example, take $K = 10$ and $\beta = 0.5$. After reading $N = 10{,}000$ words the vocabulary is $V = 10 \cdot 100 = 1{,}000$ types. After $N = 1{,}000{,}000$ words it is $V = 10 \cdot 1{,}000 = 10{,}000$ types. A hundred times more text gives ten times more types.

Because $\beta < 1$, each new chunk of text adds fewer new types than the chunk before it. The curve never flattens out, though, so there is always another new word coming. A word vocabulary for English runs to hundreds of thousands of entries, and any list you write down covers only the text you happened to read.

Which words keep arriving

Not every word feeds that growth. The vocabulary splits into two groups.

The closed class holds structural words like for, but and with. These are also called function words. A language has only so many of these, and you meet nearly all of them within a page or two.

The open class holds the content words: nouns, verbs, adjectives and adverbs. It keeps taking on new members. Every new brand name, piece of slang, borrowed word or bit of jargon lands here.

Heaps' law is really about the open class. The closed class fills up in a page or two and stops growing.

Most of the vocabulary is rare: Zipf's law

The entries you already have cause a second problem. Zipf's law says a word's frequency is roughly inversely proportional to its rank in the frequency table. With $r$ the rank (1 for the most common word), $f(r)$ the number of times that word appears and $C$ the count of the most common word,

$$f(r) \approx \frac{C}{r}$$

So if the most common word appears 60,000 times in a corpus, the second appears about 30,000 times and the third about 20,000.

Plot frequency against rank and you get a steep drop followed by a long, flat tail. Most of the vocabulary lives in that tail, as words that show up only a few times in the whole corpus. A rare word takes an entry of its own and might appear five times in the entire training set. Five examples are not enough to learn what a word means, so most of a word vocabulary is entries the model learns very little about.

Related words share nothing

Do related units share anything? No. Take play and four words built on it: plays, playing, played and replayed. All five share the same root, and a word vocabulary gives them five separate ids. To the model, id 4382 and id 8871 have nothing to do with each other.

Suppose play appears 40,000 times in your corpus and replayed 40 times. None of those 40,000 examples help with the 40, even though the two words mean almost the same thing. Many words are built from pieces that come back over and over, such as -ed, -ing and re-. Word tokens do not see any of them.

The problem grows in languages like Turkish or Finnish, which attach many endings to one root and get thousands of word forms from it. Each form needs its own id, and most of them are rare.

Where that leaves word tokens

Words lose on two of the three questions. The vocabulary is large and never closes, and related forms share nothing. Only length comes out fine.

That does not make word tokens useless. Classical NLP ran on them for decades and many systems still do, because a word vocabulary is easy to build and easy to read. The first problem does have a consequence you cannot design around, though. At some point the list has to be frozen, and text that arrives after that will contain words that are not on it.

Out-of-vocabulary words and the [UNK] token

The vocabulary is frozen when the tokenizer is built. From then on the tokenizer can only produce tokens on the list, because ids exist only for entries on the list.

An out-of-vocabulary (OOV) word is a word that is not on it. Heaps' law says new words keep arriving, so this is not a rare accident. Users of a word-level model will type words it has never seen on the first day, and the tokenizer has to return something for them.

So a fixed-vocabulary model keeps one entry in reserve and maps every unknown word to it. That entry is the unknown token, written [UNK]:

1import re
2 
3def word_tokenize(text):
4    return re.findall(r"\w+|[^\w\s]", text.lower())
5 
6train_text = "the cat sat on the mat . the dog sat on the rug ."
7vocab = {"[UNK]": 0}
8for word in word_tokenize(train_text):
9    if word not in vocab:
10        vocab[word] = len(vocab)
11print(vocab)
12 
13def encode(text):
14    return [vocab.get(word, vocab["[UNK]"]) for word in word_tokenize(text)]
15 
16print(encode("The cat sat on the rug."))
17print(encode("The axolotl sat on the sofa."))

1{'[UNK]': 0, 'the': 1, 'cat': 2, 'sat': 3, 'on': 4, 'mat': 5, '.': 6, 'dog': 7, 'rug': 8}
2[1, 2, 3, 4, 1, 8, 6]
3[1, 0, 3, 4, 1, 0, 6]

The vocabulary is built from the training text, with id 0 kept for [UNK]. The first sentence uses only known words. In the second, axolotl and sofa were never seen, and both come back as id 0.

Nothing crashes, but the cost is steep. [UNK] throws away the content of the word, so the model cannot tell axolotl from sofa and cannot use anything about what either one means. OOV words come from new coinages, names and typos, and from word forms that happen to be missing from the list. [UNK] is one of a few tokens that are not words at all, alongside markers such as [PAD], [CLS] and [SEP].

An open vocabulary is the ability to represent any input, including a word never seen in training, without falling back to [UNK]. That does not need an endless list. It needs pieces small enough that a new word can be built out of pieces the tokenizer already has. A word is either on a whole-word list or it is not, while smaller pieces can be put together into words the list never held.

Characters as tokens

Jump to the fine end of the range and take every character as its own token. The cat sat. becomes ['T', 'h', 'e', ' ', 'c', 'a', 't', ' ', 's', 'a', 't', '.'], with the spaces counted as characters too.

How big is the vocabulary? Around a hundred entries covers written English with its punctuation and digits. More useful than the size is that you can write down every character in a script and be done, which was never true of a word list. Scripts with large character sets need more, a few thousand for Chinese, and that is still a list you can finish.

Because the list is complete, out-of-vocabulary words stop happening. A word nobody has written before is still made of characters the tokenizer has, so nothing needs [UNK]. All the judgment calls disappear too. There are no contractions to split and no hyphens to rule on, because the tokenizer splits everywhere.

Characters win the vocabulary question by giving up the other two.

Do related units share anything? Yes, but not in a way that helps. play and playing share four characters, but play and plywood start with the same two letters and mean nothing to each other. The letter c on its own does not mean anything. It turns up in cat, cost and chrysanthemum with nothing in common between them, so the model has to rebuild every word from its letters during training.

How long does the text get? The cat sat. is 4 ids as words and 12 as characters. Across real text it often comes out several times longer, since each word becomes its letters plus any spaces and punctuation.

The extra ids cost you later. cat and sat sit side by side as words, and as characters their first letters are 4 ids apart. Words that a sentence ties together are often several words apart already, and character tokens multiply that distance. The further apart two tokens are, the more work it takes the model to connect them.

So both ends of the range fall short, for opposite reasons. Words carry meaning, but their list never closes. Characters close the list, but each unit carries almost no meaning and the same text takes far more ids.

Subword tokenization

Sentence, word and character tokenizers each fail at least one of the three questions. Subword tokenization uses pieces of different sizes depending on the string. It keeps common strings whole and cuts rare ones into common parts:

1"play"     -> "play"
2"playing"  -> "play" + "ing"
3"played"   -> "play" + "ed"
4"replayed" -> "re" + "play" + "ed"

Do related units share anything? Now they do. All four words share the play token, so what the model learns about one counts toward the others. The pieces line up with roots, prefixes and endings even though nobody told the tokenizer what any of those are. They come from keeping the pieces that show up often.

How big is the vocabulary? You set it. A common way to build a subword vocabulary runs a procedure over a big pile of text. It starts from single characters, then keeps gluing together the pair of pieces that appears side by side most often, adding each new piece to the list. Each round adds one entry, so you stop once the list is as long as you want.

For that kind of procedure, picking a vocabulary size means picking how many rounds to run. Stop early and the list holds short, generic pieces that cut most words up. Let it run longer and common whole words survive as single entries on a longer list. 30,000 to 50,000 entries is a common range, and some models go past 100,000. Other procedures work the other way round, starting from a large set of candidate pieces and cutting it down to the length you asked for. Either way the size is a setting you choose.

A word the tokenizer has never seen still gets represented, because it falls back to smaller pieces it does know, down to single characters if it has to. A particular tokenizer might split examples like this:

1"anthropic" -> "anthrop" + "ic"
2"COVID"     -> "COV" + "ID"

So unknown words no longer need [UNK], which meets the open-vocabulary requirement.

Characters the tokenizer never saw can still fail. The pieces bottom out at the characters seen in training, so an emoji or a script that never appeared has nothing to fall back to and still becomes [UNK]. Running the same procedure over raw bytes instead of characters fixes that part, since every character is stored as a few bytes from a fixed set of 256.

How long does the text get? Close to one token per word for ordinary text, since common words survive whole. Rare words get cut into several pieces, which is where extra ids are worth spending.

Subwords answer all three questions, which is why most models built today tokenize this way.

Which tokenizer to use

• Training or fine-tuning a modern language model: use the subword tokenizer that came with the model. The model's learned rows of numbers match that tokenizer's ids, so swapping tokenizers breaks the pairing.
• Counting words or building a classical model: a word tokenizer is a reasonable choice, since the vocabulary is easy to read and the results are easy to check.
• Working one sentence at a time: run sentence segmentation first, then tokenize each sentence.

Common mistakes

• Assuming tokens are words. A subword tokenizer can cut one word into several tokens, so a limit stated in tokens is not a limit in words.
• Comparing token counts across tokenizers. Two tokenizers split the same text differently, so their counts measure different things.
• Ignoring how many inputs become [UNK]. With a word vocabulary, check what share of your test text maps to [UNK]. Every one of those tokens is a word the model cannot read.
• Splitting sentences on every full stop. Abbreviations and numbers contain full stops too, as Dr. Chen paid 3.14 shows.

Related concepts

Subword tokenizers differ in how they decide which pieces to keep. The post on byte pair encoding (BPE) builds one step by step, from counting adjacent pairs to the version that runs over bytes. The post on Unicode and UTF-8 covers how characters are stored as bytes, which is what byte-level tokenizers read. The post on text preprocessing covers lowercasing and other normalization that often runs before tokenization.

QuiddityML gives each kind of token in this post its own concept in the NLP track's tokenization unit, and the exercises on them include working through Heaps' law and tracing how many ids a sentence costs under word and subword tokens (QuiddityML).