By Sagi Shaier · 29 October 2026 · 7 min read

BPE vs WordPiece vs Unigram tokenization

A worked example where BPE and WordPiece merge different pairs, how the ## marker works, and the score Unigram uses to choose between ways of cutting a word.

BPE, WordPiece and Unigram are three algorithms for building the vocabulary of a subword tokenizer. All three learn a list of word pieces from a body of text, and they differ in the rule that decides which pieces are worth keeping. That rule sets how every sentence a model reads gets cut up, and many pretrained models ship with a tokenizer built by one of them.

What a subword tokenizer has to decide

A model reads text as a list of integer ids, one per token. The list of every token a tokenizer can produce is its vocabulary. A subword tokenizer keeps common words whole and cuts rarer words into common parts, so playing might become play + ing.

Every subword algorithm has the same two stages. Training reads a corpus, a large collection of example text, and builds a vocabulary of pieces from it. Encoding then cuts any new text into pieces from that vocabulary. The open question is which pieces earn a place on the list, and each of the three algorithms answers it with a different rule.

BPE: merge the most frequent pair

Byte pair encoding (BPE) starts from single characters. It counts every pair of pieces that sit side by side in the corpus, merges the most frequent pair into one new piece, and adds that piece to the vocabulary. Then it counts again and repeats until the vocabulary reaches the size you asked for.

The only thing BPE looks at is the count of the pair. If t followed by h appears 6,000 times and no other pair comes close, th becomes a token. The step-by-step BPE post works through the whole loop on a small corpus.

WordPiece: merge the pair whose halves belong together

WordPiece runs the same loop as BPE. It starts from characters, merges one pair per round, adds the merged piece to the vocabulary and stops at the size you asked for. The difference is which pair it merges.

BPE takes the most frequent pair. WordPiece prefers a pair whose two halves show up together more often than their separate frequencies would explain. For a pair of pieces $A$ and $B$, let $\text{count}(AB)$ be how often the two appear side by side, and $\text{count}(A)$ and $\text{count}(B)$ how often each appears in total. WordPiece gives every pair this score and merges the pair with the highest one:

$$\text{score}(A, B) = \frac{\text{count}(AB)}{\text{count}(A) \cdot \text{count}(B)}$$

The top of the fraction is the number BPE already uses. The bottom divides by how common the two halves are on their own. Two pieces that are everywhere anyway get a large denominator and a low score. Two pieces that rarely appear apart get a high score, even when the pair itself is less frequent.

BPE and WordPiece side by side on the same corpus of lower, lowest and newer, both growing up from characters, where BPE picks the most frequent adjacent pair and WordPiece picks the highest count of AB over count of A times count of B

Here is the same idea on three pairs. BPE picks e s because 412 is the largest count. WordPiece picks q u, which appears only 90 times, because q rarely shows up without a u after it.

Three pairs e s, i n and q u, where BPE merges e s with the highest count of 412 and WordPiece merges q u with the highest score of 0.00009 because its halves rarely appear apart

A worked example

Say a corpus gives these counts for two candidate pairs:

1pair      count of the pair    counts of the halves
2(t, h)         6,000           10,000 and 8,000
3(n, g)         3,000            4,000 and 3,500

BPE merges (t, h), because 6,000 is larger than 3,000. WordPiece works out both scores first:

$$\text{score}(t, h) = \frac{6000}{10000 \cdot 8000} = 0.000075$$

$$\text{score}(n, g) = \frac{3000}{4000 \cdot 3500} \approx 0.000214$$

WordPiece merges (n, g). The t and the h are each so common that seeing them together 6,000 times is no surprise. The n and the g are rarer on their own, so their 3,000 appearances together count for more.

The same comparison in Python:

1pairs = {
2    ("t", "h"): {"pair": 6000, "first": 10000, "second": 8000},
3    ("n", "g"): {"pair": 3000, "first": 4000, "second": 3500},
4}
5 
6for pair, c in pairs.items():
7    bpe_score = c["pair"]
8    wordpiece_score = c["pair"] / (c["first"] * c["second"])
9    print(pair, bpe_score, f"{wordpiece_score:.6f}")
10 
11print("BPE picks", max(pairs, key=lambda p: pairs[p]["pair"]))
12print("WordPiece picks", max(pairs, key=lambda p: pairs[p]["pair"] / (pairs[p]["first"] * pairs[p]["second"])))

1('t', 'h') 6000 0.000075
2('n', 'g') 3000 0.000214
3BPE picks ('t', 'h')
4WordPiece picks ('n', 'g')

A table of the pairs t and h and n and g with the count of the pair, the counts of the halves and the WordPiece score, where BPE picks t and h and WordPiece picks n and g

Over a full training run, this score spends the vocabulary on pieces that hang together, such as ing or tion. Frequent letters that only happen to sit side by side get fewer merges.

The ## marker

WordPiece marks where a word continues. A piece that attaches to the piece before it carries a ## prefix, and a piece without one starts a new word. A WordPiece tokenizer might split two words like this:

1playing  ->  ['play', '##ing']
2unhappy  ->  ['un', '##hap', '##py']

So ing and ##ing are two separate entries in the vocabulary, one for the word ing on its own and one for the ending. To get the text back, join the pieces, drop the ## markers and put a space before every piece that had none.

WordPiece splitting playing into play and ##ing and unhappy into un, ##hap and ##py, where a piece with no ## can start a word and a piece with ## continues one

BERT, a widely used family of pretrained language models, ships with a WordPiece tokenizer. Hugging Face's transformers library can load it by name, and tokenize shows the pieces it picks:

1from transformers import AutoTokenizer
2 
3tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
4pieces = tokenizer.tokenize("the tokenizer handles unhappiness")
5print(pieces)
6print(tokenizer.convert_tokens_to_string(pieces))

1['the', 'token', '##izer', 'handles', 'un', '##ha', '##pp', '##iness']
2the tokenizer handles unhappiness

Tokenizer and unhappiness are not in this vocabulary as whole words, so they come out in pieces. convert_tokens_to_string joins the pieces and drops the markers.

Unigram: start big and prune

BPE and WordPiece both build up. They start from characters, glue pieces together and stop at the target size. Unigram goes the other way.

It starts with a large pile of candidate pieces, far more than it will keep, taken from common substrings of the corpus. Then it cuts the pile down:

Give every piece a probability, based on how much of the corpus it accounts for.
For each piece, work out how much worse the corpus gets explained if that piece is dropped.
Throw out the pieces whose removal costs least, usually around 10 to 20 percent of the list per round.
Recompute the probabilities and repeat until the vocabulary is the size you asked for.

The single characters are never dropped, so any input can still be cut into pieces from the list.

Two bar charts sharing one scale, where BPE and WordPiece grow from 8 percent up to a 60 percent target size and Unigram shrinks from 100 percent down to the same 60 percent, with single characters never dropped

The bigger difference is in what each method commits to. BPE and WordPiece keep every merge they make and never revisit it. Unigram judges each piece against the whole corpus every round, so it can drop a piece that looked useful earlier.

How Unigram picks a split

Because every piece has a probability, Unigram can score a whole way of cutting a word. If the pieces of one split are $x_1, x_2, \dots$ and $p(x_i)$ is the probability of piece $x_i$, the score of the split is the product of those probabilities:

$$P(\text{split}) = \prod_i p(x_i)$$

Take unhappy with these piece probabilities: $p(\text{un}) = 0.02$, $p(\text{happy}) = 0.01$, $p(\text{unh}) = 0.0004$ and $p(\text{appy}) = 0.0009$. Then the two splits score:

$$P(\text{un}, \text{happy}) = 0.02 \cdot 0.01 = 0.0002$$

$$P(\text{unh}, \text{appy}) = 0.0004 \cdot 0.0009 = 0.00000036$$

1import math
2 
3p = {"un": 0.02, "happy": 0.01, "unh": 0.0004, "appy": 0.0009}
4 
5def score(pieces):
6    return math.prod(p[piece] for piece in pieces)
7 
8print(score(["un", "happy"]))
9print(score(["unh", "appy"]))

10.0002
23.6e-07

The first split wins by a factor of more than 500. Cutting the word into its seven single letters is also valid, and it scores lowest of the three because it multiplies seven probabilities together. Unigram scores the possible splits and by default hands back the best one.

The word unhappy cut three ways, un plus happy with the highest score, unh plus appy lower, and seven single letters lowest, scored by multiplying the piece probabilities

Unigram can also sample a split instead of always taking the best. Showing a model slightly different splits of the same word during training makes it depend less on one exact segmentation. BPE and WordPiece have no equivalent, since their rules give one fixed split every time.

SentencePiece, a widely used tokenizer library, trains a Unigram tokenizer by default. Some models use Unigram and many others use BPE or WordPiece.

BPE vs WordPiece vs Unigram at a glance

Algorithm	How the vocabulary grows	What decides a piece	Split of one word
BPE	merges up from characters	highest pair count	one fixed split
WordPiece	merges up from characters	pair count over the halves' counts	one fixed split
Unigram	prunes down from many candidates	least loss when dropped	best split, or a sampled one

Which one should you use?

With a pretrained model the choice is already made. The model was trained on the ids of one specific vocabulary, so it needs the tokenizer it shipped with, whichever algorithm built it.

The choice only comes up when you train a tokenizer yourself, and the library you train with usually sets the default, such as Unigram in SentencePiece. Unigram's sampling is the one feature the other two lack. It helps when you want a model that depends less on one exact split of each word.

Common mistakes

Thinking WordPiece merges the most frequent pair. It merges the pair with the highest score, and a less frequent pair can win when its halves are rare on their own.
Treating ing and ##ing as one token. They are separate vocabulary entries with separate ids.
Decoding WordPiece pieces by joining them with spaces. That gives un ##ha ##pp ##iness. Drop the markers and glue each ## piece to the one before it.
Expecting Unigram to drop single characters. The base characters stay in the vocabulary, which keeps every input representable.
Assuming a sampled Unigram split is a bug. Sampling is a training option, and encoding normally returns the best-scoring split.

BPE is the base the other two are measured against, and the BPE post builds it in Python from scratch, including byte-level BPE. The post on tokenization covers why subword tokens are used instead of whole words or single characters.

QuiddityML teaches WordPiece and Unigram as two concepts in the NLP track's tokenization unit, and the exercises include picking the line of code that computes the WordPiece score and putting Unigram's pruning steps in order (QuiddityML).