By Sagi Shaier · 28 October 2026 · 10 min read

Byte pair encoding (BPE) explained step by step, including byte-level BPE

How BPE builds a subword vocabulary from counts alone, a worked example you can check by hand, the three short Python functions behind it, and why byte-level BPE never needs an unknown token.

Byte pair encoding (BPE) is a way to build a tokenizer's vocabulary by starting from single characters and repeatedly merging the pair of pieces that appears side by side most often. It is one of the most widely used subword tokenization algorithms, and its byte-level version is the tokenizer behind models such as GPT-2 and RoBERTa.

What problem BPE solves

A model reads text as a list of integer ids, one per token, and the list of every token a tokenizer can hand back is its vocabulary. Whole words make a poor vocabulary, because new words keep arriving and a word that is not on the list has to become a single unknown token. Single characters make a list you can finish, but each one carries almost no meaning and the same text takes many more ids.

Subword tokenization sits between the two. It keeps common strings whole and cuts rare ones into common parts, so playing becomes play + ing and replayed becomes re + play + ed. That leaves one question open: how does a tokenizer decide which pieces to keep? BPE answers it with counting.

Where the name comes from

BPE was first published as a data-compression method. It found the most frequent pair of bytes in a file and replaced every occurrence with a byte value the file did not use. Tokenization borrowed that merge loop and kept the name.

The BPE loop

BPE starts from single characters, then repeats a count and a merge until the vocabulary reaches the size you picked:

Start with the text split into single characters.
Count every adjacent pair.
Merge the most frequent pair everywhere it appears, and add the merged piece to the vocabulary.
Go back to step 2. Stop once the vocabulary is the size you picked.

The BPE loop drawn as start from characters, count the pairs, merge the top pair to add one new entry, count again on the merged pieces and stop at the target size

The same loop runs in both of BPE's lives. The compression version starts from bytes, written below in hexadecimal, where 62 is the byte for b and 61 the byte for a. In banana the pairs 61 6E (an) and 6E 61 (na) both appear twice, and the first of them is replaced by the unused byte FF. The text version starts from characters, with _ standing for a space, and the merged pair becomes a new token instead of a spare byte.

BPE as compression on the bytes of banana, where the pair 61 6E is replaced by the unused byte FF, next to BPE as tokenization on lower newer, where the pair w e becomes the new token we

A worked example

Take a tiny corpus. All the trainer needs is how often each word appears, so write it as a word list with counts:

1low     x5
2lower   x2
3newer   x6
4wider   x3

Split every word into characters, then count each adjacent pair across the whole corpus. A pair inside a word that appears 6 times counts 6:

1pair      count      from
2(e, r)      11     lower 2 + newer 6 + wider 3
3(w, e)       8     lower 2 + newer 6
4(l, o)       7     low 5 + lower 2
5(o, w)       7     low 5 + lower 2
6(n, e)       6     newer 6

The other four pairs count 6 or less, so none of them can win this round. (e, r) wins and merges into the token er, which goes on the list. Every word holding an e followed by an r now uses it:

1low     ->  l o w
2lower   ->  l o w er
3newer   ->  n e w er
4wider   ->  w i d er

Then count again on the new pieces. (w, er) is now on top at 8 (lower 2 + newer 6), so wer merges:

1low     ->  l o w
2lower   ->  l o wer
3newer   ->  n e wer
4wider   ->  w i d er

Count again and (l, o) leads at 7, giving lo. One merge per round, and the trainer keeps going until the list is the size you asked for.

The word lower going from l o w e r to l o w er after the first merge and to l o wer after the second

What the merges build

Each merge adds exactly one new token to the vocabulary. In this simple version the starting vocabulary is the set of distinct characters, which is small for English and grows large once you include the characters of every script.

Look at what the first merge picked. er ends three of the four words, so it went first, and now all three share that token. Nobody told the trainer that er is an English suffix. It found that from the counts alone.

The words low, lower, newer and wider with their counts, where the first merge gives lower, newer and wider one shared er token

The pattern holds well beyond this corpus. Frequent pieces of words become single tokens, and rare combinations stay split into smaller pieces.

Training vs encoding

BPE has two separate jobs.

The trainer runs once, on a big corpus. It counts adjacent pairs, merges the most frequent one and writes that merge down. Repeating this gives an ordered list of merge rules plus the vocabulary they build. There is no neural network and no gradient in it, only counting. After training, the merge list is fixed.

The encoder runs every time text gets tokenized: during model training, during evaluation and on every prompt afterwards. It does not recount anything. It takes the merge list from the trainer and replays those merges, in the same order, on the new text. The same rules give the same result every time.

The trainer running once on a corpus to produce five ordered merge rules, t h to th, th e to the, i n to in, e r to er and l o to lo, and the encoder applying them in order to the lower inn to get the, lo, w, er, in and n

Here is the encoder in code for those five merge rules. merge_pair is defined in the Python section below:

1merges = [('t', 'h'), ('th', 'e'), ('i', 'n'), ('e', 'r'), ('l', 'o')]
2 
3def encode(word, merges):
4    pieces = list(word)
5    for pair in merges:
6        pieces = merge_pair(pieces, pair)
7    return pieces
8 
9for word in "the lower inn".split():
10    print(word, encode(word, merges))
11# the ['the']
12# lower ['lo', 'w', 'er']
13# inn ['in', 'n']

In lower, rule 4 turns e r into er and rule 5 turns l o into lo. No rule joins w with anything, so it stays on its own.

Where words end

None of the pieces so far record where a word ended. A tokenizer hands back a flat list of pieces with no spaces in it. If you get back low and est, the original text could have been lowest as one word or low est as two, and the pieces look identical either way.

So tokenizers write the boundary into the tokens themselves. One common way is an end-of-word symbol added to every word before training:

1low     -> ['l', 'o', 'w', '</w>']
2lower   -> ['l', 'o', 'w', 'e', 'r', '</w>']

The pieces low and est that could come from lowest or low est, and the word low written as l, o, w and an end-of-word token so the ids carry the spaces

Other tokenizers do the same job from the other side, by attaching the space in front of a word to the word itself. A word at the start of a line and the same word after a space then become two separate vocabulary entries with two different ids.

BPE in Python

BPE comes down to two short functions and a loop around them.

Counting adjacent pairs

The first function takes a list of tokens and reports how often each adjacent pair shows up. It walks the list from left to right, takes each token with the one after it and adds one to that pair's count:

1def get_pair_counts(tokens):
2    counts = {}
3    for i in range(len(tokens) - 1):
4        pair = (tokens[i], tokens[i + 1])
5        counts[pair] = counts.get(pair, 0) + 1
6    return counts
7 
8print(get_pair_counts(['l', 'o', 'w', 'e', 'r']))
9# {('l', 'o'): 1, ('o', 'w'): 1, ('w', 'e'): 1, ('e', 'r'): 1}

range(len(tokens) - 1) stops one position short of the end, since the last token has nothing after it to pair with. counts.get(pair, 0) returns the stored count for pair, or 0 the first time that pair appears.

The tokens l, o, w, e and r with the four adjacent pairs bracketed underneath and a table giving each pair a count of 1

Nothing in this function knows what a word is or what English looks like. Every decision BPE makes comes out of these counts.

Merging one pair

The second function takes a list of tokens and one pair, and returns the list with every occurrence of that pair glued into a single token:

1def merge_pair(tokens, pair):
2    merged = []
3    i = 0
4    while i < len(tokens):
5        if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == pair:
6            merged.append(tokens[i] + tokens[i + 1])
7            i += 2
8        else:
9            merged.append(tokens[i])
10            i += 1
11    return merged
12 
13print(merge_pair(['l', 'o', 'w', 'e', 'r'], ('e', 'r')))
14# ['l', 'o', 'w', 'er']
15print(merge_pair(['l', 'l', 'l'], ('l', 'l')))
16# ['ll', 'l']

When the current token and the next one match the pair, the function adds the two joined together and moves i forward by two. Otherwise it adds the token as it is and moves on by one. A while loop is used because i has to move by different amounts.

Skipping by two decides the second example. Merging ('l', 'l') in ['l', 'l', 'l'] gives ['ll', 'l'] and not ['ll', 'll'], because the middle l was used up by the first merge. The two tokens also have to sit side by side. An e early in the list and an r late in it are not a pair.

Merging e and r in l o w e r to get l o w er, and merging l and l in l l l to get ll and l, where the middle token was already used

The loop

run_bpe repeats the same round num_merges times. It counts the pairs, takes the most frequent one and merges it, then carries the merged list into the next round:

1def run_bpe(tokens, num_merges):
2    for _ in range(num_merges):
3        counts = get_pair_counts(tokens)
4        if not counts:
5            break
6        best_pair = max(counts, key=lambda p: counts[p])
7        tokens = merge_pair(tokens, best_pair)
8    return tokens
9 
10print(run_bpe(list("abababc"), 2))
11# ['abab', 'ab', 'c']

max(counts, key=lambda p: counts[p]) returns the pair with the highest count. When everything has merged into one token there are no pairs left, so counts comes back empty and break ends the loop early. In the example, round 1 merges (a, b), which appears 3 times. Round 2 merges (ab, ab), which appears twice in ab ab ab c.

The loop count pairs, choose the max and merge the pair, where each round adds one vocabulary entry and the next round recounts the merged pieces

The counts are recomputed every round, because merging changes which tokens sit next to which. And num_merges sets the vocabulary size, since each round adds one entry.

Training on the corpus with counts

run_bpe works on one flat list and returns only the merged tokens. A real trainer counts pairs across a corpus of words and keeps the ordered merge list for the encoder. Both changes fit in a few lines on top of the two functions above:

1corpus = {"low": 5, "lower": 2, "newer": 6, "wider": 3}
2words = {word: list(word) for word in corpus}
3merges = []
4 
5for round_number in range(3):
6    counts = {}
7    for word, pieces in words.items():
8        for pair, n in get_pair_counts(pieces).items():
9            counts[pair] = counts.get(pair, 0) + n * corpus[word]
10    best_pair = max(counts, key=lambda p: counts[p])
11    merges.append(best_pair)
12    words = {word: merge_pair(pieces, best_pair) for word, pieces in words.items()}
13    print(round_number + 1, best_pair, counts[best_pair])
14 
15print(merges)

11 ('e', 'r') 11
22 ('w', 'er') 8
33 ('l', 'o') 7
4[('e', 'r'), ('w', 'er'), ('l', 'o')]

Each pair count is multiplied by how often its word appears, and pairs are never counted across two words. The three rounds pick er, wer and lo with counts 11, 8 and 7. That matches the worked example by hand.

Byte-level BPE

Character-level BPE has a catch. Its starting vocabulary has to include every character that might appear, and Unicode has over 150,000 assigned characters across its scripts and symbols. Starting from all of them gives a huge list, and a character missing from the training text still has nothing to fall back to.

Byte-level BPE runs the same algorithm over raw bytes instead of characters. Text is stored as bytes, and there are only 256 possible byte values, so the starting vocabulary is small and fixed. Every character is one or more bytes, so nothing is ever out of vocabulary. In the worst case a strange character falls back to its individual bytes:

1for text in ["cat", "é", "🙂"]:
2    data = text.encode("utf-8")
3    print(text, list(data), len(data))
4# cat [99, 97, 116] 3
5# é [195, 169] 2
6# 🙂 [240, 159, 153, 130] 4

encode("utf-8") turns a string into its bytes. The letters of cat take one byte each, é takes two and the emoji takes four.

The text cat turned into the bytes 99, 97 and 116, then BPE merging 99 and 97 into one token, with only 256 base symbols so nothing is out of vocabulary

The trade-off is length. A rare character or an emoji is several bytes, so it can turn into several byte tokens when no merge covers it. One glyph on screen can cost three or four tokens, which makes text outside the common cases longer and more expensive to process.

Common mistakes

Merging pairs that are not adjacent. A pair is two tokens side by side. An e and an r with other tokens between them do not count.
Moving one step after a merge. After joining two tokens the scan has to skip both, or the second token gets used twice.
Counting pairs once at the start. Each merge changes the neighbours, so the counts have to be recomputed every round.
Recounting at encoding time. The encoder replays the saved merge list in order. Counting pairs on new text would give different merges and different ids from the ones the model was trained on.
Counting characters to estimate cost. With byte-level BPE, accented letters, other scripts and emoji can cost several tokens each.

BPE is one of several ways to build a subword vocabulary. WordPiece runs a similar merge loop with a different rule for choosing the pair, and Unigram starts from a large set of candidate pieces and removes them until the list is the size you asked for. The post on tokenization covers why subword tokens beat whole words and single characters, and the post on Unicode and UTF-8 covers how characters turn into the bytes that byte-level BPE reads.

QuiddityML teaches BPE as its own concept in the NLP track's tokenization unit. Its exercises include filling in get_pair_counts, putting the lines of merge_pair in order and writing run_bpe from scratch, plus spotting the bug in a broken merge loop (QuiddityML).