By Sagi Shaier · 29 October 2026 · 6 min read

Why LLMs miscount letters: tokenization quirks and non-English text

What GPT-2's tokenizer does to a word, a number and an emoji, how many tokens one sentence takes in four languages, and why some tokens make a model behave strangely.

A large language model (LLM) does not read letters. Before any text reaches the model, a tokenizer cuts it into tokens and replaces each one with an integer id. A token is often a whole word or a piece of one. The model only ever sees those ids. That one step explains why a model can stumble on counting the letters in a word or on adding two long numbers, and why a sentence in Hindi can cost five times as many tokens as the same sentence in English.

What the model sees when you type a word

Most tokenizers used with LLMs are subword tokenizers. They learn a vocabulary, a fixed list of pieces, from a large body of text. Common strings stay whole and rare ones get cut into smaller pieces. GPT-2's tokenizer is a well-documented example, and Hugging Face's transformers library can load it by name:

1from transformers import AutoTokenizer
2 
3tokenizer = AutoTokenizer.from_pretrained("gpt2")
4for word in [" strawberry", "strawberry", " STRAWBERRY"]:
5    print(repr(word), tokenizer.tokenize(word))

1' strawberry' ['Ġstrawberry']
2'strawberry' ['st', 'raw', 'berry']
3' STRAWBERRY' ['ĠST', 'RAW', 'BER', 'RY']

The Ġ marks a space in front of the word. In the middle of a sentence, strawberry has a space before it and arrives as one token, which becomes one id. Nothing in that id spells out the ten letters s-t-r-a-w-b-e-r-r-y. To count the rs, a model would have to have learned the spelling of that token from its training text, because the letters are not in its input. That is one reason models get questions like "how many r's are in strawberry?" wrong.

The same word also splits three different ways depending on a space and on capital letters. These splits come from how the tokenizer was trained on its corpus, not from any rule about words.

Where tokenizers break down

Tokenizers are trained mostly on English-heavy text, and the effects show up in a few concrete places:

Text without spaces gives the first splitting step nothing obvious to cut on.
Some languages break into many more tokens than English for the same meaning.
Small changes, like a leading space or a capital letter, change how text splits.

Three panels, Japanese text with no spaces where pretokenization has nothing to split on, an English sentence of 8 tokens next to another script needing 16 tokens for the same meaning, and cat versus space then cat giving different tokens

None of these are bugs in the code. They follow from what the tokenizer saw during training, and they still cause real problems: higher cost for some languages and models that fumble tasks that look easy.

Numbers, spaces and capital letters

A long number is cut into chunks that follow the training data, not the digits:

1print(tokenizer.tokenize("1234567890"))
2print(tokenizer.tokenize("1234 + 5678"))

1['123', '45', '678', '90']
2['12', '34', 'Ġ+', 'Ġ5', '678']

1234 arrives as 12 and 34, while 5678 arrives as 5 and 678. The two numbers are cut at different places, so the digits that should line up for addition sit in differently shaped tokens. That makes arithmetic harder for a model reading tokens, since it does not see the columns of digits that a person lines up on paper.

Spaces and capitals change the tokens too. cat on its own is id 9246, while cat with a space in front is a different token, id 3797. hello is one token, and HELLO is three.

Three panels on GPT-2, the number 1234567890 split into 123, 45, 678 and 90, cat as id 9246 versus space then cat as Ġcat with id 3797, and hello as one token versus HELLO as HE, LL and O

A single emoji can take several tokens:

1print(tokenizer.tokenize("🤔"))

1['ðŁ', '¤', 'Ķ']

GPT-2's tokenizer works on bytes, and this emoji is 4 bytes in UTF-8, the standard way to store text as bytes. No merge in GPT-2's vocabulary covers all four, so it comes out as three byte-level tokens, which print as odd-looking characters.

Why non-English text costs more tokens

Token inflation is when the same meaning takes many more tokens in one language than in another. It happens to languages that build long words out of many pieces and to any language the tokenizer saw little of during training. Here is one sentence in four languages, counted with GPT-2's tokenizer:

1sentences = {
2    "English": "Good morning, how are you?",
3    "German": "Guten Morgen, wie geht es dir?",
4    "Hindi": "सुप्रभात, आप कैसे हैं?",
5    "Chinese": "早上好，你好吗？",
6}
7for language, text in sentences.items():
8    print(language, len(tokenizer(text)["input_ids"]))

1English 7
2German 12
3Hindi 35
4Chinese 17

The Hindi sentence takes 5 times as many tokens as the English one. The Chinese sentence has 8 characters and takes 17 tokens. GPT-2 is an older tokenizer trained mostly on English, and other tokenizers give different counts.

No spaces in a Chinese sentence where pretokenization has nothing to split on, and an illustration of fragmentation where an English sentence of four tokens becomes many more pieces in Turkish

Token inflation has two direct costs. More tokens means a higher cost per request. And a model's context window, the maximum number of tokens it can read at once, holds fewer actual words. A language that inflates is more expensive and gets less room to work with.

Languages without spaces

Chinese, Japanese and Thai do not separate words with spaces. A tokenizer usually starts with pretokenization, a rough split into word-like chunks, and for these languages there is no space to split on. The tokenizer has to find boundaries some other way, and it often does a rougher job than for English.

Chinese adds a second problem. Rival segmentation standards place the word boundaries in different spots, so there is no single agreed answer to learn from. Many systems skip word segmentation and cut the text into individual characters, the hanzi. That avoids the disagreement, and it works reasonably well because a hanzi often carries more meaning than one Latin letter does inside an English word.

The Chinese string 研究生命科学 with no spaces, split three ways by standard A, by standard B and into single characters, which is what many systems do since there is no authoritative segmentation

Long compound words

Spaces help, and they still leave the question open. German joins words into compounds with no space between them. Freundschaftsbezeigungen is one written word for displays of friendship: Freundschaft (friendship), a linking s, then Bezeigungen (displays).

The German word Freundschaftsbezeigungen made of Freundschaft, a linking s and Bezeigungen, next to the three English words displays of friendship that whitespace already separates

A split on spaces sees it as one chunk. A tokenizer then either stores the whole rare word as one vocabulary entry or breaks it into subword pieces. GPT-2 does the second, and the pieces do not follow the parts of the word:

1print(tokenizer.tokenize(" Freundschaftsbezeigungen"))
2print(tokenizer.tokenize(" displays of friendship"))

1['ĠFre', 'und', 's', 'cha', 'fts', 'be', 'ze', 'ig', 'ung', 'en']
2['Ġdisplays', 'Ġof', 'Ġfriendship']

The German word takes ten tokens and the English phrase takes three.

Glitch tokens

A rarer failure comes from a mismatch between two training sets. The tokenizer and the model can be trained on different text. A string that was frequent in the tokenizer's text can earn its own token and then appear rarely or not at all in the model's training text.

That token's row in the model barely changes during training, so it stays close to its random starting values. Feed the model that token and it can behave strangely: fail to repeat the string back, answer with something unrelated or produce nonsense. These are called glitch tokens.

A made-up string forum_user_88291 that is frequent in the tokenizer's corpus and almost absent from the model's corpus, so its embedding row stays near where it was initialized while ordinary tokens get many updates

Tokenizers rarely fail this way. The fix is on the data side: train the tokenizer on text that matches the model's training text as closely as possible.

What to do about it

Count tokens, not characters or words. Cost and context limits are set in tokens, and the ratio of tokens to words changes from language to language.
Check how your numbers tokenize before relying on a model for arithmetic or exact digits.
Watch leading spaces and capitals in prompts. cat and cat with a space in front are different tokens, so a stray space or an all-caps word changes what the model receives.
Budget for token inflation when a product serves several languages, since the same request can cost several times more in some of them.
Test with the tokenizer of the model you use. Splits differ from one tokenizer to another, so the GPT-2 counts here show the kind of split to expect rather than the exact one.

These quirks come from how subword vocabularies are built. The BPE post builds byte pair encoding from scratch, including the byte-level version GPT-2 uses, and the post on using a Hugging Face tokenizer shows how to load a tokenizer and inspect its pieces and ids. The post on Unicode and UTF-8 covers why one emoji is several bytes.

QuiddityML teaches these quirks in the NLP track's tokenization unit, and its questions include working out why a GPT model struggles with a sum like 1234 + 5678 and what token inflation does to cost (QuiddityML).