By Sagi Shaier · 3 November 2026 · 6 min read

Perplexity explained

How perplexity is computed from a model's next-token probabilities, why it equals the exponential of the training loss, how to read a value such as 3.7 or 100, and how to compute it correctly over a whole test set in PyTorch.

Perplexity is the standard score for how well a causal language model predicts a piece of text. It measures how uncertain the model is, on average, about each next token. A perplexity of 10 means the model is about as uncertain as if it had to pick evenly among 10 tokens at every step, and lower is better. It is the number reported when two language models are compared on the same test text.

What a language model gives you to score

A token is the unit of text a model reads and predicts, a whole word or a piece of one. A causal language model reads text left to right and, at each position, gives a probability to every token in its vocabulary for what comes next. For a test sentence, the useful number at position $t$ is the probability the model gave to the token that really came next:

$$P(x_t \mid x_1, \ldots, x_{t-1})$$

Here $x_t$ is the token at position $t$, and the vertical bar reads "given", so this is the probability of $x_t$ given every token before it. A good model gives these probabilities high values on real text. Perplexity turns all of them, one per position, into a single number.

The perplexity formula

Take the negative log of each of those probabilities, average them over the $T$ positions in the text, and raise $e$ to the result:

$$\text{PPL} = \exp\left(-\frac{1}{T}\sum_{t=1}^{T} \log P(x_t \mid x_1, \ldots, x_{t-1})\right)$$

The negative log of a probability is often called the surprise of that token. A token the model expected, with probability near 1, has a surprise near 0. A token the model gave probability 0.01 has a surprise of $-\ln 0.01 \approx 4.6$. The average inside the brackets is the average surprise per token, and the outer $\exp$ turns it back into something that reads as a number of choices.

The formula PPL = exp of the average negative log probability, a high perplexity drawn as many plausible choices, a lower one as fewer, perplexity 1 as perfect prediction, and two models compared on the same tokenized test set.

How to read a perplexity value

A perplexity of $K$ means that, on average, the model is as uncertain as if it were choosing evenly among $K$ tokens at each step.

Perplexity 100: as uncertain as picking evenly among 100 tokens.
Perplexity 10: much more certain, about like narrowing every step down to 10 likely tokens.
Perplexity 1: perfect prediction, probability 1 on the true next token every time.

A model that spreads its probability evenly over the whole vocabulary has a perplexity equal to the vocabulary size. That gives a quick sense of scale for any value you see.

A worked example

Take a 4-token sequence and a model with a vocabulary of only 6 tokens. The model gave the true next token probability 0.10 at the first two positions, 0.70 at the third and 0.75 at the fourth. Each one becomes a surprise, the surprises are averaged, and $e$ is raised to the average:

1import math
2 
3probs = [0.10, 0.10, 0.70, 0.75]   # probability given to each true next token
4nlls = [-math.log(p) for p in probs]
5loss = sum(nlls) / len(nlls)
6print([round(x, 4) for x in nlls])
7print(round(loss, 4), round(math.exp(loss), 2))
8 
9uniform = [1 / 6] * 4               # spread evenly over a 6-token vocabulary
10loss_u = sum(-math.log(p) for p in uniform) / 4
11print(round(math.exp(loss_u), 2))

1[2.3026, 2.3026, 0.3567, 0.2877]
21.3124 3.72
36.0

The four surprises add up to 5.2496, and dividing by 4 gives 1.3124. Raising $e$ to that gives a perplexity of about 3.72. A model that spread its probability evenly over the 6 tokens would score exactly 6, so 3.72 says this model has narrowed the choices down without being close to certain. Two confident positions at 0.70 and 0.75 cannot fully make up for the two positions where the true token got only 0.10.

The four probabilities 0.10, 0.10, 0.70 and 0.75 as bars, their negative logs 2.3026, 2.3026, 0.3567 and 0.2877, the sum 5.2496 divided by 4 giving L = 1.3124, and PPL = exp(1.3124) = 3.72.

Perplexity is the exponential of the loss

A causal language model is usually trained with cross-entropy loss. At each position the loss is the negative log of the probability given to the true next token, and the training loss is the average over positions:

$$\mathcal{L} = -\frac{1}{T}\sum_{t=1}^{T} \log P(x_t \mid x_1, \ldots, x_{t-1})$$

That is the exact quantity inside the perplexity formula, so

$$\text{PPL} = \exp(\mathcal{L})$$

Loss and perplexity are the same measurement written two ways. Raising $e$ to a number keeps the order of numbers, so the model with the lower loss on a test set also has the lower perplexity, and lowering one lowers the other.

PyTorch computes the loss with the natural log, and a loss measured that way is in nats. That makes the conversion quick in both directions. A loss of 2.30 nats is a perplexity of $\exp(2.30) \approx 10$, and a loss of 0 is a perplexity of 1. A training log in nats and a results table in perplexity are converted with one $\exp$.

L as the average negative log probability, PPL = exp(L), a curve showing perplexity rising with the loss in the same order, L = 2.30 nats giving PPL about 10, and L = 0 giving PPL = 1.

Computing perplexity over a test set in PyTorch

A test set arrives in batches, and batches hold different numbers of real tokens, especially when shorter sequences are padded with a pad token to fill out the batch. The average has to be taken per token across the whole test set. Averaging the per-batch averages would give a batch with 3 tokens the same weight as a batch with 300.

The code below multiplies each batch's mean loss by the number of tokens it scored, adds those totals up, and divides by every scored token at the end:

1import math
2import torch
3import torch.nn as nn
4import torch.nn.functional as F
5 
6pad_id = 0
7 
8def batch_loss(model, tokens):
9    logits = model(tokens[:, :-1])
10    targets = tokens[:, 1:]
11    loss = F.cross_entropy(
12        logits.reshape(-1, logits.shape[-1]),
13        targets.reshape(-1),
14        ignore_index=pad_id,
15    )
16    num_tokens = (targets != pad_id).sum().item()
17    return loss.item(), num_tokens
18 
19def perplexity(model, dataloader):
20    model.eval()
21    total_loss = 0.0
22    total_tokens = 0
23    with torch.no_grad():
24        for tokens in dataloader:
25            avg_loss, num_tokens = batch_loss(model, tokens)
26            total_loss += avg_loss * num_tokens
27            total_tokens += num_tokens
28    return math.exp(total_loss / total_tokens)
29 
30torch.manual_seed(0)
31model = nn.Sequential(nn.Embedding(100, 32), nn.Linear(32, 100))
32dataloader = [
33    torch.tensor([[5, 12, 47, 8, 91], [33, 61, 27, 0, 0]]),
34    torch.tensor([[2, 40, 71, 18, 55]]),
35]
36print(round(perplexity(model, dataloader), 1))

1117.4

batch_loss shifts each sequence by one position so every token is scored against the one after it, and ignore_index leaves the pad positions out. model.eval() turns off behavior meant only for training, and torch.no_grad() skips gradient tracking, since nothing is being trained. The model here is untrained and has a vocabulary of 100 tokens, so its perplexity of 117.4 is close to the 100 you would get from spreading probability evenly. Its random weights make it slightly worse than even.

When perplexity can be compared

Perplexity numbers can be compared only when the models use the same tokenizer and are scored on the same test set. A different tokenizer cuts the same text into different tokens, so the model makes a different number of predictions and each one is a different task. The per-token average then changes even if the two models are equally good. A harder or easier test set moves the number too. Lower perplexity is better only under those two conditions.

Common mistakes

Comparing perplexity across different tokenizers. Per-token averages over different tokens measure different things.
Averaging per-batch losses. Batches with few real tokens get too much weight, so weight each batch by its token count.
Counting pad tokens. Padding is not text, and scoring it changes the average.
Reading a low training perplexity as a good model. A model that memorized its training text scores well there, so report perplexity on held-out test text.

The post on how language models are trained covers next-token prediction, softmax, cross-entropy and the chain rule that turns per-token probabilities into the probability of a whole sentence. Perplexity is that sentence probability rewritten as a per-token average: a lower perplexity means the model gave the real text a higher probability. The language model post covers what a language model outputs and how a bigram model gets its probabilities by counting.

QuiddityML teaches perplexity in the language models unit of its NLP track, right after the training loss. The exercises on it include saying what a perplexity of 10 means, spotting the buggy lines in a perplexity function and writing the per-token perplexity loop from scratch (QuiddityML).