By Sagi Shaier · 22 October 2026 · 5 min read

The log-sum-exp trick explained: overflow, underflow, and adding logs

Why exp(1000) returns inf, why a product of 500 small probabilities returns 0, and how factoring out the largest value lets you add numbers stored as logs in PyTorch.

The log-sum-exp trick is a way to compute $\log \sum_j e^{x_j}$ without ever exponentiating a large number, by pulling the largest value out of the sum first. You need it whenever numbers are stored as logarithms, often because they are tiny probabilities, and you have to add them back together without the result turning into inf or -inf.

What overflow and underflow are

A float32, the default number type in PyTorch, covers a fixed range. The largest float32 is about $3.4 \times 10^{38}$. A result bigger than that has no representation, so it becomes inf, which is called overflow. On the other end, the smallest positive float32 is about $1.4 \times 10^{-45}$, and anything closer to zero rounds to exactly 0.0, which is called underflow.

The exponential function $e^x$, written exp(x) in code, runs into both edges quickly. Each step of 1 in $x$ multiplies the result by $e \approx 2.718$, so $e^2 \approx 7.39$ and $e^4 \approx 54.6$. In the other direction $e^{-2} \approx 0.135$, and it keeps shrinking toward zero.

1import torch
2 
3print(torch.exp(torch.tensor(1000.0)))
4print(torch.exp(torch.tensor(-1000.0)))
5# tensor(inf)
6# tensor(0.)

Neither case raises an error. PyTorch hands back inf or 0.0, and the next line of code carries on with it.

Where exp(x) overflows

$e^{88}$ is still a valid float32, and $e^{89}$ is already too large:

1print(torch.exp(torch.tensor(88.0)), torch.exp(torch.tensor(89.0)))
2print(torch.exp(torch.tensor(-103.0)), torch.exp(torch.tensor(-104.0)))
3# tensor(1.6516e+38) tensor(inf)
4# tensor(1.4013e-45) tensor(0.)

So in float32, exp overflows past about $x = 88.7$, and on the negative side it rounds to exactly 0 once $x$ drops below about $-104$. In float64 the same limits are about $710$ and $-745$. That means $e^{10}$ is safe in either format, while $e^{100}$ already overflows float32.

Multiplying many small probabilities

Underflow shows up whenever many small numbers get multiplied together. Take 500 independent events, each with probability 0.01. The chance that all of them happen is $0.01^{500} = 10^{-1000}$, far below anything float32 can hold:

1p = torch.full((500,), 0.01)
2print(p.prod())
3print(torch.log(p).sum())
4# tensor(0.)
5# tensor(-2302.5850)

The product is exactly 0.0, so two different products that both underflow can no longer be told apart. The fix is to add logarithms instead of multiplying probabilities. Here $\prod$ means multiply all the terms and $\sum$ means add them:

$$\log \prod_i p_i = \sum_i \log p_i$$

The log of a product equals the sum of the logs. Each $\log 0.01 \approx -4.605$ is a modest negative number, and 500 of them add up to about $-2303$, nowhere near either edge of float32.

Adding numbers stored as logs

Storing probabilities as logs keeps tiny values representable, but it makes adding them harder. Say two probabilities $p$ and $q$ are stored only as their logs, $a = \log p$ and $b = \log q$, and you need $\log(p + q)$. The direct route converts back, adds, and takes the log again:

$$\log(p + q) = \log(e^a + e^b)$$

With $a = -1000$ and $b = -1001$, both $e^a$ and $e^b$ underflow to 0, so the code takes $\log(0)$:

1a, b = torch.tensor(-1000.0), torch.tensor(-1001.0)
2print(torch.log(torch.exp(a) + torch.exp(b)))
3# tensor(-inf)

The true answer is about $-999.69$, a perfectly ordinary number. Large values break it the other way: with $a = 1000$, $e^a$ overflows to inf.

Written for any number of values, the expression $\log \sum_j e^{x_j}$ is called log-sum-exp, or LSE.

Factoring out the max

The fix is to pull the largest value out before exponentiating. For any constant $c$, $e^{x_j} = e^{c} \cdot e^{x_j - c}$, so the $e^c$ can come out of the sum and out of the log:

$$\log \sum_j e^{x_j} = c + \log \sum_j e^{x_j - c}$$

Choosing $c = \max_j x_j$, the largest value, makes the largest shifted value exactly 0, and $e^0 = 1$. Every other shifted value is negative, so every other term sits between 0 and 1. Nothing can overflow, and the sum is at least 1, so its log never hits $\log(0)$.

For example, take $x = [-1000, -1001, -1003]$. The max is $-1000$, so the shifted values are $[0, -1, -3]$, their exponentials are $[1, 0.368, 0.050]$, and the sum is $1.418$:

$$\text{LSE}(x) = -1000 + \log(1.418) \approx -999.651$$

This rewrite is the log-sum-exp trick:

$$\text{LSE}(x) = \max(x) + \log \sum_j e^{x_j - \max(x)}$$

Both sides equal the same number, and only the right side is safe to compute. In code it takes three lines, and PyTorch has it built in as torch.logsumexp:

1def log_sum_exp(x):
2    m = x.max()
3    return m + torch.log(torch.exp(x - m).sum())
4 
5x = torch.tensor([-1000.0, -1001.0, -1003.0])
6print(log_sum_exp(x), torch.logsumexp(x, dim=0))
7# tensor(-999.6510) tensor(-999.6510)

Both failures, fixed

The trick handles the underflow side and the overflow side the same way. For $x = [-1000, -1001]$ the shifted values are $[0, -1]$:

$$\text{LSE}(x) = -1000 + \log(e^{0} + e^{-1}) = -1000 + \log(1.368) \approx -999.69$$

For $x = [1000, 0, 0]$ the max is 1000 and the shifted values are $[0, -1000, -1000]$:

$$\text{LSE}(x) = 1000 + \log(1 + 2e^{-1000}) \approx 1000 + \log(1) = 1000$$

$e^{-1000}$ still underflows to 0 in the second case, which does no harm because it was far too small to change the sum.

1for values in ([-1000.0, -1001.0], [1000.0, 0.0, 0.0]):
2    x = torch.tensor(values)
3    print(torch.log(torch.exp(x).sum()), torch.logsumexp(x, dim=0))
4# tensor(-inf) tensor(-999.6868)
5# tensor(inf) tensor(1000.)

Common mistakes

• Multiplying probabilities over a long sequence. Sum their logs instead.
• Computing torch.log(torch.exp(x).sum()) directly. Use torch.logsumexp, or subtract the max first.
• Subtracting the max and forgetting to add it back. The shifted sum gives $\text{LSE}(x) - \max(x)$, not $\text{LSE}(x)$.
• Shifting by a value other than the max. Any constant gives the same answer on paper, but only the max guarantees the largest term is exactly 1.

Related concepts

Floating point for deep learning covers how floats store numbers and why they round. Softmax uses the same max shift to keep its probabilities finite. Exponents and logarithms for machine learning covers the log rules this trick relies on.

QuiddityML teaches overflow and the log-sum-exp trick as two concepts in the Math track's numerical computation unit. The exercises on them include writing a log_prob function that adds logs instead of multiplying probabilities, ordering the lines of a stable log_sum_exp, and spotting the bug in a version that shifts by the smallest value instead of the largest.