QuiddityML

16 September 2026 · debuggingtrainingpytorchml-foundationlearning-rate

My loss is not decreasing: a checklist

A training loss that stays flat usually has one of a short list of causes. This checklist goes through them in a practical order: the data, the training loop, the learning rate, the loss function, initialization, and a one-batch test that separates a bug from a hard problem.

A loss that stays flat, or barely moves, is a common way for a training run to fail. Most of the usual causes are quick to check once you know what they are. This post goes through them in a practical order, cheapest checks first.

What "not decreasing" means

The loss is the number the training loop is trying to make smaller; it measures how far the model's outputs are from the targets on the current batch. Three shapes count as "not decreasing" and they point at different causes.

Before anything else, note which of the three you have. Then work down the list.

1. Check the data

Print one batch and look at it. This step is easy to skip and often finds the problem.

2. Check the training loop

Every PyTorch loop needs three calls in this order on every step. A missing or reordered one produces a flat loss with no error message.

for X, y in loader:
    optimizer.zero_grad()        # clear the gradients from the previous step
    loss = loss_fn(model(X), y)
    loss.backward()              # compute new gradients
    optimizer.step()             # move the parameters

3. Check the learning rate

If the loop is correct and the loss is noisy with no trend, the learning rate is the next suspect.

A learning rate range test, which multiplies the rate by a constant every step while logging the loss, finds a usable value in a hundred steps and is worth running once per new setup.

4. Check the loss function and the output layer

The loss function and the last layer of the model have to agree on what the model outputs. Several mismatches run without any error.

Cross-entropy takes raw logits; applying softmax first and then cross-entropy applies softmax twice and gives a wrong loss and wrong gradients

5. Check initialization and scale

6. The overfit-one-batch test

This test separates "there is a bug" from "the problem is hard". Take one small batch, 8 to 32 examples, and train on that same batch for a few hundred steps.

X, y = next(iter(loader))
for step in range(300):
    optimizer.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()
    optimizer.step()
    if step % 50 == 0:
        print(step, loss.item())

A working model and loop drive the loss on one batch to nearly zero, because the model can memorize a few examples regardless of whether it generalizes. If the loss does not go to nearly zero on one batch, something in sections 1 to 5 is wrong, and no amount of more data or more epochs will fix it. If it does go to zero, the loop is fine and the remaining question is capacity, data, or regularization.

7. Compare to a baseline

Once one batch overfits, put the loss in context. Compute the loss of a trivial model: predicting the mean target for regression, or the class frequencies for classification. If the trained model's loss is not below that baseline after a full epoch, the model is not using the inputs, which points at scaling, at a frozen layer, or at inputs that carry no signal for this target.

When the loss is nan

A loss that becomes nan is a separate failure with its own short list.

Three sources of NaN: log of zero, softmax overflow from exponentiating a large value, and division by a variance that rounds to zero in half precision

Turn on torch.autograd.set_detect_anomaly(True) to have PyTorch report the first operation that produced a nan.

Related concepts

QuiddityML covers each item on this checklist as its own concept in the ML Foundation and PyTorch tracks, and its exercises include spotting the missing zero_grad() in a loop, predicting the loss a classifier reports when it ignores its inputs, and finding the softmax that should not be in front of CrossEntropyLoss.