QuiddityML

16 September 2026 · gradient-descentoptimizationml-foundationlearning-rate

What is a learning rate, and how do you pick one?

The learning rate is the number that most often decides whether a model trains well. This post explains what it does inside the update rule, how to read the three shapes of a loss curve, why good values differ between SGD and Adam, and a five-line test that finds a usable value.

The learning rate is a number that controls how far a model moves its parameters on each training step. It is usually written $\eta$ (eta) and set once when the optimizer is created. Many training problems that are not bugs in the code trace back to this one number, which is why it is worth understanding before adjusting other settings.

Where the learning rate sits

Training a model means repeating one update many times. The gradient $\nabla L(\theta)$ is the vector of partial derivatives of the loss $L$ with respect to the parameters $\theta$, and it points in the direction that increases the loss fastest. The update moves the opposite way, scaled by the learning rate:

$$ \theta \leftarrow \theta - \eta , \nabla L(\theta) $$

The gradient decides the direction. The learning rate decides the size of the step in that direction. A gradient of 2.0 with $\eta = 0.1$ moves the parameter by 0.2; the same gradient with $\eta = 0.001$ moves it by 0.002. Nothing else in the update changes, so the difference between a model that trains quickly and one that barely moves can come down to this one multiplier.

In PyTorch the number is the lr argument of the optimizer:

import torch

model = torch.nn.Linear(10, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

The three regimes

A loss curve is a plot of the loss value after each step. Its shape tells you which of three situations you are in.

Three loss curves: too high diverges in a growing sawtooth, a good value decays smoothly, too low barely moves

Too high. Each step overshoots the low point and lands on the far side, higher than before. The loss oscillates and grows, and often ends as nan once a value overflows. If the loss goes up on the first few steps or jumps around wildly, the learning rate is the first thing to lower.

Too low. Each step is so small that the loss barely changes. The curve slopes down but so slowly that a full run ends far from a good solution. This regime can be mistaken for a bug in the data or the model, because the loss does move, just not enough.

In range. The loss drops fast in the first steps and then flattens as the parameters approach a low point. Within the usable range, larger values reach the flat part sooner, so the goal is the largest value that still produces this smooth shape.

The usable range is wide, often a factor of 10 or more, so exact tuning is rarely needed. What is needed is to land inside it at all.

Why the right value does not transfer between setups

A learning rate that worked in one script is only a starting point in the next, because several other choices change what a given step size means.

So the useful question is not "what is a good learning rate" but "what is a good learning rate for this optimizer, batch size, and model".

How to pick one

Start from the default for the optimizer. 1e-3 for Adam and AdamW, 0.01 for SGD without momentum, 0.1 for SGD with momentum on image models. These are inside the usable range for most small and medium problems.

Sweep by factors of 10. Train for a few hundred steps at 1e-4, 1e-3, 1e-2, and 1e-1 and compare the loss curves. Pick the largest value that still gives a smooth decrease, then optionally try 3 times larger and 3 times smaller. Because the usable range is wide, a coarse sweep is enough.

Run a learning rate range test. Start at a very small value and multiply it by a constant every step while recording the loss. The loss stays flat while the rate is too small, drops once it enters the usable range, and blows up once it is too large. A good choice is a value in the dropping section, somewhat below the point where the curve turns back up.

import math
import torch

model = torch.nn.Linear(10, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-6)
loss_fn = torch.nn.MSELoss()
X, y = torch.randn(256, 10), torch.randn(256, 1)

factor = math.exp(math.log(1e1 / 1e-6) / 100)  # 100 steps from 1e-6 to 10
for step in range(100):
    optimizer.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()
    optimizer.step()
    print(f"lr={optimizer.param_groups[0]['lr']:.2e} loss={loss.item():.4f}")
    optimizer.param_groups[0]["lr"] *= factor

Every parameter group in the optimizer carries its own lr, which is why the code reads and writes param_groups[0]["lr"].

Changing it during training

A single constant value is a compromise: the value that makes fast progress early is often too large to settle precisely at the end. A learning rate schedule changes $\eta$ over the run.

Warmup then cosine annealing: the rate ramps linearly to a peak and decays along a cosine curve to near zero

In PyTorch a scheduler wraps the optimizer and is stepped once per epoch or once per batch:

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=1000)

for step in range(1000):
    optimizer.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()
    optimizer.step()
    scheduler.step()

The scheduler step goes after the optimizer step; PyTorch warns if the order is reversed.

Common mistakes

Related concepts

QuiddityML teaches the learning rate as its own concept in the ML Foundation track, and its exercises include matching each of the three loss curves to its cause, spotting an Adam optimizer created with an SGD-sized learning rate, and predicting what a training loop prints when the scheduler step is missing.