17 September 2026 · overfittingregularizationml-foundation

Overfitting vs underfitting: how to tell which one you have, and how to fix it

Overfitting is when a model memorizes its training data and fails on new data. Underfitting is when it cannot even fit the training data. This post shows how to read which one you have from the loss curves and what to change for each.

Overfitting is when a model learns its training examples so closely that it stops working on examples it has not seen. Underfitting is when the model cannot even get the training examples right. Both show up as a model that performs worse than you hoped, and the fixes for one make the other worse, so the first job is to tell them apart. That comes down to comparing how the model does on the data it trained on against how it does on data it did not.

Training, validation, and test sets

A dataset is usually split three ways before training starts. The training set is the examples the model updates its parameters on. The validation set is held out from training and used to measure the model while you are still making decisions about it, such as how long to train or how big to make it. The test set is held out from both and used once, at the end, so the final number is not shaped by any of those decisions.

The loss is the number the model is trying to push down, and it can be computed on any of the three sets. Training loss says how well the model fits what it has seen, and validation loss says how well that carries over to what it has not. Every diagnosis in this post compares those two numbers over the course of training.

What underfitting looks like

An underfitting model has training loss that stays high. It is not memorizing anything because it is not even fitting. The validation loss is high too, and roughly as high, because there is no learned detail to fail to transfer. Three things usually cause it. The model is too small to represent the pattern, for example a straight line asked to separate points that lie in a ring. Training was cut short, with too few epochs or a learning rate so low that the loss was still dropping when it stopped. Or the regularization, the set of techniques that hold a model back from fitting too closely, is turned up so far that it blocks fitting altogether.

Three causes of underfitting side by side: a model with too little capacity, training stopped too early, and regularization set too strong, each with its fix.

What overfitting looks like

An overfitting model has training loss that keeps falling while validation loss stops falling and then rises. The model has moved from learning the pattern to memorizing the specific examples, including their noise, and that memorized detail does not carry over. This happens when the model has far more capacity than the data needs, when there is too little data for the model size, when training runs long past the point where validation stopped improving, or when there is no regularization at all to resist memorizing.

A related idea worth naming is the bias-variance tradeoff. Bias is error from a model that is too simple to capture the pattern, and high bias is underfitting. Variance is error from a model that changes a lot depending on which particular examples it saw, and high variance is overfitting. Shrinking one tends to grow the other, which is why the fixes below pull in opposite directions.

Read the curves

Plot training loss and validation loss against epochs on the same axes. Three shapes cover most runs.

Healthy training has both losses falling and staying close, overfitting has validation loss turning upward while training loss keeps falling, and underfitting has both losses flat and high.

Both high and flat: underfitting. The model is not learning the training set.
Training falls, validation falls then climbs: overfitting. The point where validation turns is the best model you had.
Both fall and stay close: healthy. A small gap is normal, since the model has seen one set and not the other.

Before changing anything about the model, rule out a bug. Take a single batch of 8 or 16 examples and train on just that batch for a few hundred steps. A working model and training loop will drive the loss on that batch close to zero. If it cannot, the problem is in the code or the data, not in the fit, and none of the fixes below will help.

Fixing underfitting

Give the model more room and more time. Add layers or width. Train for more epochs. Raise the learning rate by a factor of 10 and see whether the loss starts moving. Check the input features: a model cannot fit a pattern the inputs do not contain, so a tabular model with an important column missing will underfit no matter how big it is. And turn regularization down or off, since every one of those techniques is a brake.

Fixing overfitting

Each cause has a matching fix, and several can run at once.

More data, or augmentation. Augmentation makes new training examples by transforming existing ones, for example flipping or cropping an image, so the model cannot memorize a fixed set.
Weight decay. Shrink every parameter slightly on every step so the model cannot rely on a few huge weights. Passing weight_decay to the optimizer does this.
Dropout. During training, randomly zero a fraction of the activations in a layer on each step, so no single unit can be relied on and the model spreads its evidence across many.
A smaller model. Fewer parameters means less capacity to memorize with.
L1 regularization. Instead of penalizing squared weights like weight decay does, penalize their absolute values, which pushes many weights to exactly zero. Useful when you want a sparse model, less common in deep learning than weight decay.

Four overfitting causes paired with fixes: a model too large gets less capacity, training too long gets early stopping, too little data gets more data or augmentation, no regularization gets weight decay or dropout.

Another fix is early stopping, which needs no change to the model or the data. Measure validation loss every epoch, keep the checkpoint with the lowest value, and stop after it has failed to improve for some number of epochs called the patience. The model you ship is the best checkpoint, which is usually not the last one.

The validation loss bottoms out around epoch 100 and then rises while training loss keeps falling, and the checkpoint at the bottom is the one to keep.

When the validation set leaks

Validation loss is only trustworthy if the validation set really is unseen. Common ways it leaks: splitting after augmentation so a flipped copy of a validation image sits in training, splitting a time series randomly so the model trains on the future of the examples it is validated on, or normalizing with statistics computed over the whole dataset before splitting. A leaked validation set makes an overfitting model look healthy. The same rules apply to the test set, and it has one more: use it once. Every choice made while looking at test results turns it into a second validation set.

Code

1import torch
2 
3model = torch.nn.Sequential(
4    torch.nn.Linear(20, 64), torch.nn.ReLU(), torch.nn.Dropout(0.2),
5    torch.nn.Linear(64, 1),
6)
7optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
8loss_fn = torch.nn.MSELoss()
9 
10best_val, best_state, patience, bad_epochs = float("inf"), None, 5, 0
11for epoch in range(200):
12    model.train()
13    for x, y in train_loader:
14        optimizer.zero_grad()
15        loss_fn(model(x), y).backward()
16        optimizer.step()
17    model.eval()
18    with torch.no_grad():
19        val = sum(loss_fn(model(x), y).item() for x, y in val_loader) / len(val_loader)
20    if val < best_val:
21        best_val, best_state, bad_epochs = val, model.state_dict(), 0
22    else:
23        bad_epochs += 1
24        if bad_epochs >= patience:
25            break
26model.load_state_dict(best_state)

If validation loss never turns upward and training loss is still high, this is underfitting and the first thing to change is weight_decay=0 and a larger hidden size, not more patience.

Common mistakes

Judging the model on training accuracy. Training accuracy only rises. It cannot show overfitting.
Tuning on the test set. After a few rounds of changing the model based on test results, the test number stops meaning anything.
Adding regularization to a model that is underfitting. Dropout and weight decay make an underfitting model worse.
Calling model.eval() only sometimes. Dropout behaves differently in training and evaluation mode, so a validation loss measured in training mode is noisier and higher than the real one.
Reading one noisy epoch as a trend. Real curves wobble. Look at several epochs before deciding validation has turned.

Related concepts

Regularization is the name for the family of overfitting fixes above, and weight decay, dropout, and early stopping each have enough detail for a post of their own. The bias-variance tradeoff is the older statistical framing of the same two failure modes. Cross-validation rotates which slice of the data is held out, so the diagnosis does not depend on one particular split, and matters most when the dataset is small.

QuiddityML teaches underfitting, overfitting, and the bias-variance tradeoff as separate concepts in the ML Foundation track, with exercises that show a pair of loss curves and ask which failure they show, and one where the reader orders the lines of an early-stopping loop, and each fix, from weight decay to dropout, gets its own lesson and a write-it-from-scratch exercise (quiddityml.com).