4 October 2026 · 5 min read

Cross-validation explained: k-fold, stratified, and when not to use it

A score from one train/validation split can swing a lot depending on which examples landed where. This post explains k-fold and stratified k-fold cross-validation, builds both in plain PyTorch, and says when a single split or a time-ordered split is the better choice.

Cross-validation is a way to estimate how well a model will perform on data it has not seen, by splitting the dataset several different ways, training and scoring a model on each split, and averaging the scores. People reach for it when the dataset is small, because a score from one split of a small dataset can move by several percentage points depending on which examples happened to land in the held-out part.

What problem does cross-validation solve?

The usual way to check a model is to hold out part of the data. The model trains on the training set and is scored on a validation set, examples it did not train on. The validation score stands in for how the model will do on new data. (The train, validation, and test split post covers what each of the three sets is for.)

With 1,000 examples and an 80/20 split, 800 go to training and 200 to validation. The validation score now depends on which 200 examples were picked. If those 200 happen to be easy, the score looks good, and a different random split of the same data could come out 5 points lower. The model is the same in both runs, and the split is what changed.

How k-fold cross-validation works

K-fold cross-validation averages over several splits so no single lucky or unlucky split decides the result. The steps:

1. Shuffle the dataset and cut it into $K$ equal parts, called folds. Common choices are $K = 5$ and $K = 10$.
2. For each fold $k$, train a fresh model on the other $K - 1$ folds and score it on fold $k$.
3. Report the average of the $K$ scores.

Each example sits in the validation fold exactly once and in the training folds $K - 1$ times. The price is compute: 5-fold cross-validation trains 5 separate models, so it costs about 5 times as much as one split.

K-fold cross-validation in PyTorch

The snippet below builds k-fold by hand in plain PyTorch, trains a small logistic regression (a single linear layer whose output goes through a sigmoid) on each fold of 200 noisy examples, and then compares the spread of one-split scores against the spread of 5-fold averages.

1import torch
2 
3def k_fold_split(n, k=5, seed=0):
4    indices = torch.randperm(n, generator=torch.Generator().manual_seed(seed))
5    folds = indices.tensor_split(k)                # k pieces whose sizes differ by at most 1
6    return [(torch.cat(folds[:i] + folds[i + 1:]), folds[i]) for i in range(k)]
7 
8def train_and_score(X, y, train_idx, val_idx):
9    torch.manual_seed(0)
10    model = torch.nn.Linear(X.shape[1], 1)
11    opt = torch.optim.SGD(model.parameters(), lr=0.1)
12    loss_fn = torch.nn.BCEWithLogitsLoss()
13    for _ in range(200):
14        opt.zero_grad()
15        loss_fn(model(X[train_idx]).squeeze(1), y[train_idx]).backward()
16        opt.step()
17    preds = model(X[val_idx]).squeeze(1) > 0
18    return (preds == y[val_idx].bool()).float().mean().item()
19 
20torch.manual_seed(42)
21X = torch.randn(200, 5)
22y = (X[:, 0] + X[:, 1] + 1.5 * torch.randn(200) > 0).float()   # noisy labels
23 
24scores = [train_and_score(X, y, tr, va) for tr, va in k_fold_split(len(X), k=5)]
25print("fold accuracies:", [round(s, 3) for s in scores])
26print(f"5-fold mean {torch.tensor(scores).mean():.3f}, std {torch.tensor(scores).std():.3f}")
27 
28single = [train_and_score(X, y, *k_fold_split(len(X), k=5, seed=s)[0]) for s in range(10)]
29print(f"10 single 80/20 splits: min {min(single):.3f}, max {max(single):.3f}")
30cv = [torch.tensor([train_and_score(X, y, tr, va) for tr, va in k_fold_split(len(X), k=5, seed=s)]).mean().item() for s in range(10)]
31print(f"10 runs of 5-fold CV:   min {min(cv):.3f}, max {max(cv):.3f}")

1fold accuracies: [0.725, 0.775, 0.75, 0.725, 0.725]
25-fold mean 0.740, std 0.022
310 single 80/20 splits: min 0.625, max 0.900
410 runs of 5-fold CV:   min 0.735, max 0.770

The five folds score between 72.5% and 77.5%, and the cross-validation estimate is 74.0%. The last two lines are the reason to bother. Ten different single 80/20 splits of the same 200 examples reported anything from 62.5% to 90.0%. Ten runs of 5-fold cross-validation, each with a different shuffle, stayed between 73.5% and 77.0%.

Stratified k-fold for imbalanced classes

Plain k-fold shuffles without looking at the labels. When one class is rare, some folds can end up with almost none of it. Take 100 examples where 95 are class A and 5 are class B, split into 5 folds. A fold with zero class B examples cannot say anything about how well the model finds class B.

Stratified k-fold fixes this by splitting each class on its own and then combining the pieces, so each fold has the same class proportions as the full dataset.

1import torch
2 
3def stratified_folds(y, k=5, seed=0):
4    g = torch.Generator().manual_seed(seed)
5    folds = [[] for _ in range(k)]
6    for c in y.unique():
7        idx = (y == c).nonzero().squeeze(1)
8        idx = idx[torch.randperm(len(idx), generator=g)]
9        for i, piece in enumerate(idx.tensor_split(k)):   # each class is split into k pieces on its own
10            folds[i].append(piece)
11    return [torch.cat(f) for f in folds]
12 
13y = torch.cat([torch.zeros(95), torch.ones(5)]).long()   # 95 of class A, 5 of class B
14 
15for seed in range(3):
16    plain = torch.randperm(100, generator=torch.Generator().manual_seed(seed)).tensor_split(5)
17    print(f"seed {seed} plain:      class B per fold", [int(y[f].sum()) for f in plain])
18print("stratified:        class B per fold", [int(y[f].sum()) for f in stratified_folds(y)])

1seed 0 plain:      class B per fold [1, 2, 0, 1, 1]
2seed 1 plain:      class B per fold [1, 2, 1, 0, 1]
3seed 2 plain:      class B per fold [1, 3, 1, 0, 0]
4stratified:        class B per fold [1, 1, 1, 1, 1]

Each of the three plain shuffles produced at least one fold with zero class B examples, and the third produced two. The stratified version put exactly one class B example in each fold. For classification, stratified k-fold is a sensible default. For regression there are no classes to balance, so plain k-fold is the usual choice.

When not to use cross-validation

Cross-validation helps when the dataset is small enough that one split gives a noisy score and when training several models is affordable. Outside that, it is often the wrong tool.

Time-series data. When examples are ordered in time, such as daily sales or sensor readings, a shuffled split lets the model train on next month and get scored on last month. In deployment the model predicts the future from the past, so the shuffled score comes out higher than what it reaches in practice. This is a form of data leakage, where information the model will not have at prediction time sneaks into training. The fix is time-series cross-validation: keep the data in time order, train on an early stretch, validate on the stretch right after it, then grow the training window forward and repeat.

Large datasets. With 1 million examples, a single well-chosen split already gives a reliable score. Cross-validation would cost 5 times the compute for little gain.

Deep learning. Training a neural network is expensive, and 5-fold cross-validation means training 5 separate networks, which is often impractical. The usual approach is a single split, with careful attention to how well that split is built.

Related concepts

A single train and validation split is the one-split version of the same idea. Nearby posts cover tuning hyperparameters, precision, recall, and F1, and the ROC curve and AUC.

QuiddityML teaches cross-validation, stratified folds, and the cases to skip them as three concepts in the ML Foundation track, and the exercises on them include putting the lines of a k-fold split function in order, spotting a training slice that duplicates the validation fold, and predicting which examples a 7-fold split of 100 examples leaves unscored.