27 September 2026 · 5 min read

Multi-class vs multi-label classification: softmax vs sigmoid outputs

Multi-class classification picks exactly one label per input, and multi-label classification can pick several or none. This post explains the difference, why the first uses softmax and the second uses one sigmoid per label, and how to write both in PyTorch.

In multi-class classification, each input belongs to exactly one of several classes: a photo is a cat, a dog, or a bird. In multi-label classification, each input can have any number of labels at once: a photo can be tagged outdoor, daytime, and has-a-dog together, or none of them. The two setups need a different output function, a different loss, and a different way to read predictions, and a tagging model built with the multi-class setup can only ever predict one tag per photo.

Multi-class: exactly one right answer

A multi-class classifier with $K$ classes outputs $K$ raw scores, one per class, called logits, and turns them into probabilities with softmax:

$$\text{softmax}(z)k = \frac{e^{z_k}}{\sum{j=1}^{K} e^{z_j}}$$

Each probability is between 0 and 1, and the $K$ probabilities add up to 1. With logits cat 2.3, dog 0.6, and bird -1.2, softmax gives cat 0.824, dog 0.151, and bird 0.025. The prediction is the class with the largest probability, and the loss is cross-entropy, $-\log$ of the probability given to the correct class. The softmax post covers the formula in detail.

The sum-to-1 rule builds in the assumption that the classes exclude each other. Raising one class's probability lowers the others, which is correct when only one class can be true.

Multi-label: one yes/no decision per label

A multi-label problem is a set of separate yes/no questions about the same input. Tagging a photo with outdoor, daytime, and has-a-dog is three questions, and the answer to one does not limit the others. Topic tags on an article, genres of a film, and labels on a bug report are other examples.

The model still outputs one logit per label, but each logit goes through its own sigmoid:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Sigmoid maps any real number to a value between 0 and 1, and each label's probability depends only on its own logit. With logits outdoor 1.9, daytime 1.9, and has-a-dog 1.1, the probabilities are 0.87, 0.87, and 0.75. Each one is compared with its own threshold, 0.5 by default, so all three come out yes. The three probabilities add up to 2.49, which is fine, because they answer three separate questions.

The labels for one example are written as a multi-hot vector, with one entry per label, 1 if the label applies and 0 if not. A photo taken outdoors at night with a dog in it is $[1, 0, 1]$ for outdoor, daytime, and has-a-dog.

The loss is binary cross-entropy (BCE) applied to each label on its own and averaged. With $L$ labels, $y_l$ the true 0 or 1 for label $l$, and $\hat{y}_l$ the sigmoid output for label $l$:

$$\mathcal{L} = -\frac{1}{L}\sum_{l=1}^{L} \bigl[y_l \log \hat{y}_l + (1-y_l)\log(1-\hat{y}_l)\bigr]$$

Each term inside the sum is the loss of one yes/no question, the same loss logistic regression trains with for a single label.

Why the top classes of a softmax do not work as labels

A tempting shortcut is to train with softmax and take the top two or three classes as the predicted labels. It fails for two reasons. Softmax probabilities share one total of 1, so a photo with both a dog and a cat cannot get 0.9 for dog and 0.9 for cat, and training pushes two labels that are both true against each other. Taking the top $k$ classes also fixes the number of labels in advance, so a photo with zero tags or five tags still gets exactly $k$. Sigmoid outputs avoid both problems: each label is scored on its own, and any number of labels, from zero to all $L$, can come out above the threshold.

Side by side

Both in PyTorch

1import torch
2from torch import nn
3 
4torch.manual_seed(0)
5batch_size, d = 4, 16
6x = torch.randn(batch_size, d)
7 
8# multi-class: 3 classes, exactly one per example
9K = 3
10mc_model = nn.Linear(in_features=d, out_features=K)
11mc_loss_fn = nn.CrossEntropyLoss()                 # applies softmax inside
12y_class = torch.tensor([0, 2, 1, 0])               # class indices, shape (batch_size,)
13mc_logits = mc_model(x)                            # shape (batch_size, K)
14mc_loss = mc_loss_fn(mc_logits, y_class)
15mc_pred = mc_logits.argmax(dim=-1)                 # one class per example
16 
17# multi-label: 3 labels, any number per example
18num_labels = 3
19ml_model = nn.Linear(in_features=d, out_features=num_labels)
20ml_loss_fn = nn.BCEWithLogitsLoss()                # applies sigmoid per label inside
21y_multi = torch.tensor([[1., 0., 1.],
22                        [1., 1., 1.],
23                        [0., 0., 0.],
24                        [0., 1., 0.]])             # multi-hot floats, shape (batch_size, num_labels)
25ml_logits = ml_model(x)                            # shape (batch_size, num_labels)
26ml_loss = ml_loss_fn(ml_logits, y_multi)
27ml_pred = (torch.sigmoid(ml_logits) > 0.5).int()   # a 0 or 1 per label
28 
29print(mc_pred.shape, ml_pred.shape)                # torch.Size([4]) torch.Size([4, 3])

The two heads are the same nn.Linear layer. They differ in the loss, the target format, and the line that turns logits into predictions. argmax on the logits picks the same class as argmax on the softmax probabilities, since softmax keeps the order, so the multi-class head skips the softmax at prediction time. Both models return raw logits, because each loss applies its own softmax or sigmoid.

Choosing thresholds and handling rare labels

A threshold of 0.5 per label is a starting point. Labels that are rare in the training data often get low sigmoid outputs even when present, so a lower threshold for those labels, picked on validation data, catches more of them. nn.BCEWithLogitsLoss(pos_weight=...) takes one weight per label and raises the loss on missed positives. A label present in 5% of examples is commonly given a weight near 19, the number of negatives per positive.

Common mistakes

• A multi-hot target in nn.CrossEntropyLoss. PyTorch reads a float target of shape (batch_size, K) as per-class probabilities, so a multi-hot target raises no error, and the model trains as if the labels competed for one total of 1.
• Integer multi-hot targets. nn.BCEWithLogitsLoss needs float targets, so a target built as integers needs .float() before the loss.
• Sigmoid in the model and in the loss. Returning torch.sigmoid(logits) from forward and then using nn.BCEWithLogitsLoss applies sigmoid twice, so every probability the loss works with stays between 0.5 and 0.73.
• Accuracy as the metric. With 20 labels and about 2 present per example, predicting no labels at all gets 90% of the yes/no decisions right. Per-label precision (of the labels predicted yes, how many were right) and recall (of the labels actually present, how many were predicted) show what the model misses.

Related concepts

Binary classification is multi-label classification with a single label, which is the logistic regression setup. Softmax covers the multi-class output in detail, including the overflow fix used inside nn.CrossEntropyLoss. Multi-label is one case of multi-output models, which predict several targets per input, and those targets can also mix types, such as a class and a number from the same model.

QuiddityML teaches multi-label classification as its own concept in the ML Foundation track, with exercises that include explaining why the top two softmax classes cannot stand in for two labels, filling in the loss and threshold lines of a multi-label module, and writing that module from scratch.