By Sagi Shaier · 30 October 2026 · 5 min read

What are word embeddings?

How an embedding table turns a token id into a vector in PyTorch, why training can move those vectors, and how relationships like king to queen show up as directions, with the bug that breaks analogy search.

A word embedding is a short list of real numbers for one word, usually a few hundred long, learned from data so that words with similar meanings end up with similar numbers. It is the form a model uses when it needs a word's meaning as numbers. In a good set of embeddings, cosine similarity between violin and cello comes out high and between violin and tax comes out low.

From one-hot vectors to embeddings

Before embeddings, the two common ways to give a word numbers were its id and its one-hot vector. A one-hot vector has one position per vocabulary word, all zeros except a single 1, so a 50,000-word vocabulary gives each word 50,000 numbers. Every pair of different one-hot vectors is the same distance apart, which means they cannot show that violin is closer to cello than to tax.

An embedding for violin uses a few hundred dimensions instead, and every dimension holds a learned real number such as 0.21 or -0.44. A vector where nearly every number is non-zero is called dense. The numbers are placed so that related words land near each other, which a one-hot vector cannot do.

A long one-hot vector of mostly zeros with a single 1, next to a short embedding of eight signed numbers learned so similar words land close

The word itself is still looked up by its id. The embedding is the row of numbers that id points to.

Vector, word vector, word embedding

Three terms get used for these lists, and each one is narrower than the last.

A vector is any ordered list of numbers. A house record like [85, 12, 430000] for floor area, age and price is a vector with nothing to do with words.
A word vector is a vector assigned to one word. A word's id, its one-hot vector and its embedding are three different word vectors for the same word.
A word embedding is a word vector that is also dense and learned from data so similar words get similar numbers. The id and the one-hot vector are word vectors but not embeddings, because neither is learned from text.

Nested boxes: vector contains word vector, which contains id, one-hot and embedding, with only the embedding counted as a word embedding

The embedding table in PyTorch

In PyTorch the embeddings for a vocabulary live in an nn.Embedding layer. Its weights are a matrix with one row per token id and one column per embedding number, so its shape is (vocab_size, embedding_dim). A vocabulary of 50,000 tokens with 300 numbers each gives a $50{,}000 \times 300$ matrix.

A token id selects one row of that matrix. Token 4 selects row 4 and token 2034 selects row 2034, and the row is the vector the model uses for that token.

A vocabulary table mapping words to ids, a V by D embedding matrix where the id 4 for runs selects row 4, and that row returned as the embedding vector

Here is the lookup with a tiny table of 10 rows and 2 numbers per row:

1import torch
2import torch.nn as nn
3 
4torch.manual_seed(0)
5emb = nn.Embedding(10, 2)        # 10 token ids, 2 numbers each
6tokens = torch.tensor([[4, 7, 9]])
7x = emb(tokens)
8 
9print(emb.weight.shape)          # torch.Size([10, 2])
10print(x.shape)                   # torch.Size([1, 3, 2])
11print(torch.equal(x[0, 0], emb.weight[4]))   # True
12 
13x.sum().backward()               # any loss will do for this check
14print(emb.weight.grad.abs().sum(dim=1))
15# tensor([0., 0., 0., 0., 2., 0., 0., 2., 0., 2.])

tokens has shape (1, 3), meaning one sequence of three token ids. The output x has shape (1, 3, 2): one sequence, three vectors and two numbers per vector. The first vector equals row 4 of the weight matrix exactly.

The code nn.Embedding(10, 2) applied to the ids 4, 7 and 9, picking rows 4, 7 and 9 of a 10 by 2 matrix to make the output tensor

The rows of the table are parameters, so training can change them. When a token appears in a training example, the loss sends gradients back into the rows that were looked up. The last line of the code shows it: after backward(), only rows 4, 7 and 9 have a non-zero gradient. Over many updates the rows move into positions that make the training task easier, and that movement is what places related words near each other.

What embeddings make possible

Once the rows are trained, the geometry of the space becomes usable. You can score how related two words are with cosine similarity, list a word's nearest neighbours, and group words by topic.

Words in an embedding space with violin and cello together, dog, cat, puppy and kitten together, and king, queen, monarch and prince together, plus a cosine score, a nearest-neighbour list and a man to woman direction

Ids and one-hot vectors give none of this, because their scores carry no meaning. Vectors built by counting which words appear near each other can give useful geometry, but they start tens of thousands of dimensions wide and usually need a separate compression step. Training a small prediction model and keeping its learned rows gives vectors that are dense from the start. Word2Vec is a well-known example of that approach.

Relationships as directions

After training, Word2Vec vectors can carry more than similarity. Some relationships between words show up as directions in the space. Write $\text{vec}(w)$ for the embedding of word $w$:

$$\text{vec}(\text{king}) - \text{vec}(\text{man})$$

$$+ \text{vec}(\text{woman}) \approx \text{vec}(\text{queen})$$

The difference $\text{vec}(\text{king}) - \text{vec}(\text{man})$ is a step through the space. Adding it to $\text{vec}(\text{woman})$ lands near $\text{vec}(\text{queen})$, because the step from man to king points roughly the same way as the step from woman to queen. The same pattern shows up for countries and capitals and for verb tenses:

$$\text{vec}(\text{Paris}) - \text{vec}(\text{France})$$

$$+ \text{vec}(\text{Germany}) \approx \text{vec}(\text{Berlin})$$

$$\text{vec}(\text{walked}) - \text{vec}(\text{walking})$$

$$+ \text{vec}(\text{swimming}) \approx \text{vec}(\text{swam})$$

Nobody gives the model these rules. The pattern comes from the training objective, since words in these pairs appear in similar contexts.

Countries, capitals, people and verb forms in an embedding space, with parallel arrows from Germany to Berlin and France to Paris, king to queen and man to woman, and walking to walked and swimming to swam

The parallelogram model

This direction pattern is called the parallelogram model. To answer bee is to hive as bird is to what, measure the step from bee to hive. Then apply the same step starting from bird and read off the word closest to where you land. In a good embedding that word is nest. The four words trace a parallelogram, because the arrow from bee to hive runs parallel to the arrow from bird to nest and both encode a creature and the home it builds.

Bee, hive, bird and nest at the corners of a parallelogram, with the same offset carrying bee to hive and bird to nest

Common mistakes with embeddings and analogies

Forgetting to exclude the query words. Compute $\text{vec}(\text{king}) - \text{vec}(\text{man}) + \text{vec}(\text{woman})$ and take the nearest vector, and the answer that comes back is usually king rather than queen. The result lands so close to the words you fed in that they win the search. Remove the three query words from the candidates before ranking.
Expecting every relationship to work. Some relationships come out clean while others are skewed or barely show up, and part of what looks like a clean hit comes from how the scoring is done.
Reading meaning into single dimensions. Dimension 17 of a 300-number embedding has no name. The meaning is in where a word sits relative to other words.
Using the id as if it were the embedding. The id only picks a row, and the row holds the numbers the model computes with.

The point king minus man plus woman lands right next to king, so the nearest vector is usually king itself unless the input words are excluded

The post on turning words into numbers covers ids, one-hot vectors and the similarity measures used on embeddings, including cosine similarity. The post on bag of words, co-occurrence matrices and PMI builds word vectors by counting neighbours instead of learning them, from the same idea that a word is described by the words around it.

QuiddityML teaches word embeddings in the word representations unit of its NLP track, and the exercises include writing an nn.Embedding lookup from scratch and spotting the analogy search that forgets to strike out its own query words (QuiddityML).