By Sagi Shaier · 30 October 2026 · 12 min read

How to turn words into numbers: integer encoding, one-hot, and dense vectors

How to measure whether two word vectors are close with distance and cosine similarity, why token ids and one-hot vectors cannot show which words are related, and what a dense vector changes.

A word representation is the list of numbers a model uses in place of a word. A neural network can only do arithmetic, so every word it reads has to become numbers first, and the choice of numbers decides whether the model can tell that violin is closer to cello than to tax. There are three common ways to make that list: reuse the word's id, write it as a one-hot vector, or give it a short dense vector.

Why a token id is not enough

A tokenizer cuts text into pieces called tokens and gives each piece an integer id. For example, a tokenizer might turn The cat sat. into four ids:

1'The cat sat.'  ->  [464, 3797, 3332, 13]

Those integers are useful as addresses. 3797 means row 3797 of a table the model keeps, and the digits describe a position in that table rather than anything about cats. The real question is what numbers go in each row.

Modern tokenizers cut text into subword pieces, so strictly the table has one row per token. This post says word because the methods were first built for whole words and are named after them. Everything below holds for a token's row in the same way.

The rows are what the model computes with for every task. For next-word prediction, The cat sat on the becomes a sequence of vectors that the model combines to score continuations such as mat or floor. For sentiment, This movie was funny becomes vectors that a classifier turns into positive. For search, the question and every candidate answer become vectors, and a similarity score ranks the answers.

What a vector is

Start with houses instead of words. Describe each house by its floor area in square meters, its age in years and its price, always in that order:

1house_a = [85, 12, 430000]
2house_b = [120, 3, 610000]

An ordered list of numbers like that is a vector. The order is what makes it useful. Position 1 is floor area in every house and position 2 is age in every house, so the two houses can be compared position by position. Read that way, house_b is bigger, newer and more expensive. Shuffle one list to [12, 85, 430000] and it now claims a 12 square meter house that is 85 years old.

A vector of $n$ numbers is also a point in a space with $n$ axes. The list $[3, 2]$ is the point 3 along the first axis and 2 along the second. Add a third number and the point moves into three dimensions. A vector of 300 numbers is a point in 300 dimensions, which is hard to picture while the arithmetic stays the same. Each of the $n$ numbers is one dimension of the vector.

Give every word in the vocabulary a vector and every word becomes a point in one shared space. No single number in a word's vector holds its meaning. What can carry meaning is where the words sit relative to each other: in a good arrangement violin and cello land near one another and tax lands somewhere else.

Two questions for any word representation

Every way of building word vectors can be checked with the same two questions.

1. Does closeness between vectors match closeness in meaning? Take a pair of related words and a pair of unrelated ones. A good representation scores violin and cello as closer than violin and tax. If it gives both pairs the same score, it has ranked nothing.
2. How many numbers does each word get? Every word in the vocabulary gets the same count. That count sets how much memory the table takes and how much arithmetic runs on every sentence.

The second question is answered by counting. The first needs a way to measure closeness, which comes next.

Measuring how close two vectors are

Two vectors are two points in the same space, so geometry gives two ways to measure closeness: the straight-line distance between the points, and the angle between the arrows drawn from the origin to each point.

Euclidean distance

Euclidean distance is the straight-line distance between two points. On a flat page it is the Pythagorean theorem, and it works the same way for any number of axes. Take the gap along each axis and square it, then add the squares and take the square root. For two vectors $\mathbf{u}$ and $\mathbf{v}$ with $n$ numbers each, where $u_i$ is the $i$-th number of $\mathbf{u}$:

$$d(\mathbf{u}, \mathbf{v}) = \sqrt{\sum_{i=1}^{n}(u_i - v_i)^2}$$

For example, with $\mathbf{u} = [3, 2]$ and $\mathbf{v} = [0, 6]$ the gaps are 3 along the first axis and 4 along the second:

$$d = \sqrt{3^2 + 4^2} = \sqrt{25} = 5$$

A distance of zero means the two vectors are the same point.

The angle between two vectors

Draw an arrow from the origin to each point. Arrows pointing the same way meet at $0°$, arrows at right angles meet at $90°$, and arrows pointing opposite ways meet at $180°$.

The angle ignores how far out each point sits. $[1, 2]$ and $[10, 20]$ lie in exactly the same direction, so the angle between them is $0°$, while the distance between them is about 20. Distance and angle can disagree, and which one to use depends on whether a vector's length carries information you care about.

The dot product

Word vectors have hundreds of dimensions and cannot be drawn, so the angle has to be computed. The computation starts from the dot product: multiply the two vectors dimension by dimension and add up the results.

$$\mathbf{u}^\top \mathbf{v} = \sum_{i=1}^{n} u_i v_i$$

For $\mathbf{u} = [1, 2]$ and $\mathbf{v} = [3, 1]$ that is $1 \cdot 3 + 2 \cdot 1 = 5$.

The dot product is large and positive when two vectors point the same way, near zero when they are at right angles, and negative when they point in opposite directions. It also grows with length. Double one vector and the dot product doubles without either arrow moving, so a large score can mean alike or long, and the number alone does not say which.

Cosine similarity

The cosine of an angle is a number between $-1$ and $1$. It is $1$ at $0°$, $0$ at $90°$ and $-1$ at $180°$, falling steadily as the arrows open apart. A similarity score should go up as two things get more alike, which the cosine does and the angle does not.

Cosine similarity computes the cosine of the angle straight from the two vectors. Write $\|\mathbf{u}\|$ for the length of $\mathbf{u}$, which is its distance from the origin. Cosine similarity is the dot product divided by both lengths:

$$\cos(\theta) = \frac{\mathbf{u}^\top \mathbf{v}}{\|\mathbf{u}\| \, \|\mathbf{v}\|}$$

Dividing by the lengths removes the size effect the dot product mixes in, so what is left depends on direction alone. For $[1, 2]$ and $[3, 1]$ the lengths are $\sqrt{5}$ and $\sqrt{10}$, so the cosine is $5 / \sqrt{50} \approx 0.707$, which is $\cos(45°)$.

Here are all three in PyTorch:

1import torch
2import torch.nn.functional as F
3 
4u = torch.tensor([3.0, 2.0])
5v = torch.tensor([0.0, 6.0])
6print(((u - v) ** 2).sum().sqrt().item())   # Euclidean distance: 5.0
7 
8a = torch.tensor([1.0, 2.0])
9b = torch.tensor([3.0, 1.0])
10print(torch.dot(a, b).item())               # dot product: 5.0
11print(round(F.cosine_similarity(a, b, dim=0).item(), 4))       # 0.7071
12print(round(F.cosine_similarity(a, 10 * b, dim=0).item(), 4))  # 0.7071

The last line stretches b to ten times its length. The dot product would grow tenfold, and the cosine stays at 0.7071.

Which measure to use

For comparing word meanings, length is usually noise. A word that appears more often tends to end up with a longer vector because more text went into it, and a plain dot product would then rank common words above rare ones whatever they mean. So cosine is the usual default for comparing words. The dot product fits when length carries real information, for example in a learned scoring layer.

Integer encoding

The simplest way to fill in the numbers is to reuse the id the tokenizer already produced, so each word becomes a vector of one number. This is integer encoding.

1vocab = {'the': 0, 'cat': 1, 'dog': 2, 'house': 3, 'runs': 4}

The size question has the shortest possible answer: one number per word. The similarity question needs the arithmetic. With one number, the distance formula is the size of the gap between the two ids:

$$d(\text{cat}, \text{dog}) = |1 - 2| = 1$$

$$d(\text{cat}, \text{house}) = |1 - 3| = 2$$

The arithmetic says cat is twice as close to dog as to house. That happens to be true here by luck. Ids follow the order of the tokenizer's list, which is usually set by how often each piece appeared or the order the pieces were created. Build the vocabulary from a different corpus and cat might land at 1 with chocolate at 2, and the distance comes out 1 again with no sign that anything changed.

The angle says even less. On one axis every id above 0 points the same way from the origin, so every pair of words gets the same cosine of 1. A line also has a limit no numbering can fix, since each word on it has at most two immediate neighbours. Cat is close to dog and kitten and also to pet, tiger and feline.

So ids fail the similarity check with numbers that rank words by an order the tokenizer happened to produce, and a model reading those numbers will act on them.

One-hot encoding

One-hot encoding removes the shared line. Every word gets its own dimension, and its vector is all zeros except for a single 1 in that dimension. A vocabulary of 50,000 words gives vectors of length 50,000, one axis per word. No word is larger than another and no word sits between two others.

Now run the similarity check. Subtract two different one-hot vectors and the difference is zero everywhere except a $+1$ and a $-1$, so the distance between any two different words is

$$d = \sqrt{1^2 + (-1)^2} = \sqrt{2} \approx 1.414$$

Which two words they are plays no part in that calculation. Violin and cello are $\sqrt{2}$ apart, and so are violin and tax.

The angle comes out the same way. Two different one-hot vectors never have a 1 in the same position, so their dot product is 0 and their cosine is 0, which is $\cos(90°)$. A word compared with itself scores 1. Those are the only two scores one-hot can produce:

1import torch
2import torch.nn.functional as F
3 
4vocab = {'violin': 0, 'cello': 1, 'tax': 2}
5ids = torch.tensor(list(vocab.values()))
6one_hot = F.one_hot(ids, num_classes=len(vocab)).float()
7print(one_hot)
8# tensor([[1., 0., 0.],
9#         [0., 1., 0.],
10#         [0., 0., 1.]])
11 
12violin, cello, tax = one_hot
13print((violin - cello).norm().item(), (violin - tax).norm().item())
14# 1.4142135381698608 1.4142135381698608
15print(F.cosine_similarity(violin, cello, dim=0).item(),
16      F.cosine_similarity(violin, tax, dim=0).item())
17# 0.0 0.0

F.one_hot takes a tensor of ids and the vocabulary size and returns one row per id. .norm() is the length of a vector, so (violin - cello).norm() is the Euclidean distance.

One-hot was built to stop a model doing arithmetic on arbitrary id spacing, and it does that. It carries the same content as the id, written out as a position instead of a number, so it cannot carry more meaning than the id did.

Sparse and dense vectors

A vector is sparse when most of its dimensions are zero. One-hot is the extreme case: out of 50,000 dimensions exactly one holds a value, and a bigger vocabulary means more zeros.

Put real figures on that. A table of one-hot vectors for 50,000 words, stored as 32-bit floats of 4 bytes each, holds

$$50{,}000 \times 50{,}000 = 2{,}500{,}000{,}000 \text{ numbers}$$

$$2{,}500{,}000{,}000 \times 4 \text{ bytes} = 10 \text{ GB}$$

The same information fits in one 4-byte integer per word, which is 200 KB for the whole vocabulary. Practical systems store sparse data in compressed form, and the ratio still shows how little each one-hot dimension carries.

A vector is dense when nearly every dimension holds a non-zero value. For violin, the id used 1 number and the one-hot vector used 50,000 positions. A dense vector uses a few hundred, each an ordinary signed real number such as 0.21 or -0.47. Because each number can take any size in either direction, two words can land at any distance from each other instead of the two distances one-hot allowed. Dense vectors typically have between 50 and 1,000 dimensions, with 300 a common choice.

Dense vectors also store meaning in a different place. In a one-hot vector each dimension stands for one word, so reading dimension 4,182 answers whether this word is cup. In a dense vector no single dimension has a name, and the meaning is spread across the whole list relative to every other word's list. A representation that spreads meaning across many numbers like this is called distributed.

Dense inputs also shrink the model that reads them. A linear layer needs one weight per input dimension for each output unit. With 100 output units, a one-hot input needs $50{,}000 \times 100 = 5{,}000{,}000$ weights while a 300-number dense input needs $300 \times 100 = 30{,}000$. A model with more free parameters than its data can pin down tends to overfit, so the smaller layer has less room to memorize noise. The dense table itself is still a $50{,}000 \times 300$ matrix, so the saving is cleanest when that table is pretrained and kept fixed.

Treating a word's meaning as a point in a continuous space of a few hundred dimensions, where nearby points have related meanings, is called vector semantics. Musical instruments end up in one region, months in another and finance words in a third, and the axes themselves have no labels.

Dense vectors solve the size question and make the similarity question answerable. They leave open where the few hundred numbers for each word come from.

When to use each, and common mistakes

• Use ids as lookup addresses, never as features. Feeding a raw id into a model as a number makes it treat house as three times cat.
• Use one-hot where only identity is needed. A common use is the target a classifier is trained against, where the model has to pick exactly one class.
• Use dense vectors when similarity has to show up in the numbers. That covers most models that read words.
• Compare word vectors with cosine similarity by default. Raw dot products and Euclidean distance both move with vector length, which tracks how often a word appears more than what it means.
• Normalize once when comparing many vectors. Dividing every vector by its length gives a unit vector of length 1, and the dot product of two unit vectors is their cosine similarity. A system comparing one query against 50,000 stored words can normalize them all up front and then use plain dot products.

Related concepts

Dense vectors need a source for their numbers. The post on bag of words, co-occurrence matrices and PMI builds word vectors by counting which words appear near each other, and the post on word embeddings covers dense vectors learned from data and the nn.Embedding table that stores them. The ids themselves come from a tokenizer, which the post on tokenization explains.

QuiddityML teaches these representations in the word representations unit of its NLP track, and the exercises include working out the distance between two one-hot vectors and turning the cosine similarity formula into PyTorch code (QuiddityML).