By Sagi Shaier · 2 November 2026 · 5 min read

Static vs contextual embeddings: why Word2Vec gives "bank" one vector

Why Word2Vec, GloVe and fastText cannot tell a river bank from a money bank, what the older multi-sense fix tried, and what a contextual embedding changes.

A static embedding gives every word one fixed vector, the same in every sentence. Models usually work with tokens, which can be whole words or pieces of words, and this post says "word" to keep the examples simple. Word2Vec, GloVe and fastText all work this way: the vector belongs to the word itself, so bank gets the same numbers in "deposit money at the bank" and "sit on the river bank". A contextual embedding gives a word a different vector in each sentence, built from the words around it. The difference decides whether a model can tell which meaning of a word a sentence is using.

What does "static" mean in practice?

A static embedding is a lookup table with one row per word in the vocabulary. To get a word's vector, the model finds the word's row number and reads that row. The rest of the sentence is not used.

You can see it in PyTorch with nn.Embedding, the table layer that holds one row of numbers per word. Here the table is untrained and has 4 numbers per row, which is enough to show the lookup:

1import torch
2import torch.nn as nn
3 
4vocab = {"i": 0, "went": 1, "to": 2, "the": 3, "bank": 4, "deposit": 5, "money": 6,
5         "sat": 7, "on": 8, "of": 9, "river": 10}
6emb = nn.Embedding(len(vocab), 4)   # one row of 4 numbers per word
7 
8s1 = "i went to the bank to deposit money".split()
9s2 = "i sat on the bank of the river".split()
10v1 = emb(torch.tensor([vocab[w] for w in s1]))[s1.index("bank")]
11v2 = emb(torch.tensor([vocab[w] for w in s2]))[s2.index("bank")]
12print(torch.equal(v1, v2))

1True

Both sentences read row 4, so bank gets exactly the same vector. Training a Word2Vec or GloVe table changes what is in the rows, but every sentence still reads the same row for the same word.

The polysemy problem

Many words have more than one meaning, which is called polysemy. Bank is the usual example:

1'I went to the bank to deposit money'
2-> bank = financial
3'I sat on the bank of the river'
4-> bank = river bank

During training, the single bank row is pushed by every sentence that contains bank. Sentences about money pull it toward words like deposit and account. Sentences about rivers pull it toward water and shore. The vector that comes out is a blend of both meanings, and a model reading it has no way to know which meaning a given sentence intends.

This is not something a better training setting inside Word2Vec or GloVe can fix. It follows from the shape of the representation itself: one word, one row, one vector. Fixing it needs a vector that changes with the surrounding sentence.

An early fix: one vector per meaning

An older approach kept more than one vector for a word, which is called a multi-prototype embedding. It works in three steps:

1. Collect every sentence in the training text that contains bank.
2. Describe each sentence by its nearby words. Some have deposit, money and account around bank. Others have river, water and shore. Cluster these descriptions into two groups.
3. Train a separate vector for each group: bank_1 for the money sentences and bank_2 for the river sentences.

The code below runs that idea on made-up numbers. Each of 40 sentences is described by a context vector, a vector built from the words around bank. Twenty sentences lean toward a "money" direction and twenty toward a "river" direction. A single vector built from every sentence at once stands in for the static bank, and a 2-group k-means split stands in for the multi-prototype step:

1import torch
2import torch.nn.functional as F
3 
4torch.manual_seed(0)
5money, river = torch.randn(50), torch.randn(50)       # two directions in a made-up 50-number space
6# context vectors for 40 sentences containing "bank": 20 near money words, 20 near river words
7contexts = torch.cat([money + 0.5 * torch.randn(20, 50), river + 0.5 * torch.randn(20, 50)])
8 
9bank = contexts.mean(dim=0)                            # one vector built from every sentence at once
10 
11# multi-prototype: split the sentences into 2 groups by their context, one vector per group
12centroids = contexts[[0, 39]]
13for _ in range(10):
14    assign = torch.cdist(contexts, centroids).argmin(dim=1)
15    centroids = torch.stack([contexts[assign == c].mean(dim=0) for c in range(2)])
16bank_1, bank_2 = centroids
17 
18cos = lambda a, b: round(F.cosine_similarity(a, b, dim=0).item(), 2)
19print("bank_1 vs bank_2:", cos(bank_1, bank_2))
20print("bank vs bank_1:  ", cos(bank, bank_1))
21print("bank vs bank_2:  ", cos(bank, bank_2))

1bank_1 vs bank_2: -0.11
2bank vs bank_1:   0.7
3bank vs bank_2:   0.63

Cosine similarity is 1 for vectors pointing the same way and 0 for vectors at a right angle. The two sense vectors point in different directions, at $-0.11$. The single vector sits between them, at about 0.7 from each, close to neither meaning.

Why the multi-prototype fix fell out of use

The single bank vector was trained on every sentence at once, so it lands close to an average of bank_1 and bank_2. That means the two senses can usually be recovered after training without any extra vectors. Look at the single vector's nearest words, and group the ones that pull it toward deposit and account apart from the ones that pull it toward river and water. Those groups match the two senses that multi-prototype trained for directly.

So training extra vectors added little that grouping the single vector's neighbors could not already give.

What contextual embeddings change

The fix that stuck makes the vector depend on the sentence. A contextual model reads the whole sentence and produces a vector for each word in it, so bank next to money and bank next to river come out as different vectors. Every sentence gets its own vector for the word.

When static embeddings are still a fine choice

Static embeddings are small, fast and easy to inspect. They work well when the words you care about mostly have one meaning, when you need word-level similarity or nearest neighbors, or when you need a cheap input layer for a small model. They struggle when the task depends on telling meanings apart, such as reading bank, bat or cell correctly inside a sentence.

Common mistakes

• Expecting a sense split from a static vector. A Word2Vec or GloVe lookup returns the same row for every sentence, however different the sentences are.
• Reading a static word's nearest neighbors as one meaning. The neighbors of bank mix both senses, because the vector itself is a blend.

Related concepts

The Word2Vec vs GloVe vs fastText post shows how the three static methods train their tables, and the word embeddings post covers the lookup table itself. The post on visualizing and evaluating word embeddings covers nearest neighbors and the k-means clustering used above.

QuiddityML teaches the limits of static embeddings as the last concept in the word representations unit of its NLP track, right before the track moves on to language models, and the exercise on it asks how Word2Vec represents bank in a money sentence and a river sentence (QuiddityML).