A search query "How to fix a faucet?" may return an article about repairing a mixer tap leak as the first link. The query and the text share not a single word. The reason is that language models work not with letters but with vectors that encode meaning. Such vectors are called embeddings. They underlie search, recommendations, and chatbots that work with documents.

From Text to Numbers

A computer works only with numbers, so text is first split into tokens. A token is a word or part of a word. Each token is assigned a number in a vocabulary. For example, the word "кошка" (cat, feminine) might correspond to the number 18342, and the word "кот" (tomcat) to the number 51907.

The number does not reflect meaning. In numerical order, "кошка" and "кот" are far apart, while the neighbor of "кошки" might turn out to be "кошелёк" (wallet). Adding and comparing such numbers makes no sense.

An alternative is the one-hot representation. Each word corresponds to a string of zeros as long as the entire vocabulary, with a one in the position of that word. The drawback is obvious: any two words are equally far from each other. "Кошка" is no more similar to "кота" than it is to "холодильник" (refrigerator).

What is needed is a representation that is short, dense, and places semantically close words near each other. Such a representation is precisely the embedding. The term comes from the English embedding, which literally means "insertion" or "embedding". Words are, as it were, embedded into a space.

Semantic Space

Imagine a map where each word has coordinates. One axis shows how alive an object is. Another shows how large it is. A mouse, a cat, a dog, and an elephant are located on the right, by size. A teapot and a truck are on the left.

In this system, "cat" is described by two numbers: 0.9 and 0.2. They can be shown as a point or as an arrow from the center of the coordinates. The arrow is called a vector, which is why embeddings are also called vectors.

Two axes are not enough. "Tiger" and "cat" almost coincide on such a map. Adding a third axis ("domestic" or "wild") separates them. Each new axis adds a number and a distinction that the model can take into account.

Real models use hundreds or thousands of axes. Embedding dimensions are 768, 1024, or more than 3,000 numbers. Such a space cannot be drawn, but the mathematics in it is the same as the mathematics of a flat map.

It is important to note that in a real model, the axes do not have names like "alive" or "large." The model decides on its own how to distribute meaning across the numbers. One property is usually spread across many axes at the same time. What matters is the position of the point as a whole, not any single number.

How to Measure Similarity

The distance between points often does not reflect closeness in meaning. The length of a vector depends on factors unrelated to the word's meaning, for example, the frequency of the word in the training data. A vector can become longer while its direction remains unchanged.

Therefore, similarity is measured by the angle between vectors, more precisely by the cosine of the angle:

• vectors pointing in the same direction: the cosine equals 1;
• vectors are perpendicular: the cosine equals 0, meaning there is no shared sense;
• vectors point in opposite directions: the cosine equals −1.

Cosine similarity is calculated by multiplying the corresponding coordinates of the vectors, adding up the results, and dividing by the product of the vectors' lengths.

For the words "cat" and "kitten," the value will be high; for "cat" and "dog," it will be lower; for "cat" and "refrigerator," it will be low.

A non-obvious detail: in real models, even unrelated texts often yield a value not of 0, but, for example, 0.3. Therefore, you evaluate not the absolute value but the relative one: which pair is closer and which is farther.

Key point: an embedding is a point in the space of meanings. Similarity is the angle between points when viewed from the center.

How the Model Places the Dots

No one assigns word coordinates by hand. The method rests on an idea put forward by the linguist John Rupert Firth in 1957: a word can be understood by the company it keeps.

Consider sentences with a blank. "Tea," "coffee," and "cocoa" all fit the gap. Words with similar surroundings usually have close meanings, so their points should be located near each other.

In 2013, Google developed the Word2Vec model, which made embeddings widely known. The training process works as follows:

1. Each word is assigned a random point.
2. The model goes through a large body of text and learns to predict which words appear next to each other.
3. Each error slightly shifts the points: neighbors move closer together, random pairs move apart.
4. After billions of such shifts, the random cloud turns into a structured map.

As a result, drinks group with drinks, animals with animals, and cities with cities. The model was never given a definition of the concept "drink." It identified this group from the way people write texts.

Later, a property emerged that surprised even the authors of the method: directions in the space begin to correspond to relationships between words. If you take the vector "king," subtract "man," and add "woman," the nearest word, excluding the original ones, will be "queen." The pairs "country" and "capital" work in the same way. In modern models this arithmetic is performed less precisely, but the principle holds.

Embedding Table

The first step of any language model is the embedding table. It has a row for each token in the vocabulary. The model receives a token's number and retrieves the corresponding row. This is how a word turns into a vector.

For example, Llama 3 with 8 billion parameters has a vocabulary of 128 thousand tokens, and each embedding consists of 4096 numbers. A single table contains more than half a billion numbers, all of which are determined during training.

The table has a limitation. The word "key" has one row, that is, one point, for all of its meanings: "door key," "treble clef," "spring key" (a spring of water).

Layers of the model help resolve the ambiguity. Each layer adds information from neighboring words to the word's vector. "Key" next to "door" shifts toward locks, next to "violin" toward musical notes, and next to "water" toward springs. Such vectors are called contextual embeddings.

At the final stage, the model computes the probabilities of the next word based on the last vector. Thus, any response from a neural network begins with vectors and ends with them.

From Word to Text

Search requires more than a single word: phrases and documents need to be compared. The model outputs a vector for each token, so a single vector for the entire text must be obtained. There are two main approaches:

• Mean pooling: the average of the vectors of all tokens is calculated.
• Special token: a special token is added to the beginning of the text, and the model learns to gather the meaning of the whole phrase into it.

This process is called pooling.

A regular language model is not trained to create vectors for search. For this purpose, separate embedding models are used, which are fine-tuned on millions of pairs. A question and its correct answer are pulled toward each other, while random pairs are pushed apart. This method is called contrastive learning.

After such training, the phrases "how to fix a faucet" and "elimination of a mixer leak" end up close to each other. This explains the search result from the beginning of the article.

Application Areas

Semantic search. Documents are converted into vectors in advance and stored in a database. The query is also converted into a vector, after which the system finds the nearest vectors. Such storage systems are called vector databases.

Document chatbots (RAG). Documents are split into fragments, which are converted into vectors. When a question is asked, the system finds the nearest fragments and passes them to a language model. The model forms its answer based on the retrieved text, not only on data obtained during training.

Recommendations and clustering. Systems identify similar products and movies, and group thousands of reviews by topic without manual labeling. In all cases, one operation is performed: searching for the nearest vectors.

Multilingual models. In such models, the words "кошка", cat, and Katze land at almost the same point in the space. The meaning is the same, while the languages differ.

Multimodal models. Models such as CLIP place texts and images in the same space. A photo of a cat ends up next to the text "фото кота." This principle underlies image search by description.

Practical Test

The test was run on the BGE-M3 model. It works well with Russian and was run locally via Ollama. The script performs three actions: it sends phrases to the model, receives vectors, and calculates the cosine similarity of each pair.

The set of phrases included two phrases about a faucet, two about cats, and one about the weather. Each phrase was represented by 1024 numbers.

Results:

• The pairs "faucet" and "mixer tap" showed high similarity, even though they have no words in common.
• "A cat on the windowsill" and "a kitten by the window" also showed high similarity.
• The weather phrase had low similarity with all the other phrases.

A search for "how do I fix a faucet?" returned the text about a mixer leak first and the text about the weather last. Thus, semantic search was implemented in less than thirty lines of code.

Ограничения эмбеддингов

Логика распознаётся плохо. Эмбеддинги хорошо определяют тему, но плохо различают смысловые противоположности. Фразы «Я люблю кошек» и «Я не люблю кошек» оказываются очень близко, так как слова почти совпадают и тема одна и та же. Для поиска такое поведение допустимо, для понимания смысла — нет.

Модели наследуют стереотипы. Эмбеддинги обучаются на текстах людей, которые содержат их стереотипы. В работе 2016 года на аналогии «мужчина относится к программисту, как женщина к…» Word2Vec ответил «домохозяйка».

Векторы разных моделей несовместимы. У каждой модели собственная карта пространства. При смене модели всю базу векторов требуется пересчитать.

Summary

An embedding is an address of meaning in a multidimensional space. The similarity of two objects is determined by the angle between their addresses. All language models begin with an embedding table, which maps tokens to vectors, and subsequent layers refine these vectors in light of context.