What Is an AI Embedding? Turning Meaning Into Numbers
An embedding turns a word, sentence or picture into a list of numbers so that similar things sit close together. Here is how that works and what it is used for.
In this article
An embedding is a list of numbers that stands for something, such as a word, a sentence or a meal on a menu. The numbers are chosen so that things with similar meaning get similar lists. A computer cannot judge meaning directly, but it can compare numbers, so embeddings give software a way to say that two things are alike.
The problem embeddings solve
The simplest way to give a word to a computer is one-hot encoding. Give every item in a vocabulary its own position in a very long list, put a 1 in that position and a 0 everywhere else. Google's machine learning course uses a food example: with 5,000 meals, each meal becomes a list 5,000 numbers long, and almost all of them are zero.
That is wasteful. According to the same course, such long vectors mean a neural network needs more weights, more data, more computation and more memory, which also makes models harder to run on devices. And one-hot lists say nothing about how items relate: borscht is as far from a hot dog as it is from a salad.
An embedding is a lower-dimensional representation of that sparse data. Instead of 5,000 mostly empty slots, each item gets a short list of real numbers that carries more information per slot.
Meaning as position in a space
Think of each embedding as a point on a map. Google's course builds the idea step by step with meals. On one axis, call it "sandwichness", a hot dog and shawarma land close together and apple strudel lands far away. Add a second axis for "dessertness" and the strudel moves even further from the sandwiches. A third axis for "liquidness" separates borscht from the rest.
The distance between two points can then be read as how related the items are. Real embedding spaces have many more dimensions than three; the course says word embeddings often use 256, 512 or 1,024. Unlike the food example, the individual dimensions are rarely easy to name. The meaning sits in the pattern across all of them, not in any one number.
From distributional ideas to word2vec
The idea behind word embeddings is old. Wikipedia's article on the subject traces it to the linguist J. R. Firth, who in 1957 described a word as characterised by the company it keeps. Words that appear in similar surroundings tend to mean similar things. Methods such as latent semantic analysis, from the 1980s, used this by compressing counts of which words occur together.
In 2000, Bengio and colleagues proposed learning word representations with neural networks. In 2013 a Google team led by Tomas Mikolov released word2vec, a toolkit that trained word vectors faster than earlier approaches and helped bring them into wide use. Google's course gives a classic example of what such a model learns: horse and burro appear in similar contexts, so they end up near each other.
These are called static embeddings because each word has exactly one vector, whatever sentence it appears in. The vectors also depend on the text used for training. Wikipedia notes a 2016 study showing that word2vec trained on Google News reflected gender and racial biases found in that text.
Contextual embeddings
A single vector per word struggles with words that have several senses. "Bank" can be a riverbank or a place to keep money, and a static embedding merges them. Models such as ELMo and BERT, developed in the late 2010s, instead produce an embedding for each occurrence of a word, so the vector reflects the sentence around it. The same approach extends to whole sentences and documents; Wikipedia lists skip-thought vectors from 2015 and Sentence-BERT among the sentence-level methods.
What embeddings are used for
- Recommendations: Google's course uses a food app that suggests crepes to someone who likes pancakes because the two sit near each other.
- Search: documents and queries can both be turned into vectors, and the closest vectors are returned, which is the vector space idea Wikipedia calls the first generation of semantic space models.
- Text analysis: embeddings are fed into systems for tasks such as sentiment analysis and syntactic parsing, where Wikipedia says they improve performance.
- Other data: the same idea has been applied to DNA, RNA and protein sequences, and to analysing gameplay logs.
Embeddings are also commonly used to find passages for chatbots to read before answering, but neither source used for this article covers that, so it is not described in detail here.
What embeddings cannot do
An embedding is a compressed summary, not a definition. Nearby points are expected to be similar in meaning, as Wikipedia puts it, but closeness depends on the training data and on the task the embedding was built for. Two items that are close for one purpose may be far apart for another. Because the dimensions are hard to read, it is also hard to explain exactly why two items were judged similar.
The short version: an embedding replaces a long, empty list with a short, dense one in which distance stands for similarity. That one trick underlies a great deal of modern language software.
Sources
Share
Keep reading
Related articles
What Is Retrieval-Augmented Generation (RAG) in AI?
RAG lets an AI model look up documents before it answers. Learn the steps, why it was proposed in 2020, what it helps with and where it still falls short.
What Is Model Quantization in AI? Smaller, Faster Models
Quantization stores an AI model's numbers in fewer bits so it needs less memory and can run faster. Here is how it works and what it costs in accuracy.
Chain-of-Thought Prompting: Making AI Show Its Steps
Chain-of-thought prompting asks a language model to write out its reasoning steps before answering. Here is what the 2022 paper found and where the idea has limits.