Books › AI Foundations: from zero

Embeddings and vector databases, in depth

By Kapil Kumar Gupta

Why not just give every word a number?

The simplest idea is a list of words, numbered: "apple" is 1, "banana" is 2, "cat" is 3. But those numbers mean nothing. Apple and banana are no more alike than apple and cat, and the number 2 is not "twice" anything.

Another idea: give every word its own long row of zeros with a single 1 in its own place. This works, but every word is exactly as different from every other word as it can be, so the computer learns nothing about meaning. And with 100,000 words, every row would be 100,000 numbers long, almost all zeros.

An embedding fixes both problems. It is a short list of numbers, say 1,000, in which every number carries a little of the word's meaning. Words that mean similar things get similar lists.

Where do the numbers come from?

Nobody writes them by hand. A model learns them, as in the picture.

1. The model reads a huge amount of text. 2. It plays a guessing game: given "I fed the ___ some milk", which word fits? 3. Each word has a list of numbers. Every time the guess is wrong, the numbers are nudged a little, using the same learning loop you saw in Chapter 4. 4. After billions of guesses, words that fit the same gaps, such as *cat*, *kitten* and *puppy*, end up with similar numbers. Words that fit different gaps, such as *bus*, end up far away.

Another popular way to train is with pairs. Show the model two texts that mean the same thing (for example, a question and its answer) and nudge their numbers closer. Show it two texts that do not match and nudge them apart.

One word, many meanings

The word *bank* can mean a river's edge or a place for money. The earliest embeddings gave each word a single list, so *bank* had to be a mixture. Modern models do better. They give each word a list that depends on the sentence around it. "She sat on the river bank" and "She went to the bank for a loan" produce different lists for *bank*. These are called contextual embeddings, and they come from the attention process you will see in Deep dive D.

What are the numbers?

Each number is not a thing you can read, like "animal-ness" or "size". The meaning is spread across all the numbers together. What matters is the pattern. If you only remember one thing: you cannot read a single number, but you can compare two lists.

How many numbers? A few hundred to a few thousand is common. More numbers can capture finer differences but cost more to store and search. Here is a quick sum. Suppose each embedding has 1,000 numbers, and each number uses 4 bytes. One embedding then takes 4,000 bytes, about 4 KB. A library of one million chunks of text needs about 4 GB just for the embeddings. (This is an illustration. Real sizes depend on the model.)

Measuring closeness with real numbers

Let us use three-number lists, small enough to calculate by hand.

Think of each list as an arrow starting at the origin. The most common measure of closeness is the angle between two arrows, and it is calculated as cosine similarity. The recipe has three steps.

1. Multiply the matching numbers and add them up (the dot product). For dog and puppy: 0.9 × 0.85 + 0.8 × 0.75 + 0.1 × 0.2 = 0.765 + 0.60 + 0.02 = 1.385. 2. Work out each arrow's length: the square root of the sum of squares. Dog: √(0.81 + 0.64 + 0.01) = √1.46 ≈ 1.208. Puppy: √(0.7225 + 0.5625 + 0.04) = √1.325 ≈ 1.151. 3. Divide: 1.385 ÷ (1.208 × 1.151) ≈ 0.996.

A score of 1 means the arrows point exactly the same way. A score near 0 means unrelated. For dog and car: the dot product is 0.09 + 0.16 + 0.09 = 0.34. Car's length is √0.86 ≈ 0.927. So the score is 0.34 ÷ (1.208 × 0.927) ≈ 0.30.

So dog and puppy score 0.996, and dog and car score 0.30. The search for the closest meaning is the search for the highest score.

Why the angle and not the straight-line distance between the tips? The angle ignores how long the arrows are, which often depends on things like how long the text was. What you usually want to compare is the direction, meaning the topic.

Why a vector database?

Suppose Asha has 40 notes. To answer a question, her helper can compare the question with all 40 and pick the best. That takes no time at all, and a plain list is fine.

Now suppose a large organisation has one million chunks of text. Comparing a question with every chunk means 1,000,000 comparisons, each of 1,000 multiplications: a billion multiplications for every single question. With many users asking at once, this is far too slow and too costly.

A vector database is built to avoid that. Its secret is an index: a structure that sorts the vectors by neighbourhood in advance, so a search can jump straight to the right area and compare only a few thousand candidates instead of a million. One common design is a map of "shortcut" links between nearby vectors, so a search can hop from neighbour to neighbour towards the question. Another divides the space into clusters and searches only the closest few.

These indexes are approximate. They may occasionally miss the true best match, and in exchange they are very much faster. That trade is almost always worth it. The method is called approximate nearest neighbour search.

What else a vector database does

Fast search is only part of it. A vector database also handles the ordinary jobs of a database.

• It stores three things for each item: an ID, the vector, and extra details (metadata) such as the original text, its source, the date, the topic and who may see it.
• Filtering. You can say "find the closest items, but only from this year", or "only pages this person is allowed to read". This is very important for privacy. Without it, a search could leak private documents.
• Updates and deletes. Documents change. When one changes, its old vector must be replaced. When one is deleted, its vector must go too.
• Scale and safety. Large systems split the data across machines, keep copies for safety, and save it to disk.
• Keyword help. Many combine meaning-based search with word-based search (hybrid search), because exact words, such as product codes, still matter.

Do you always need one?

No. For a few thousand chunks, a simple list in memory, or a normal database with a built-in vector feature, is plenty. Buying and running a special system before you need it is a common waste. Reach for a vector database when the amount of data, the speed you need, or the filtering rules make a plain list too slow or too risky.

Other uses of embeddings

The same idea does more than answer questions.

• Finding near-duplicates: two texts with a very high similarity score are probably the same.
• Grouping: cluster messages by topic without labels.
• Recommendations: items close to the ones you liked.
• Search by example: a photo, a sound or a piece of code can have embeddings too.

Things that go wrong

• Mixing models. Lists from two different embedding models live on two different maps. Comparing them is meaningless. Always embed the documents and the questions with the same model, and if you change the model, re-embed everything.
• Bad chunks. If a chunk is too small it loses its context. If it is too big it mixes topics. Both blur the meaning.
• Close does not mean true. The search finds text about the right topic. It does not check that the text is correct or up to date.
• Inherited bias. Embeddings learn from human writing, including its unfair habits.
• Hidden orders. Retrieved text is still just text. A page in the library can contain an instruction meant to trick the model (Chapter 9).

Back to Asha

Asha's forty notes live in a simple list, and her helper compares the customer's question with all of them in a blink. Her cousin's catering company has 40,000 notes and several staff asking at once, and it uses a vector database with an index and a filter, so a kitchen helper only sees kitchen notes. Same idea, different scale.

Words explained

• Embedding: a list of numbers that places a piece of text on a map of meaning.
• Contextual embedding: an embedding that changes with the words around it.
• Cosine similarity: a score of how much two arrows point the same way, from near 0 (unrelated) to 1 (the same direction).
• Index: a structure built in advance so searches can skip most of the data.
• Approximate nearest neighbour search: finding very close matches quickly, accepting that the best one might occasionally be missed.
• Metadata filter: limiting a search by details such as date or permission.

The limits of this chapter

• The three-number lists are made up so the arithmetic is easy. Real lists have hundreds of numbers, and their values come from a model.
• The sizes are examples. Real storage depends on the model, the number format and the index.
• Search quality depends on many things beyond the database: the embedding model, the chunking and the questions.

For engineers

• Training objectives include masked or next-word prediction (older word-vector methods), and contrastive learning on pairs (modern text-embedding models), where matched pairs are pulled together and mismatched pairs pushed apart.
• Cosine similarity of vectors a and b is (a · b) / (|a| |b|). If vectors are normalised to length 1, cosine similarity equals the dot product, which is cheaper. Check which metric a given embedding model is trained for.
• Approximate nearest-neighbour indexes: graph-based (for example HNSW), cluster-based (inverted file), and quantisation to shrink vectors (for example product quantisation). They trade recall, latency, memory and build time. Measure recall against exact search on a sample.
• Filtering can be done before the search (restrict candidates), during it, or after it (post-filter, which can return too few results). Store access labels in metadata and enforce them at query time.
• Operations: version the embedding model with the index, re-embed on change, monitor recall and latency, and plan for growth. Hybrid search often combines dense vectors with keyword scoring (such as BM25) and then re-ranks the top results.

Check yourself

• Why is cosine similarity about the angle between two lists? Answer: It compares the direction of the arrows, which reflects the topic, and ignores their length, which can depend on things like how long the text was.
• In the example, which is closer to "dog": "puppy" or "car"? Answer: "puppy", with a score of about 0.996, against about 0.30 for "car".
• Why not scan all the vectors for every question? Answer: With millions of vectors, that means billions of multiplications per question. An index lets the search skip most of the data, at the price of occasionally missing the very best match.