Books › AI Foundations: from zero

From words to numbers

By Kapil Kumar Gupta

The problem

In Chapter 1 we learned that computers store everything as numbers. A photo is a grid of numbers. But how do you turn a sentence into numbers in a way that keeps its meaning? If you just number the words alphabetically, then "apple" and "apricot" end up next to each other, but "cat" and "kitten" are far apart. That tells the computer nothing useful.

We need numbers that capture what words mean. It takes three small steps.

Step 1: cut text into tokens

First the text is cut into small pieces called tokens. A token is often a whole short word, such as "love". A longer or rarer word may be split into parts, such as "bak" and "ing". Punctuation marks and spaces count too.

Why not just use whole words? There are too many words, and new ones appear all the time, like names and slang. Pieces are a clever compromise. A small set of pieces can build almost any word.

This has a practical effect you will meet again and again. Token counts do not match word counts. In English, roughly three tokens cover two words. Systems usually charge by the token, and they limit how many tokens they can handle at once.

Step 2: give each token a number

Each token has a place in a big list, so each gets an ID number. Think of a school roll call: the name on the list becomes a number. These numbers are only labels. "Token 9211" does not yet mean anything.

Step 3: turn each number into a map position

Here comes the clever part. Each ID is replaced by a long list of numbers called a vector. This list is the token's address on a map of meaning. We call this list an embedding.

On this map, things with similar meaning sit close together. "Apple", "banana" and "mango" are neighbours. "Cat", "dog" and "horse" are neighbours somewhere else. A brand new word like "puppy" lands close to "dog" because it is used in similar sentences.

The map in the picture has two directions, so we can draw it. Real maps have hundreds or thousands of directions. That gives them room to hold many kinds of closeness at once: both "is a pet" and "is small" and "is friendly".

Nobody draws this map by hand. A model learns it by reading huge amounts of text. A good way to see why it works is the saying "you know a word by the company it keeps". Words that appear in similar sentences end up with similar addresses.

Why this is so useful

Once meaning is a position, finding related things is just finding the nearest neighbours. A customer writes, "Do you have anything for someone who cannot eat eggs?" The word "eggless" is not in the message. But its position is close to the meaning of the question. So the search finds the eggless cakes. A plain word search would have missed them.

An older idea, for comparison

Before embeddings, a common trick was to count words. A word such as "the" appears everywhere, so it says very little. A word such as "sourdough" is rare overall, but if it appears many times in one document, it tells you a lot about that document. Scoring words this way is called TF-IDF. It still works well for exact keywords. But it cannot see that "eggless" and "no eggs" mean the same thing. Embeddings can.

A shelf you can search by meaning

If you have thousands of documents, you can turn each piece into a vector and store them. A vector database is a store built for exactly this. Each entry has three parts:

• An ID: a unique label, so you can update, delete and trace the entry later.
• The vector: the address of the meaning.
• Extra details (metadata): the original text, where it came from, the date, the topic, who may see it. These are not used to measure closeness. They are used to filter and explain. For example: "only search documents from this year", or "show where this answer came from".

Think of a library with a very clever librarian. You describe what you want in your own words, and she walks straight to the right shelf. The index cards behind her desk hold the extra details.

Back to Asha

Asha types her menu, ingredient lists and shop rules into a list of short notes. Each note becomes a vector and goes onto the shelf with details like "vegan: yes" and "last updated: this month". Now a customer's question can find the right note even when the words differ. In Chapter 10, we will see how her helper uses these notes to answer.

Words explained

• Token: a small piece of text, often a word or part of a word.
• Vector: a list of numbers.
• Embedding: a vector that stands for the meaning of a token, a sentence or a document.
• Similarity search: finding the stored vectors closest to a new one.
• Vector database: a store built to hold vectors and search them by closeness.
• Metadata: extra details attached to an entry, used for filtering and explaining.

The limits of this chapter

• Close does not mean true. A search may return text that is about the right topic but gives the wrong answer.
• Embeddings learn from human text, so they can carry its unfair habits and gaps.
• Different embedding models make different maps. Never compare vectors from two different models, and always use the same model for the documents and for the questions.
• The token splits and ID numbers in the picture are made up to show the idea. Real ones differ from model to model.

For engineers

• Subword tokenisers such as byte-pair encoding or SentencePiece build a fixed vocabulary of pieces. The same text can have different token counts in different models.
• An embedding model maps a token, sentence or document to a dense vector, commonly with hundreds to a few thousand dimensions. Closeness is usually measured by cosine similarity or dot product.
• At scale, vector databases use approximate nearest-neighbour indexes (for example graph-based or cluster-based) to trade a little accuracy for large speed gains. Many systems combine vector search with keyword search (such as BM25) and then re-rank, which usually improves recall.
• Metadata filtering can be applied before or after the vector search. Store provenance and a timestamp in the metadata so answers can cite sources and show how old they are.
• If you change the embedding model, re-embed everything.

Check yourself

• Why are texts cut into tokens instead of whole words? Answer: There are too many words, and new ones keep appearing. A smaller set of pieces can build almost any word.
• What does an embedding do? Answer: It turns a piece of text into a list of numbers that places it on a map of meaning, where similar things are close.
• Why can a vector search find "eggless cakes" for the question "something for someone who cannot eat eggs"? Answer: The meanings are close on the map, even though the words differ.