Books › AI Foundations: from zero
Attention and the transformer
Words need their neighbours
Read this sentence: "The animal did not cross the road because it was tired."
What does "it" mean? You know at once that "it" is the animal. The road cannot be tired. To understand one word, you looked back at other words in the sentence.
Now change one word: "The animal did not cross the road because it was wet." Now "it" probably means the road. The same word, a different answer, and only the neighbours changed.
A model that reads language has to do this over and over. It has to work out which other words matter for each word.
The old way: one word at a time
Before 2017, the leading designs read text one word at a time, like a person reading a line through a small window. After each word, they updated a short memory note and passed it on. This had two problems.
- The note forgot. By the end of a long paragraph, early words had faded from the note.
- It was slow. Each word had to wait for the one before it, so you could not read many words at the same time.
The new idea: attention
Attention says: do not squeeze everything into one note. Let each word glance across the whole sentence and judge how much every other word matters to it.
In the picture, when the model reads "it", it pays a lot of attention to "animal", some to "tired" and very little to "road". The thickness of each arrow is the strength of attention. Those strengths are decided by the model, using patterns it learned in training.
Here is a way to picture how it works. Think of a classroom. A student has a question: "Who can help me understand this word?" Each classmate holds up a card that says what they know about, and each one also has some actual information to share. The student compares the question with every card. The best matches get the strongest vote. Then the student listens to everyone, but listens most to the strongest matches, and mixes what they say into a new, better understanding. The model does this for every word at the same time. Its three parts have names: the query is the question, the key is the card, and the value is the information shared.
A model usually does this in several ways at once. One "head" may track grammar, another may track who did what, and another may track topic. The results are combined.
The transformer
In 2017, a group of researchers, mostly at Google, published a paper called "Attention Is All You Need". It introduced the transformer, a design built on attention. The idea was powerful enough that nearly every modern language model uses it.
A transformer is a stack of identical blocks. Each block has two parts:
1. Attention: every word looks at the others and mixes in what matters. 2. A small thinking step: each word, on its own, is processed through a little network (as in Chapter 4).
The model stacks many of these blocks. Early blocks sort out simple things such as grammar. Later blocks work with deeper meaning. Because every word can be handled at the same time, transformers can be trained much faster than the older one-at-a-time designs. That speed is what made it possible to train on enormous libraries of text.
Three flavours
The original transformer was built for translation. It had two halves. One half read the sentence, and the other half wrote the translation. Later people found that one half alone does many jobs well.
| Kind | What it does | Typical use |
|---|---|---|
| Reader only (encoder) | Reads and understands text | Sorting messages, search, producing embeddings |
| Reader and writer (encoder and decoder) | Reads one text and writes another | Translation, summaries |
| Writer only (decoder) | Reads what is written so far and adds the next word | Chatbots and most language models |
The chatbots you know are writer-only models. They read your message and everything written so far, and they continue it. That is the topic of the next chapter.
Back to Asha
A customer writes: "I ordered the lemon cake and the rolls. Can you make it eggless?" Does "it" mean the cake or the rolls? Attention helps the model weigh the clues, such as which item was named first and which one could contain eggs. It is still not certain, so a careful helper would confirm: "Do you mean the lemon cake?"
Words explained
- Attention: a way for each word to look at all the other words and weigh how much each one matters.
- Transformer: a design for language models built from stacked blocks of attention and small thinking steps.
- Encoder: the part of a model that reads and understands text.
- Decoder: the part of a model that writes text, one word at a time.
The limits of this chapter
- The classroom is a picture, not what happens inside literally. In a real model, attention is arithmetic on lists of numbers.
- Attention weights show where a model looked, but they do not fully explain why it answered as it did.
- The cost of attention grows quickly as text gets longer. That is one reason models can only read a limited amount at once (Chapter 7).
For engineers
- Scaled dot-product attention computes softmax(QKᵀ / √d_k) V. Q, K and V are linear projections of the token representations. Multi-head attention runs several such attention operations in parallel with different projections and concatenates the results.
- Each transformer block adds residual connections and layer normalisation around the attention and a position-wise feed-forward network. Because attention has no built-in sense of order, positional information is added to the inputs.
- In a decoder-only model, attention is masked so that a token can only look at earlier tokens. This is what allows left-to-right generation.
- Self-attention compares every token with every other token, so compute and memory grow roughly with the square of the sequence length. Many optimised implementations exist to reduce the cost.
- Encoder-only models such as BERT are common for classification and embeddings. Decoder-only models such as the GPT family dominate text generation.
Check yourself
- Why did the older one-word-at-a-time designs struggle with long text? Answer: Their short memory note forgot early words, and they were slow because each word had to wait for the one before.
- In your own words, what does attention do? Answer: Each word looks at all the other words and decides which ones matter most for understanding it.
- Which flavour of transformer do chatbots use? Answer: The writer-only kind (decoder-only), which continues the text one word at a time.