Books › AI Foundations: from zero

How a language model writes

By Kapil Kumar Gupta

Autocomplete, grown up

Your phone suggests the next word while you type. Type "Happy birthday to" and it offers "you". It has seen that pattern many times.

A language model does the same thing, but on a much larger scale. It has read a huge amount of text and has learned which words tend to follow which. Given some text, it works out a list of likely next words, picks one, adds it, and repeats.

That is it. Everything a chatbot writes, a poem or a plan or some code, is produced this way, one small piece at a time. The surprising thing is how much skill can come out of such a simple job.

The journey of one word

Here is what happens when you type a question.

1. Your text is cut into tokens and turned into vectors (Chapter 5). 2. The transformer layers read them with attention (Chapter 6), so that every word is understood in context. 3. The model scores every possible next token. A typical model knows tens of thousands of pieces, and each one gets a score. 4. The scores become chances that add up to 100%. 5. One token is chosen. More likely tokens are chosen more often, but not always. 6. The chosen token is added to the text, and the whole process runs again for the next one. 7. The finished pieces are turned back into readable words and often appear on your screen one at a time. That is why chat answers seem to type themselves.

In the picture, after "The sky is", the word "blue" is the most likely at 55%. But "clear" or "cloudy" could be chosen too. Even "falling" has a tiny chance. These numbers are made up to show the idea.

Why the same question can get different answers

If the model always picked the single most likely word, its answers would be dull and always the same. So most systems add a little controlled chance. It picks among the likely words, in proportion to their chances. This is why asking the same question twice can give two different answers, both sensible.

You can steer this with a few settings:

• Temperature. This is a dial for adventurousness. At a low temperature, the model almost always takes the top word. Answers are steady and predictable. At a high temperature, less likely words get more of a chance. Answers are more varied and creative, but also more likely to wander or go wrong.
• Top-k. Only consider the k best words, such as the top 5, and ignore the rest. It is like only looking at the first five dishes on a menu.
• Top-p. Consider the smallest group of words whose chances add up to a chosen total, such as 90%. When the model is sure, that group is small. When it is unsure, the group is larger. It is like only looking at the dishes that together cover most of what people order.

None of these change what the model knows. They only change how it picks. The same model with different settings behaves differently.

The desk that holds the conversation

A model cannot read unlimited text at once. The amount it can look at is called its context window, and it is counted in tokens. It includes your question, any earlier messages and documents you gave it, and the answer being written.

Picture a desk. Everything on the desk, the model can see. If the pile grows too big, the oldest pages fall off the desk, and the model no longer sees them. Newer models have bigger desks, but a bigger desk is slower and costs more.

Size: parameters

You will often hear that a model has billions of parameters. These are the adjustable numbers inside it, the weights from Chapter 4. More parameters usually means a bigger and more capable model, but also more cost to train and to run. The number alone does not tell you how good a model is for your job.

Reading is quick, writing is slow

There is one more practical fact. When you send a long question, the model can read all of it at once, so reading is fast. But when it writes, it must produce one token at a time, because each new token depends on the one before it. So writing takes longer than reading. This is one reason long answers take time and cost more.

Back to Asha

Asha tries two uses of her helper. For replying to customers, she wants steady, factual answers, so she sets a low temperature. For fun captions on her shop's photos, she raises the temperature and gets more playful ideas. The same model, with a different dial, does both jobs.

Words explained

• Language model: a model that has learned which words tend to follow which, and writes by choosing the next word.
• Temperature: a dial that controls how adventurous the word choice is.
• Top-k and top-p: two ways to limit the choice to the likeliest words.
• Context window: how much text a model can look at at once, counted in tokens.
• Parameters: the adjustable numbers inside a model.

The limits of this chapter

• The word "word" is a simplification. The model really picks tokens, which may be parts of words.
• A language model does not look facts up. It continues the text in a way that fits its patterns. That is why it can sound sure and be wrong, which Chapter 9 explains.
• The chances in the picture are invented. Real ones come from the model and are not shown to most users.

For engineers

• The final layer produces a vector of logits, one per vocabulary item. Softmax converts logits to a probability distribution. Temperature divides the logits before the softmax: values near 0 approach greedy decoding (always choose the top token), and higher values flatten the distribution.
• Top-k truncates to the k highest-probability tokens. Top-p (nucleus sampling) keeps the smallest set whose cumulative probability exceeds p. Then a token is sampled from the renormalised distribution.
• Inference has two phases. Prefill processes all prompt tokens in parallel and is mostly limited by compute. Decode generates one token per step and is mostly limited by memory bandwidth. The KV cache stores each earlier token's keys and values so they are not recomputed, and its size grows as the text gets longer, so very long inputs need a lot of memory.
• Speculative decoding lets a small draft model propose several tokens that the large model then verifies in one pass, which can reduce latency without changing the output distribution.
• Even at temperature 0, outputs can differ slightly between runs because of how parallel computation is arranged, so do not assume strict repeatability.

Check yourself

• In one sentence, how does a language model write a paragraph? Answer: It chooses the next token, adds it to the text, and repeats, many times.
• What does a higher temperature do? Answer: It gives less likely words a better chance, so answers become more varied and creative, and also more likely to go wrong.
• What happens when a conversation grows longer than the context window? Answer: The oldest text falls out of view, and the model can no longer use it.