Books › AI Foundations: from zero

Machine learning: learning from examples

By Kapil Kumar Gupta

An everyday picture

Think of a child learning to tell cats from dogs. Nobody gives the child a rulebook. You point and say, "cat", "dog", "dog", "cat". After many examples, the child can name animals never seen before.

Machine learning works the same way. We call the examples data, and the thing that learns is the model. When we show the model examples, we say we are training it.

Two kinds of learning

• With answers (supervised). Each example comes with the right answer, called a label. An email marked "junk". A photo marked "cat". Asha's message marked "order". The model learns to predict the label for new examples.
• Without answers (unsupervised). The examples have no labels. The model looks for groups on its own. Asha could give it her customers' buying habits, and it might find three groups: the morning bread buyers, the weekend cake buyers and the party-order buyers.

Most of the AI you meet in daily life uses learning with answers.

The learning loop

Training is a simple loop.

1. Look at one example, with its right answer hidden. 2. Guess. "I think this message is a complaint." 3. Check. Was the guess right? If wrong, by how much? 4. Adjust a little. Change the model's inner numbers slightly so that the guess would have been better.

Then repeat with the next example, and the next, thousands or millions of times. Each step makes only a tiny change. Together they add up to real skill. Chapter 4 shows what those "inner numbers" are.

Keep an exam pile secret

Here is the most important habit in machine learning. Split your examples into three piles, like school.

• Study pile (training). The model learns from these.
• Practice test (validation). You use these to compare versions of the model and pick the best settings.
• Final exam (test). You keep these locked away and use them once, at the end, to see how the model does on questions it has never seen.

Why so strict? If the exam questions leak into the study pile, the model can score full marks just by remembering them. The score looks wonderful and means nothing. This mistake is called data leakage, and it is one of the most common ways to fool yourself.

Too simple, just right, or memorised

A model can go wrong in two opposite ways.

• Too simple (underfitting). It misses the pattern. It is like a student who only learned the first page of the book. It does badly on the study pile and on the exam.
• Memorised (overfitting). It learns the study pile so closely that it copies even the accidents and noise. It is like a student who memorised last year's paper word for word. It scores very well on the study pile and badly on the exam.

You can spot the second problem by comparing the two scores. A big gap between "study pile score" and "exam score" is the warning sign. Common fixes are to give the model more and better examples, to keep it simpler, or to stop training earlier.

How do we measure "good"?

The obvious measure is accuracy: out of all guesses, how many were right? But accuracy can fool you.

Imagine a smoke alarm tested on 1,000 days. Only one day has a real fire. A lazy alarm that always says "no fire" is right on 999 days. That is 99.9% accurate and completely useless, because it misses the one fire.

So we use two more measures, and each has a plain meaning.

• Precision: when the model says "yes, junk", how often is it right? High precision means few false alarms. This matters when a wrong "yes" is costly, for example blocking a customer's real order.
• Recall: out of all the real junk, how much did the model catch? High recall means few misses. This matters when a miss is costly, for example missing a real fire.

You often cannot have both at their best. Choosing the balance is a decision about what mistakes cost, not only a technical choice.

When the answer is a number instead of a label, such as "how many loaves will we sell tomorrow?", we measure how far off the guess is on average.

The whole workflow, start to finish

Training is only one step in a longer story. A real machine learning project follows ten steps, and it goes round in a loop.

Here is each step, using Asha's message sorter as the example.

1. Define the problem. Say exactly what the model must predict, and how you will know it is good. "Sort each message into order, question or complaint, and catch at least 95 in 100 complaints." 2. Collect data. Gather real examples, with the right answers attached. Asha uses a year of her own messages, which a person has marked. 3. Clean and prepare. Fix gaps, typing mistakes and duplicates, and remove private details. Then split the data into the three piles from earlier, before anything else. 4. Explore the data. Look before you trust. How many messages are complaints? Asha finds only 3 in 100, so complaints are rare. That matters for how she measures success. 5. Build features. Choose the clues the model will see: the words in the message, how long it is, the time it arrived. Be careful not to include a clue that gives the answer away, such as "refund sent", which is only known after the fact. 6. Choose a model. Start with a simple one. A simple model that works is better than a fancy one you cannot explain. 7. Train. The model learns from the study pile, using the loop above. 8. Evaluate and tune. Check the practice pile, adjust the settings, and look at the mistakes. When you are done, use the final exam pile once. 9. Deploy. Put the model where people can use it: the shop's inbox, a website, a phone. 10. Monitor and maintain. Watch it. The world changes (new products, new words), and the model slowly gets worse. When it slips below your alarm line, collect fresh examples and train again.

Two things to notice. First, most of the time goes into the data (steps 2 to 5), not into the model. Second, the work does not end at launch. Steps 9 and 10 are where a model either helps people for years or quietly fails.

Back to Asha

Asha trains two helpers. One sorts messages into order, question or complaint. It uses labelled examples, so it is learning with answers. The other predicts how many loaves to bake tomorrow, using past sales, the weather and the day of the week. Before she trusts either, she tests them on a pile of old messages and past days they never saw. The sorter gets most right, but it misses a few complaints. Since a missed complaint is costly, she tunes it to catch more of them, even if it sometimes raises a false alarm.

Words explained

• Training: showing a model examples so it can learn.
• Label: the right answer attached to an example.
• Model: the learned thing that turns an input into a guess.
• Overfitting: learning the study examples too closely, so it fails on new ones.
• Data leakage: exam information sneaking into the study pile.
• Accuracy, precision, recall: three ways to score a model, each telling a different story.
• Workflow (lifecycle): the ten steps of a machine learning project, from defining the problem to watching the model after launch.
• Drift: the slow change in the world that makes a once-good model worse.

The limits of this chapter

• A model is only as good as its examples. Old, biased or sloppy examples give old, biased or sloppy results.
• A good exam score means "good on questions like the exam". The real world can change, and models can slowly get worse. Teams must keep watching.
• Real projects have more steps, such as cleaning the data and choosing which clues to give the model.

For engineers

• Splits. Fit parameters on the training set, tune hyperparameters on the validation set, and report once on the held-out test set. With small datasets, use k-fold cross-validation and report the average. Fit preprocessing (scaling, encoding, feature selection) inside each training fold only, so no information leaks from validation or test data.
• Classification metrics. Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 is their harmonic mean and suits imbalanced classes. ROC-AUC measures how well classes are separated across thresholds. Use a precision-recall curve when you must choose a threshold.
• Regression metrics. MAE is in the same unit as the target and is robust to outliers. RMSE penalises large errors more. R squared is the share of variance explained.
• Fit diagnostics. Low training error with high test error means overfitting (high variance). High error on both means underfitting (high bias). Fixes: regularisation (L1, L2), dropout, early stopping, a simpler model or more data for overfitting; a richer model, better features or longer training for underfitting.
• Workflow. Define the problem, collect and explore data, split, train, evaluate, deploy, then monitor for drift.

Check yourself

• Why do we keep a final exam pile secret? Answer: So we can see how the model does on questions it has never seen. If it leaks into training, the score is meaningless.
• A smoke alarm that always says "no fire" is 99.9% accurate. Why is it useless? Answer: Fires are rare, so being right on the no-fire days hides that it misses every real fire. Recall catches this.
• What does it mean if a model scores very high on the study pile and low on the exam? Answer: It memorised the examples (overfitting) instead of learning the pattern.