What Is Training Data? How AI Learns From Examples

Training data is the collection of examples a machine-learning model is shown while it learns. The model is not given a list of rules. It adjusts its internal settings, called parameters, so that its outputs match the patterns in the examples. What the data contains, how it is labelled and how it is kept separate from the data used for testing all shape how well the finished model works.

Labelled data

DataCamp defines labelled data as raw data that has been assigned one or more labels to add context or meaning. Each example pairs an input with a known answer. In a spam filter, emails are tagged as spam or not spam. In speech recognition, audio is paired with a written transcript. In medical imaging, scans are labelled with the correct diagnosis, and for driving systems, images are marked with objects such as pedestrians and other vehicles.

Toloka, a data-labelling company, calls these labels "ground truth": the answer the model is trying to predict. The model makes a prediction, compares it with the label and adjusts to reduce the error. This approach is called supervised learning.

Unlabelled data

Unlabelled data is raw data with no tags. It is usually much easier and cheaper to collect, but the model gets no answer key. In unsupervised learning, the model looks for structure on its own. Toloka lists clustering (grouping similar items) and association (finding things that tend to occur together) as typical methods, with uses such as grouping customers by buying habits or spotting unusual transactions.

There are middle paths. Toloka describes a hybrid approach in which automated methods propose labels and people review and correct them, trading some accuracy for lower cost. Wikipedia's article on data sets notes that the exact methods vary with the problem and the amount of data available.

• Supervised learning: labelled examples, predicts a known kind of answer (a category or a number)
• Unsupervised learning: unlabelled examples, finds groups and patterns without being told what to look for

Why labelling is hard

DataCamp points out that labelling is slow and costly, especially for specialised material like radiology images that may need expert knowledge. People make mistakes, and different annotators can apply different standards. In niche fields, labelled data can be scarce. Toloka adds that labels can carry bias and may expose sensitive information.

These weaknesses pass into the model. If labels are inconsistent, the model learns inconsistent behaviour. If the examples cover some situations far more than others, the model will do better on the common ones. Neither source claims any model is free of this, and it is one reason people examine what data went into a system.

Three sets: training, validation, test

Wikipedia explains that data used to build a model is usually split into three sets, each with a different job.

• Training set: the examples used to fit the model's parameters, such as the connection weights in a neural network
• Validation set: examples used to tune settings chosen by the builder (for instance the number of hidden units) and to compare candidate models
• Test set: data kept apart from training, used at the end to judge how well the final model works on new examples

The reason for the split is overfitting. A model can learn quirks of its training examples that do not hold in general, a bit like a student memorising past exam answers. Wikipedia says that a model that fits the training or validation data much better than the test data is likely overfitted. It also says the test set should never be used to train the model or fine-tune its settings, otherwise it no longer gives an honest check. Where data is scarce, builders may use cross-validation, which rotates which part of the data is held back.

Large language models

Chatbots built on large language models are trained on very large amounts of text, much of which has no human-written label. The usual trick is that the text supplies its own answer: the model sees part of a passage and learns to predict what comes next. The sources used for this article do not cover those systems in detail, so this paragraph is a general outline only. Companies differ in what they say about their data, and many do not publish full details.

What to take from this

A model can only learn what its examples show. More examples, better labels and a fair, held-back test set usually make for a more trustworthy result, while gaps and errors in the data show up later as gaps and errors in the output. When you read that a system was "trained on" something, the useful follow-up questions are what the data was, who labelled it, and how the maker checked the result on examples it had not seen.

Sources

• What is Labeled Data?, DataCamp
• Understanding the difference between labeled and unlabeled data, Toloka
• Training, validation, and test data sets, Wikipedia