validation set

advancedwidespread

The validation set is the reserved part of the data on which developers check and tune the model during training, but from which the model does not learn directly.

In plain terms

Imagine you are preparing for your school-leaving exam. You learn from textbook exercises, and the mock exam reveals whether you truly understand the material or have merely memorized the solutions. The validation set is AI's mock exam: examples that the model does not receive as study material during training, but only as a test.

It is useful for developers to make decisions on the fly: how long training should last, how large the model should be, and which settings work better. If the model improves on the training data but starts to worsen on the validation set, that is a sign of rote memorization. This is called overfitting.

It is different from the training set, because the model actually learns from that. And it is different from the test set, because that is the real, one-time exam at the very end. You can retake the mock exam multiple times and adjust your preparation based on the result. However, after the actual exam, you cannot study again to get a better grade, otherwise the result would no longer show what you truly know.

How it works

Developers split the available data before training: the largest part is the training set, a smaller part is the validation set, and another smaller part is the test set (a common ratio is 80–10–10 percent).
The model learns only from examples in the training set; it does not see the validation examples at this stage.
During training, the model's performance on the validation set is regularly measured and compared with the results achieved on the training set.
Decisions are made based on measurement: they adjust settings (so-called hyperparameters), select the best among several model variants, or stop training before the model starts to overfit.
The final, selected model is measured once on the untouched test set, and this provides the most reliable estimate of what to expect in live use.

An everyday example

A bank creates a model to filter out suspicious card payments. It trains the model on past transactions, sets aside the most recent few months of transactions as a validation set, and uses this to decide how strict the filter should be. This reduces the chance of your card being wrongly blocked due to a foreign purchase, while still catching most actual fraud.

Who is affected

developerThis set decides which model variant and which settings are released to users. → Code

entrepreneur, executiveIf you order an AI solution, it is worth asking on what data it was validated, as a poorly validated model may excel in the demo but disappoint in production.

patient, doctorA diagnostic AI is only reliable if it has been tested on data from patients it did not see during training, preferably from other hospitals or using images taken with different machines. → Health

everyoneIt helps you read news about "99 percent accurate AI" critically, because you can always ask: on what data was this measured?

A common misconception

MythThe validation set and the test set are the same, just with different names.

In realityThe validation set is used multiple times during development for decisions, so the model indirectly adapts to it. The test set is for the final, one-time measurement and remains untouched until then. Confusion is increased by the fact that literature and some software libraries sometimes mix the two terms, while in medical research, 'validation' often refers specifically to the independent, final check.

Limits and risks

If developers tune too often against the same validation set, the model indirectly adapts to it as well, and the results look better than they really are.
It is only valuable if it resembles real-world situations: if the held-out data does not reflect future users, good results can be misleading.
In data leakage, where the same or a very similar example ends up in both the training and validation sets, the measurement gives a falsely positive picture.
With time-varying data (such as exchange rates or transactions), random splitting yields misleading results because the model thus sees examples from the "future." In such cases, the validation set must contain data from periods later than the training data.
With little data, the reserved part may be too small for reliable measurement; in such cases, cross-validation is usually applied.

Similar, but not the same

training set – The model learns directly from the training set, while the validation set is used only to check how well it handles unknown examples.
teszthalmaz – The test set is the final exam taken once at the end of development; the validation set is the practice exam taken along the way, several times.
cross-validation – Cross-validation is a procedure where different slices of data alternately serve as the validation set, rather than a single fixed portion.

Related concepts

Broader

machine learning, adathalmaz, model evaluation, supervised learning

Narrower

–

On the edge

training set, teszthalmaz, cross-validation, early stopping, overfitting, Hyperparameter, benchmark

A brief history

The idea originates from statistics: the cross-validation method was widely popularized by Mervyn Stone's 1974 and Seymour Geisser's 1975 studies. The current three-way split into training, validation, and test sets became standard practice in the 1990s during research on neural networks and machine pattern recognition.

Where things stand

At the end of 2026, using a validation set is a fundamental, unavoidable practice in machine learning, including when fine-tuning large language models. Growing attention is being paid to the question of how to keep the set-aside data truly separate from the huge training datasets collected from the internet.

Status: 2026 November

Also known as

validation set · validation dataset · validation set · validation data · validation set · validation data · development set · dev set

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.