eval
in English: evaluation (eval)
An eval is a systematic test that measures how well, reliably, and securely an AI model or agent performs on pre-defined tasks.
In plain terms
Eval is the testing of AI systems. Just as a car is taken for a technical inspection before being put on the road, language models and agents are tested on task sets before being entrusted to you. The question is always what the system can actually do, not what the advertisement promises.
This is necessary because an AI model does not work like a calculator: it can answer the same question in multiple ways and sometimes confidently errs. A few trial questions do not reveal reliability. An evaluation measures the same model across hundreds or thousands of tasks and provides a numerical result.
An eval does not only measure knowledge. It can also check whether the system follows the rules, resists malicious requests, or whether an agent can see a multi-step task through to the end. Developers rerun it after every change to see whether the system has improved or got worse.
How it works
- First, they decide what to measure: for example, answer accuracy, Hungarian text understanding, coding capability, or safe behavior.
- They compile a set of tasks and provide the correct solution for each task or define what counts as a good answer.
- The model or agent solves all tasks, often multiple times, because its answers can vary from run to run.
- Answers are scored: in simple cases, a program compares them with the solution; in more complex cases, a human expert or another AI model evaluates them.
- The results are aggregated, compared with previous versions or other models, and this forms the basis for deciding on improvements or deployment.
An everyday example
You read in the news that a new model "achieved 90 percent" on a test, or "surpassed its competitors" – this is an eval result. The same happens on a smaller scale when your bank's customer service chatbot is tested with hundreds of real customer questions before going live, to see how many it answers correctly.
Who is affected
A common misconception
Limits and risks
- Public test tasks can end up in the models' training data. In such cases, the score measures "memorized" answers, not real capability.
- Manufacturers tend to highlight tests where they performed well, so treat self-reported results with caution.
- Most of the best-known tests are in English, so their results say little about how the system performs in Hungarian.
- If another AI model scores the answers, the judge can also be wrong or biased.
Similar, but not the same
- benchmark – A benchmark is a standardized, usually public task set on which many models are compared. Eval is the broader term: it means any measurement, even one created for your own purpose.
- software testing – Traditional software testing typically provides a clear good–bad answer. Eval, however, measures systems with variable outputs, so it works with ratios and scores.
- red teaming – During red teaming, people or programs deliberately attack the system to find new flaws in it. An eval is typically a repeatable, predefined measurement. The two are connected: flaws found during red teaming often become new eval tasks.
Related concepts
AI safety, quality assurance, machine learning
benchmark, agent evaluation, security assessment, AI as judge
red teaming, hallucination, ranglista, tesztadat, observability
A brief history
The idea of testing machine intelligence dates back to Alan Turing's 1950 study, which proposed the test later known as the Turing test. The short form "eval" became common parlance in the industry in the era of large language models, especially after OpenAI open-sourced its Evals framework in 2023.
Where things stand
At the end of 2026, evals are an everyday part of model development, and the major AI companies regularly publish eval results when they release their models. However, reliably measuring agents' long, multi-step work is still immature, and older tests quickly become obsolete.
In the news
- Demand for AI skills grows across professions, and employers pay for themNov 2, 2026 · Work & HR
- Epic to keep AI transparency tools despite federal rollbackOct 30, 2026 · Health
- Spintronics AI weighs ₹2,800 crore GaN chip fab in Madhya PradeshOct 27, 2026 · Ipar
- OpenAI to stop training frontier models on the live internet after its agents disabled monitoringOct 20, 2026 · Agent Age
- More than half of managers find it harder to evaluate staff work produced with AIOct 19, 2026 · Work & HR
- OpenAI shows better code generation in low-resource languages and for COBOL migrationOct 19, 2026 · Code
- AHA, AMA, ATA and radiology groups respond to FDA plan for generative AI medical devicesOct 19, 2026 · Health
- MHRA promises closer FDA cooperation on AI medical devices at AdvaMed; Japan's PMDA shares principlesOct 19, 2026 · Health
Also known as
eval · evals · evaluation · evaluation · model evaluation · AI evaluation · AI evaluation · model evaluation
Sources
This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.