benchmark

basicwidespread

A benchmark is a standardized set of tasks and evaluation method used to measure the performance of different AI models on the same scale, ensuring comparable results.

In plain terms

Think of a high school exit exam: everyone gets the same set of tasks, within the same time, with the same scoring, so results can be compared at the end. A benchmark is the same for AI models: pre-set questions or tasks that every model solves, resulting in a score or percentage.

It serves to evaluate which model is capable of what based on numbers rather than flashy promises. Separate benchmarks measure, for example, mathematical reasoning, text comprehension, programming, or whether an agent can independently complete a multi-step task.

It differs from self-testing in that it is standardized and repeatable. However, it only measures what is included in the task list. Just as a good high school diploma does not guarantee that someone will be a good employee, a high benchmark score does not mean the model is best for your task.

How it works

Researchers compile a set of tasks for which the correct solution is known, or success is precisely defined.
The model or agent receives every task under identical conditions: the same instruction, the same tools, the same number of attempts.
Responses are evaluated. This can be done by a program that automatically checks (for example, whether the written code runs), but it can also be done by human reviewers or another AI model.
The result becomes a score, which often appears on a public leaderboard where models can be seen side by side.
When the best models perform almost flawlessly, the benchmark 'saturates,' and a more difficult, new set of tasks is needed.

An everyday example

You read in the news that a new model "achieved 80 percent on a programming benchmark, surpassing competitors". This means it successfully solved four-fifths of a fixed set of tasks. However, it does not reveal whether it will handle your emails or spreadsheets better.

Who is affected

everyoneMost news and advertisements about AI refer to benchmark results, so it is worth knowing what these numbers mean and what they do not.

entrepreneur, executiveIf you choose an AI tool for your company, a benchmark is a good first filter, but you must test it on your own tasks before deciding.

developerYou choose a model or coding agent based on benchmarks, and you can track your own system's quality with similar tests. → Code

bank customer, investorRankings can influence the perception of AI companies and investor sentiment, so it is worth reading them critically. → Money

A common misconception

MythThe model that is first on the benchmark is the best at everything.

In realityA benchmark only measures a narrow, predefined set of tasks. On another task type, in another language (for example, Hungarian), or in your specific work, the ranking could be entirely different.

Limits and risks

Some tasks may leak into the model's training data, causing the model to return the solution from "memory" rather than genuine capability.
Developers tend to tune their models specifically for popular benchmarks, so scores may improve faster than everyday usability.
Companies mostly highlight results where they performed well, so it is worth seeking independent measurements as well.
Most benchmarks are in English, so they say little about how the model performs in Hungarian.

Similar, but not the same

evaluation – Evaluation is the broader activity: any method for testing an AI system. A benchmark is a standardized, repeatable form of this.
ranglista – The benchmark itself is the task set and measurement method, while the leaderboard is the table that ranks the results achieved on it.
training data – The model learns from training data. Benchmark tasks, however, should not be visible in advance, as only then can true capability be measured.

Related concepts

Broader

evaluation, artificial intelligence

Narrower

SWE-bench, MMLU, ImageNet

On the edge

ranglista, Turing-teszt, AI safety, large language model, agent, training data, hallucination, overfitting

A brief history

The word originates from surveying: a 'bench mark' was a chiseled stone marker to which height measurements were referenced. Computing adopted it to compare machine speeds, and in machine learning, the ImageNet image database introduced in 2009 made it the primary benchmark for progress. In the era of language models, the 2018 GLUE became the key reference point, and in the era of agents, the 2023 SWE-bench.

Where things stand

In late 2026, the release of almost every significant model is accompanied by benchmark results, and the focus is shifting increasingly from simple question-answer tests to measuring long, independently performed agent tasks. Meanwhile, older tests are becoming saturated one after another, and criticism is growing that scores reflect real-world usability less and less.

Status: 2026 November

In the news

Appears in 34 Agent Age stories; the curve shows weekly occurrence in the news magazine.

Almost half of Visa's AI shopping agent test runs failNov 5, 2026 · Marketing
Visa test: AI shopping agents failed 42.8% of runsNov 5, 2026 · Money
AI agents are taking over HR processes, but who governs them?Nov 4, 2026 · Work & HR
Surge AI tax benchmark: Claude Sonnet 5.5 leads AI agents with 10.8%Oct 29, 2026 · Money
Harvey acquires Aida, gains FedRAMP Class D certification and unveils legal AI agent benchmarkOct 26, 2026 · Jog
Semrush AI Visibility Index: AI platforms recommend the same handful of brandsOct 12, 2026 · Marketing
Coding benchmarks mismeasure ability, and even the best models fail, critics sayOct 12, 2026 · Code
The Hackett Group introduces AI world-class supply chain benchmarksOct 12, 2026 · Ipar

Also known as

benchmark · benchmarkteszt · performance test · comparative test · benchmark test · AI-benchmark · benchmarking · AI benchmark

Sources

Jimenez et al.: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv, 2023)

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.