Glossary › Language models

LLM as judge

in English: LLM-as-a-judge

An evaluation method where a language model scores or rates the responses of another AI system based on predefined criteria, replacing human judges.

In plain terms

Imagine a teacher who has to grade a thousand essays over a single weekend. Unable to do it alone, they provide a grading key to a prepared substitute, who reviews and scores them all. The LLM as a judge works the same way in the world of machines: a large language model receives the grading key and scores the answers of another AI.

It is needed because the quality of text responses is difficult to measure mechanically. In a math problem, it is enough to check if the final result is correct. However, whether a summary is accurate, a customer service reply is polite, or an explanation is understandable could previously only be reliably assessed by a human, which is slow and expensive. Older machine metrics mostly counted word matches against a reference text without understanding the content.

The judge model is not infallible, but it is much faster and cheaper than human evaluation and does not get tired: it can review thousands of answers in a few minutes. Therefore, it is mainly used for mass checks, while sensitive or disputed cases are still reviewed by humans.

How it works

1. Developers define evaluation criteria, such as: is the answer accurate, does it rely on the provided source, is it understandable, and is it non-offensive.
2. The judge model receives the original question, the answer to be evaluated, the criteria, and often a sample solution as well.
3. The judge delivers a verdict: assigns a score (for example, 1 to 5), makes a pass/fail decision, or selects the better of two answers. It usually includes a brief justification.
4. Judgments are aggregated to show whether the system under review improved or deteriorated overall after a modification.
5. The reliability of the judge is checked periodically: a smaller sample is also evaluated by humans to see how closely the machine's verdict matches theirs.

An everyday example

Who is affected

A common misconception

Limits and risks

• The judge model has its own biases: it tends to value longer, more confident-sounding responses more highly, and the order in which it receives answers can also influence its judgment.
• If the judge is the same model that wrote the answer, or a close relative, it tends to rule more favorably.
• It may assign different scores to the same answer across two runs, so it is unwise to draw sweeping conclusions from a single judgment.
• In fields where the judge model itself does not understand the topic well enough (for example, medical or legal questions), it may consider a wrong answer good, so the human expert cannot be omitted here.

Similar, but not the same

• Benchmark – A benchmark is the set of tasks on which models are measured, while an LLM acting as a judge is one possible way to score the answers given to those tasks.
• Human evaluation – There, experts or users rate the responses, which is slower and more expensive, but remains the benchmark to which the machine judge is aligned.
• Guardrail – The guardrail's goal is intervention: it prevents a harmful response from going out in real time. The judge model's goal is measurement: it typically evaluates quality after the fact or during testing. The two can overlap, as some guardrails themselves use a judge model for filtering.

Related concepts

A brief history

The method became known in 2023 when several studies showed that large language models' judgments align well with human evaluators in many tasks. The term 'LLM-as-a-judge' was popularized by a 2023 study examining the automated evaluation of chat models and its biases.

Where things stand

By the end of 2026, this has become a standard tool for testing applications and agents built on language models, and most evaluation frameworks offer it by default. Its biases are known, so careful teams regularly check the judge's rulings against a human-reviewed sample.

Status: 2026 November

In the news

• Courts and AI: Geoffrey Vos's proposal, judicial guidelines and the debate over AI judgesOct 26, 2026 · Jog
• Judges worldwide: AI may decide minor cases, but judicial conscience cannot be replacedAug 24, 2026 · Jog
• Vos: people likely to accept AI judges for small disputesAug 17, 2026 · Jog

Also known as

LLM-as-a-judge · LLM as a judge · LLM judge · judge model · AI judge · LLM judge · LLM-based evaluation · model-based evaluation

Sources

• Zheng et al. (2023): Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv)
• Liu et al. (2023): G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (arXiv)

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.