preferenciaadat

in English: preference data

advancedwidespread

Training data that records which of two or more AI responses people judged better.

In plain terms

Preference data does not tell the model what the correct answer is, but which of two answers was preferred. It is like an eye exam: the doctor does not ask how many diopters your eyes have, only whether 'this is better, or that?'. Many such small decisions eventually form a precise picture.

It is useful because a good answer rarely has a single correct solution. It is difficult to codify rules for what makes a text polite, understandable, or helpful, but if you see two versions side by side, you almost immediately know which is better. This comparative judgment is collected in large quantities.

Traditional labeled data makes a statement about a single item ('there is a cat in this image'). Preference data, by contrast, involves comparison: A is better than B. This is used to fine-tune the style, usefulness, and safe behavior of language models after initial training.

How it works

The model produces two or more different answers to the same question.
A human (paid evaluator, expert, or the user themselves) marks which is better or ranks them. Sometimes another AI model judges.
From the resulting triples (question, winning answer, losing answer), datasets of tens or hundreds of thousands of items are built.
From this, either a separate 'scoring' model is trained to predict what a human would choose, or the language model itself is tuned directly so that it makes the kind of answers that win more likely.
The process is repeated: the new model's answers are compared again, so the behavior is refined step by step.

An everyday example

If a chatbot displays two answers side by side and asks which you prefer, your choice becomes preference data. An indirect signal is also generated if you regenerate the answer and continue working with the second version. The thumbs-up or thumbs-down is a related feedback mechanism, but strictly speaking, it evaluates a single answer rather than comparing two.

Who is affected

everyoneThe tone and caution of chatbots are largely shaped by what other people previously judged to be better, and your clicks can feed into this too.

developerIf you fine-tune a model for your own purposes, the quality and consistency of the preference data determine whether it will truly be better or just make different mistakes. → Code

entrepreneur, executiveYour company's customer service or internal AI tool aligns with your expectations only if you gather such data from your own employees' judgments.

employeeData evaluation has become an independent profession; experts (doctors, lawyers, programmers) can also be hired to compare answers. → Work & HR

A common misconception

MythPreference data shows the model which answer is correct.

In realityIt only shows which one the evaluator preferred. A confident, well-written but incorrect answer can easily beat an accurate but drier one.

Limits and risks

The evaluators' taste, culture, and language become embedded in the model, so what appeals to a narrow group becomes the default for everyone.
A model tuned for preference tends to flatter and agree with you because people are more likely to rate such responses as better.
People often disagree with each other, and the same person may judge differently when tired, so the data is noisy.
Your feedback and the associated conversations raise data protection issues, so it is worth checking in the service settings whether they are used for training.

Similar, but not the same

labeled data – A label states something about an element in itself, while preference data records the relative order of two or more responses.
user preferences (personalization) – That refers to your individual settings and taste within a service, whereas preference data consists of many people's judgments used as training material for the model's general behavior.
evaluation data (benchmark) – A benchmark measures a finished model, whereas the model learns from preference data. Pairwise comparisons can also produce a model ranking, but in that case the data serves measurement, not training.

Related concepts

Broader

training data, human feedback

Narrower

pairwise comparison, ranking data, szintetikus preferenciaadat

On the edge

fine-tuning, reinforcement learning from human feedback, jutalommodell, direct preference optimization, data labeling, sycophancy

A brief history

The idea of learning from pairwise comparisons is old, but it was widely popularized in AI research by a joint study from OpenAI and DeepMind researchers in 2017. OpenAI tested it on text summarization with a language model in 2020, and the 2022 InstructGPT study showed that large, general-purpose language models can also be fine-tuned this way. In 2023, Direct Preference Optimization (DPO) was introduced, which utilizes this data without a separate scoring model.

Where things stand

As of late 2026, the standard material for fine-tuning large language models. Collection increasingly relies on expert evaluators and comparisons generated by AI models.

Status: 2026 November

Also known as

preferenciaadat · preferenciaadatok · preferencia-adat · preference data · preference dataset · human preference data · pairwise comparison data · pairwise comparison data

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.