DPO

in English: Direct Preference Optimization

professionalwidespread

A fine-tuning method where a language model learns from response pairs which answer humans prefer, without a separate reward model.

In plain terms

Imagine teaching a novice chef. Instead of writing a scoring rulebook, you have them taste two versions of every dish and tell you which is better. After many such pairs, they get a feel for what guests like. DPO does the same with a language model: it shows the model one answer judged better and one judged worse for the same question.

It is useful for refining the style and behavior of an already trained model. This makes the model more polite, precise, or concise, or teaches it when to reject a request. New facts are not taught this way; rather, it learns which type of response to choose from possible options.

In the previous solution, a separate "jury" had to be trained first to score the answers, and the model competed for this jury's points. DPO skips this detour: the model learns directly from pairs in a single training phase. This makes training simpler, cheaper, and generally more stable, so smaller teams can use it.

How it works

They gather many triplets: a question, a preferred answer, and a rejected answer. The label is provided by a human or another model.
A frozen copy is created from the starting model. This serves as the reference point against which the learning model is measured.
For every pair, training adjusts the model's internal weights so that the better answer becomes more likely than the worse one, compared to the frozen copy.
A configurable value (the beta) determines how far the model can drift from its original self. This reduces the risk that it will forget what it previously knew.
At the end, the model is tested on separate benchmarks to verify it provides better answers and has not merely memorized the training pairs.

An everyday example

In a chatbot, you receive two answers to the same question and are asked which one you prefer. Your click creates a 'this is better, that is worse' pair. These pairs provide exactly the type of data needed to tune the next version of the model using DPO.

Who is affected

developerUsing open tools or the provider's fine-tuning interface, you can adapt your own model to your own preference pairs without complex reinforcement learning. → Code

entrepreneur, executiveThe tone and style of a customer service or internal assistant can be relatively cheaply tailored to your company if you have enough good–bad response pairs.

everyoneYour like–dislike feedback in chatbots can shape how future models respond through such methods.

A common misconception

MythWith DPO, the model acquires new knowledge and becomes smarter.

In realityDPO tunes which answer the model should prefer based on its existing capabilities. It does not teach facts or fill knowledge gaps.

Limits and risks

The model will be only as good as the labels of the pairs: if the labelers are biased or inconsistent, the model will learn that too.
The model is easily pushed toward pleasing, flattering responses because people often prefer what sounds confident, even if it is wrong.
With poor settings or insufficient data, a model may overfit and perform worse on other tasks.
For complex, multi-step tasks, such as a long agent workflow, comparing a single response pair is insufficient; other methods are also needed.

Similar, but not the same

Reinforcement learning from human feedback (RLHF) – RLHF first trains a separate reward model and improves the language model based on its scores, while DPO learns directly from response pairs.
Supervised Fine-Tuning – Supervised fine-tuning shows the model only good example responses, while DPO pits a better response against a worse one.
Data Protection Officer (DPO) – The same abbreviation in the GDPR refers to the job title Data Protection Officer, which has nothing to do with model training.

Related concepts

Broader

Fine-tuning, Preference learning, Alignment, Machine learning

Narrower

–

On the edge

Reinforcement learning from human feedback (RLHF), Jutalommodell, Supervised Fine-Tuning, LoRA, Nagy nyelvi modell (LLM)

A brief history

The method was described in 2023 by researchers at Stanford University (Rafael Rafailov and colleagues) in the paper 'Direct Preference Optimization: Your Language Model is Secretly a Reward Model.' The study showed that a separate reward model can be omitted.

Where things stand

As of late 2026, DPO and its improved variants are established tools for post-training language models. They are supported by open-source training libraries and available as preference-based tuning on the fine-tuning interfaces of several major providers.

Status: 2026 November

Also known as

DPO · Direct Preference Optimization · direct preference optimization · direct preference optimization · direct preference optimization · Preference-based fine-tuning · preference fine-tuning · DPO fine-tuning

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.