RLHF

in English: reinforcement learning from human feedback

advancedwidespread

A training method where humans evaluate the language model's responses, and the model learns from these evaluations what kind of response we consider good.

In plain terms

A language model first learns from vast amounts of text how words follow one another. This does not yet tell it what constitutes a useful, polite, or safe response. RLHF fills this gap: humans show which response they prefer, and the model adapts to it.

It is like when a culinary student has read every recipe but truly learns to cook from tastings. They are not told exactly what to put in the pot, but which of two plates tasted better. After many such tastings, they get a feel for their preferences.

This is different from traditional training, where the correct solution is provided in advance for every question. There is no formula for what constitutes a 'good answer' in a conversation, but between two answers, a human can easily select the better one. RLHF transforms this easily given judgment into a signal from which the model can learn.

How it works

At the start, there is a pre-trained language model that writes fluently but has not yet aligned with our expectations.
The model provides multiple answers to the same question, and human evaluators rank or score them: which is more useful, accurate, or safe.
A separate model, called a reward model, is trained from evaluations to learn to predict how much people would like a given response.
The language model then generates many responses, the reward model scores each one, and the model's settings are adjusted step by step to achieve higher scores.
The cycle is repeated multiple times with fresh human evaluations, and it is specifically checked whether the model has learned cheap tricks to increase its score.

An everyday example

You ask a chatbot how to write a complaint letter to your service provider, and you get a well-organized, polite, to-the-point answer. This is largely because human raters previously marked which of thousands of responses was better. When you click the thumbs-up or thumbs-down below a response, you give a similar signal that developers can later use.

Who is affected

everyoneThe tone, helpfulness, and refusal patterns of chatbots are largely shaped by such training.

developerIf you tune your own model, you must know that the quality of the evaluators and the wording of the guidelines directly determine the model's behavior. → Code

entrepreneur, executiveWhen choosing an AI service provider, it is worth asking who evaluated the model's responses and by what criteria.

employeeThe evaluation is carried out worldwide by many, often underpaid data workers, meaning real human labor stands behind the method. → Work & HR

A common misconception

MythThe chatbot learns immediately from every conversation and becomes smarter on the spot based on your feedback.

In realityRLHF occurs during development in a separate training phase; the model does not change during conversation. Your feedback may be incorporated into the training of a later model version, according to the provider's rules.

Limits and risks

The model learns what the evaluators like, not what is true, so it may be prone to giving confident-sounding or flattering answers.
Evaluators' tastes, cultures, and biases become embedded in the model, so what is an appropriate response in one context may not be in another.
The model may find loopholes: it gives answers that score highly on the reward model but are not actually better.
Human evaluation is expensive and slow, so there is less feedback on rare or specialized topics, making the model less reliable there.

Similar, but not the same

Fine-tuning – Fine-tuning is a broader concept in which the model generally learns from ready-made answer examples, whereas in RLHF it learns from rewards derived from human evaluations.
Reinforcement learning – In classic reinforcement learning, the reward comes from the environment, for example a game score, while in RLHF a reward model is learned from human judgments.
Prompt engineering – Prompt engineering improves how questions are phrased without modifying the model itself, whereas RLHF changes the model's internal settings.

Related concepts

Broader

Reinforcement learning, Machine learning, Fine-tuning, AI alignment

Narrower

Jutalommodell

On the edge

Large language model, pretraining, Data labeling, sycophantic responses, Constitutional AI, preference-based training

A brief history

The basic idea, that the machine learns rewards from human preferences, was made known by a 2017 study. It was already tested on language models in 2019–2020, for example for summarizing texts. It became widely known through OpenAI's 2022 InstructGPT study, and the general public encountered it via ChatGPT, released in November 2022.

Where things stand

As of late 2026, it is an established part of training large chat models, but it is rarely used on its own. It is typically combined with related methods, such as direct preference optimization or procedures where part of the evaluation is done by another AI model.

Status: 2026 November

Also known as

RLHF · reinforcement learning from human feedback · reinforcement learning from human feedback · reinforcement learning from human feedback · reinforcement learning from human feedback · training with human feedback · reinforcement learning from human preferences · RL from human feedback

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.