PPO

in English: Proximal Policy Optimization

professionalwidespread

A training method in which AI learns from attempts based on rewards, but can only change its behavior slightly at a time.

In plain terms

PPO is one of the best-known methods in reinforcement learning. In this form of learning, the machine does not learn from ready-made examples but tries, receiving rewards for good steps and penalties for bad ones.

Imagine you are learning to play tennis. If you change your entire motion after a successful hit, you may easily ruin what was working before. A good coach therefore says: change only a little at a time. PPO builds this rule into machine learning: each step allows only a small change compared to previous behavior.

This makes learning more predictable and less prone to collapse. Compared to earlier, similarly reliable methods, it is also simpler to implement, which is why it has become the default choice for many researchers and developers in games and robotics. It played an important role in the fine-tuning of conversational language models based on human feedback, although newer, related methods are now mostly used there.

How it works

The agent often plays through the task with its current strategy (in English, policy, meaning the rule set that dictates when to take which step) and records what it did and how much reward it received.
The system calculates which steps succeeded better than expected and which performed worse.
It modifies the strategy to make better moves more likely and worse ones rarer.
A built-in limit (called "clipping") restrains excessive changes: the agent gains nothing if the new strategy deviates too far from the old one, no matter how tempting a big leap may be.
The agent gathers new experience with the updated strategy, and the cycle starts over. This can repeat thousands of times, accumulating millions of steps of experience, or even billions for large tasks.

An everyday example

When you see a video of a four-legged robot staying upright on ice or stairs, its movement was likely trained in simulation using PPO. The OpenAI Five program used the same method to defeat the world champion Dota 2 video game team in 2019.

Who is affected

developerIf you are working with reinforcement learning, this is often the first method worth trying, because it is readily available in most toolkits. → Code

industrial worker, engineerThe movement of robots and control systems trained in simulations is often based on this procedure. → Industry

everyoneIt also played a key role in fine-tuning chat AI models based on human feedback, indirectly shaping your everyday tools.

A common misconception

MythPPO is a standalone artificial intelligence that learns anything on its own.

In realityPPO is only the training recipe, not the model itself. The result depends on what task, reward, and practice environment people provide.

Limits and risks

It requires many attempts, so models are mostly trained with it in simulation, because in the real world this would be slow, expensive or dangerous.
The result depends heavily on the settings and minor implementation details; the same method can perform very differently for two teams.
If the reward is poorly defined, the agent finds a loophole: it learns what earns points, not what was actually intended.
Behavior learned in simulation does not always work as well in the real world.

Similar, but not the same

reinforcement learning – Reinforcement learning is the name of the entire field; PPO is just one specific method within it.
Q-learning – Q-learning estimates what each step is worth, while PPO directly improves the behavior policy.
fine-tuning – Fine-tuning refers to further training a pre-trained model by any method, while PPO is only one of its possible tools.

Related concepts

Broader

reinforcement learning, machine learning, deep learning

Narrower

–

On the edge

reward function, RLHF, Q-learning, fine-tuning, simulation

A brief history

The method was published in 2017 by OpenAI researchers led by John Schulman. Intended as a simpler version of an earlier, reliable but complex procedure (TRPO), it became one of the field's fundamental methods within a few years.

Where things stand

As of late 2026, PPO is one of the default methods in reinforcement learning, especially in robotics and simulation-based training. In training language models, several newer, related methods are also used alongside it.

Status: 2026 November

Also known as

PPO · Proximal Policy Optimization · proximal policy optimization · proximal policy optimization · proximal policy optimization · PPO-algoritmus · PPO algorithm · PPO-Clip

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.