Distillation (KD)

advancedwidespread

Distillation is a process where the knowledge of a large, expensive AI model is taught to a smaller, faster, and cheaper model.

In plain terms

Imagine an experienced professor not handing over their entire library to a student, but their own refined notes and how they think about problems. The student knows less than the master, but grasps the essentials much faster than if they had to learn everything from scratch. Distillation does the same with AI models: the small "student" model learns from the responses of the large "teacher" model.

It is good because running the largest language models is slow and expensive and requires a huge server farm. The distilled model delivers most of the teacher's performance while requiring much less computing capacity. The smallest variants can even run on a phone or laptop.

It differs from traditional training: the student learns not only from raw data but also imitates the teacher's behavior. If the teacher's internal numbers are accessible, they also show how confident the teacher is in an answer and which other answer would have been almost as good. This extra signal helps the small model a great deal. If only the teacher's finished text answers are available, the student learns from them as worked examples.

How it works

At the start, there is a large, already-trained model (the teacher) and a much smaller, untrained or partially trained model (the student).
The teacher is given numerous questions and tasks, and its answers are recorded. If the probabilities are accessible, the probability it assigned to each possible answer is saved as well.
The student is taught to give answers as similar as possible to the teacher's to the same questions. If the probabilities are also present, it learns not only the correct solution but also the teacher's "uncertainties".
Finished student models are compared with the teacher model on tests: how much quality was lost and how much was gained in speed and cost.
If the result is acceptable, the small model is deployed where the large one would be too expensive or too slow.

An everyday example

When your phone translates a sentence or summarizes a message without an internet connection, a distilled (and often otherwise compressed) model is likely at work. The same applies to the "mini", "lite", or "flash" versions of chat applications: these are often partially built on the knowledge of their larger counterparts, though manufacturers do not always disclose this.

Who is affected

everyoneThe responses on your phone and in fast, free AI services often come from distilled models, explaining why they are faster but sometimes weaker than large models.

entrepreneur, executiveWith a distilled model, the same task can often be performed at a fraction of the cost, so it is worth checking whether the largest model is truly necessary.

developerThis allows building a fast model tailored to a narrow task, running on its own machine, with multiple instances working in parallel, for example on subtasks of a coding agent. → Code

lawyer, citizenIt is a disputed legal question whether you may train your own model on another company's model's responses without permission. → Law

A common misconception

MythA distilled model can do the same as the original but takes up less space.

In realityThe small model absorbs only part of the teacher's knowledge: it can be nearly identical in common, typical tasks, but is typically weaker in rare, complex, multi-step questions.

Limits and risks

The student model inherits the teacher's errors, mistakes, and biases, and sometimes even amplifies them.
Quality loss is not uniform: it is barely visible on average tests but can suddenly appear with unusual or professional questions.
Most AI providers prohibit in their terms of service that competitor models be trained from their model's responses, so unauthorized distillation carries legal risks.
The product name rarely reveals whether you are using a distilled model, so for important tasks it is worth testing the larger variant as well.

Similar, but not the same

quantization – Quantization compresses the same model by reducing number precision, while distillation trains a new, smaller model.
fine-tuning – Fine-tuning trains an existing model further on new data for a specific task, while distillation makes a smaller model copy the behavior of another model.
pruning – Pruning cuts less important connections from an existing model; in distillation, a small model learns separately from a large one.

Related concepts

Broader

model compression, machine learning, model training

Narrower

teacher–student training, self-distillation

On the edge

quantization, fine-tuning, small language model, synthetic data, large language model

A brief history

The basic idea of model compression was described by Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil in 2006. The term "distillation" and the method known today were popularized by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in their 2015 study.

Where things stand

By the end of 2026, distillation is established industry practice: a significant share of the smaller, cheaper model variants from major AI companies and of the models running on devices are made this way. Meanwhile, a legal and business debate continues over whether it is permissible to distill other companies' models without permission.

Status: 2026 November

Also known as

Distillation · knowledge distillation · model distillation · knowledge distillation · distillation · knowledge distillation · model distillation · KD

Sources

Hinton, Vinyals, Dean: Distilling the Knowledge in a Neural Network (2015, arXiv)

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.