model extraction

advancedspreading

An attack in which someone asks a massive number of questions to an AI model and builds a replica from the answers at a fraction of the original development cost.

In plain terms

Imagine you want to get hold of a famous restaurant's secret recipe but cannot get into the kitchen. So over months you order the entire menu, taste everything, take notes, and at home experiment until your dish tastes the same. You did not see the recipe, yet you copied its essence. Model extraction is the same, just with an AI model.

Training large AI models costs vast amounts of money, data, and computing power, so companies keep the model's internals as trade secrets. You can only ask questions; you cannot see the 'machinery' behind the answers. Attackers exploit this query capability: they train their own copy using thousands or millions of question-answer pairs.

This is not a breach in the traditional sense. The attacker does not break into a server or steal files, but uses a public service for a purpose it was not intended for. This makes it difficult to detect and legally address.

How it works

The attacker gains access to the target model, typically via subscription or through a programming interface (API), often using many fake accounts to avoid detection.
It automatically submits a mass of carefully designed questions to cover as much of the model's knowledge and behavior as possible.
It collects the answers and creates training material from them: for every question, the "correct solution" given by the original model is there.
A custom, usually smaller model is trained on this material and learns to imitate the original. This training method is called distillation.
The created copy is used or sold as its own product, or used as a training ground to launch further attacks against the original model.

An everyday example

A Hungarian fintech company has been developing a credit assessment model for years, making it available to customers as an online calculator. A competitor sends hundreds of thousands of fake customer data points via an automated program, records the decisions, and builds its own model that makes decisions almost identically. As a customer, you only notice that the service begins to limit the number of daily queries.

Who is affected

entrepreneur, executiveIf your company offers its own model to customers, the invested money and knowledge may inadvertently end up with competitors.

developerYou must build in the protection: query limits, monitoring for suspicious usage patterns, and restricting the detail level of responses. → Code

lawyer, citizenIt operates on the boundary between trade secrets, copyright, and terms of use, and there is still no settled case law on many issues. → Law

everyoneCopied models often lack the original safety constraints, making them easier to exploit for fraud or abuse.

A common misconception

MythA model can only be stolen if someone hacks the company's servers and downloads its files.

In realityModel extraction requires only standard public access: the attacker simply asks questions and builds a copy from the answers. The resulting model is typically not a bit-perfect replica but a behaviorally similar imitation; exact copies have only succeeded with simple models or specific parts of models.

Limits and risks

There is no perfect protection: as long as a model answers questions, its answers can be learned from; defenses only make the attack more expensive and slower.
The extracted copy is almost always weaker than the original, especially in rare, complex tasks.
It is difficult to prove that a model is a copy, as similar behavior between two systems is not evidence in itself.
Security measures also affect legitimate users: query limits and restricted responses may cause inconvenience to you as well.

Similar, but not the same

knowledge distillation – Distillation is an established, legal technique when someone creates a smaller model from their own or licensed large model; model extraction occurs when it is done on another's model without permission.
training data extraction – There, the attacker aims to retrieve original training data (such as personal information) 'burned' into the model, not to copy the model's capabilities.
model weight theft – That is classic data theft, where the model file itself is taken through a break-in or an internal leak, rather than a replica being built by querying the model.

Related concepts

Broader

AI safety, adversarial machine learning, cybersecurity

Narrower

distillation attack

On the edge

knowledge distillation, training data extraction, system prompt extraction, membership inference attack, prompt injection, watermarking

A brief history

The phenomenon was popularized in 2016 by Florian Tramèr and co-authors, who demonstrated that models from cloud machine learning services could be copied solely through queries. In 2024, Nicholas Carlini and colleagues showed that a single layer (the last one) of live, commercial language models can be precisely calculated from API queries alone. In 2025–2026, several major AI companies, including OpenAI and Anthropic, publicly reported large-scale distillation attempts against their models.

Where things stand

By the end of 2026, mass distillation attempts against models had moved from research labs into everyday business and geopolitical debates: major AI companies publicly report on them. Defensive measures (account filtering, pattern monitoring, query limits) have become standard equipment for providers, but there is no reassuring solution.

Status: 2026 November

Also known as

model extraction · model theft · model extraction · model cloning · model extraction · model stealing · model theft · distillation attack

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.