security classifier
in English: safety classifier (constitutional classifiers)
A safety classifier is a separate AI model that monitors questions and answers, attempting to block harmful content before it reaches you.
In plain terms
Imagine airport security screening. Even though most passengers are trustworthy, everyone and every bag is screened. The security classifier does the same in an AI system, but at two points: it checks what goes in (your question) and what comes out (the answer). This is a separate, usually smaller model that does not answer itself but decides whether a text can pass.
This is necessary because large language models can be talked into doing things they would otherwise refuse, using clever tricks. Such tricks include role-playing, encoded requests, or long, misleading stories. The classifier acts as a second line of defense: even if the main model is fooled, the gatekeeper can still catch the dangerous request or response.
The 'constitutional classifier' is one kind of these, developed by Anthropic. Here, the rules are written in plain language in a 'constitution': what is allowed and what is forbidden. From this description, a great many training examples are generated automatically, and the classifier is trained on them. If a new risk emerges, it is enough to expand the rule text and retrain.
How it works
- Developers describe in natural language what content is prohibited (for example, steps to manufacture weapons of mass destruction) and what similar but harmless content is allowed (for example, school chemistry).
- A language model generates thousands of artificial examples from this rulebook: forbidden and allowed requests, rewritten in multiple languages, styles, and with known bypass tricks.
- An input classifier that examines your question and an output classifier that monitors the answer as it is being generated are trained on these examples.
- During use, the output classifier scores the response in real time, almost word by word. If the risk exceeds a threshold, the system interrupts the response.
- The defenses are also continuously attacked on purpose (this is called red teaming), and the gaps found are closed by updating the rules and the training examples.
An everyday example
You ask a chatbot about safe storage of household chemicals and receive an answer without any problem. However, if someone asks, in the same place, for instructions for making nerve gas wrapped in a "novel scene," the response stalls or a refusal comes back. Often, it is not the chatbot model itself that says no, but a background safety classifier that stops it.
Who is affected
A common misconception
Limits and risks
- It errs in both directions: sometimes it rejects harmless questions (for example, on medical or chemical topics), and sometimes it allows dangerous content through.
- It only filters what it was trained on, meaning a classifier tailored to a narrow area does not protect against all other types of harmful or false content.
- Every request requires additional computation, which increases the cost of the service and can slow down the response.
- What counts as prohibited is decided by the provider, and the details of the rules are often not visible from the outside.
Similar, but not the same
- content moderation – Moderation is the entire process involving people, rules, and complaint handling; the safety classifier is only one automated tool within it.
- alignment – Alignment trains the model itself to behave correctly, while the classifier checks from the outside, regardless of what the model does.
- system prompt – The system prompt is an instruction to the model itself, which it may ignore or which can be bypassed with tricks. The classifier, however, is a separate model that decides independently of the main model, although it can also err.
Related concepts
AI safety, guardrail, content moderation, classification
constitutional classifier, input filter, output filter
jailbreak, prompt injection, red teaming, alignment, large language model
A brief history
Classifiers that detect harmful content were used in social media moderation even before large language models. Anthropic researchers introduced constitutional classifiers in a study in early 2025. For over three thousand hours, testers deliberately tried to break (jailbreak) the prototype, and in that time they found no universal trick that worked on every prohibited question. In a later public test, some participants still managed to get past the protection, showing that the method is strong but not unbreakable. Since spring 2025, Anthropic has also used such classifiers in its live products against the most severe risks, mainly those related to chemical and biological weapons.
Where things stand
At the end of 2026, safety filters placed before and after the model are standard in the products of the major AI providers, and freely downloadable filter models are also available (for example, Llama Guard). The constitutional approach, trained from a written policy, runs in production in a narrower range of uses, mainly against the most severe risks.
In the news
- Claude Code vulnerabilities: CI/CD leak, auto mode bypassed, AWS credential theftSep 7, 2026 · Code
- OpenAI launches ChatGPT for Teens with study features, parental controls, for ages 13–17Aug 24, 2026 · Education & Family
Also known as
security classifier · security classifier · constitutional classifier · safety classifier · constitutional classifiers · guard model · content filter · moderation model
Sources
- Sharma et al.: Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming (arXiv, 2025)
- Anthropic: Constitutional Classifiers – research summary
This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.