red teaming

advancedwidespread

A test in which experts deliberately act as attackers to identify weak points and dangerous behaviors of an AI system before actual deployment.

In plain terms

Imagine a bank asks a team of experts to play the bank robber, try to get into the vault, and then tell them where a window was left open. Red teaming is the same thing in the AI world. The team does not examine whether the system works well, but how it can be pushed into working badly.

Members of a 'red team' try trick questions, misleading instructions, and hidden traps. They want to know if the model gives dangerous advice, reveals personal data, or violates its own rules. What they find, developers fix before a real malicious actor discovers it.

Traditional testing checks whether the system does what it is supposed to do under normal use. Red teaming looks for what happens if someone intentionally tries to abuse it. This is particularly important for agents, because an agent not only speaks but also acts: it sends emails, deletes files, and makes purchases.

How it works

They define the goal and scope: what is being tested (a model, a chatbot, or an agent) and what harms are being sought, such as dangerous content, data leakage, or unauthorized operations.
The team gathers attack ideas: requests wrapped in role-play, instructions designed to bypass rules, and commands hidden in documents or web pages.
They try these out one by one, partly manually, partly automatically. Often, another AI model generates thousands of attacking questions.
Every successful attack is recorded: what was the input, what did the system do, and how severe were the consequences.
Developers fix the bugs and then repeat the test, because a fix can open a new vulnerability, and every new model version behaves differently.

An everyday example

A bank introduces a customer service chatbot. Before launch, a team spends days trying to get it to reveal other customers' balances or give misleading credit advice. You are already using the version where such found bugs have been fixed.

Who is affected

developerIf you build an AI-based product or agent, red teaming is part of the pre-release check, not a post-hoc add-on. → Code

entrepreneur, executiveIf you are introducing an AI system at your company, ask the vendor whether they have conducted such tests and what they found.

lawyer, citizenSince August 2025, the EU AI Act has required providers of general-purpose models with systemic risk to carry out adversarial testing and document it. → Law

everyoneThe security of the chatbots you use depends partly on how thoroughly people tried to break them in advance.

A common misconception

MythIf an AI system passes red teaming, it is secure.

In realityThe test only proves which of the tried attacks succeeded and which did not. It cannot confirm that there are no more vulnerabilities, because attackers continuously develop new methods.

Limits and risks

Never complete: the number of possible inputs is practically infinite, and the team only finds what they thought of.
The result depends heavily on the testers' knowledge, background, and language. A model tested mainly in English may still surprise users in Hungarian.
There is no uniform standard, so the label 'red-teamed' may mean different things across companies and can be used as a marketing tactic.
It provides a snapshot: after integrating a new model version, tool, or data source, previous results may no longer be valid, so the test must be repeated.

Similar, but not the same

penetration test – A penetration test typically seeks technical vulnerabilities in the network and software, while AI red teaming targets the model's behavior, meaning what the system says and does.
jailbreak – A jailbreak is a specific trick to bypass the model's limits, while red teaming is the organized, authorized work that searches for such tricks, among other things.
evaluation – Evaluation measures performance on a predefined set of tasks, while red teaming is creative, open-ended debugging with an adversary's mindset.

Related concepts

Broader

AI safety, cybersecurity, risk management

Narrower

automated red teaming, Prompt injection testing

On the edge

jailbreak, prompt injection, guardrail, evaluation, penetration test

A brief history

The term originates from Cold War-era American military and strategic wargames, where the 'red team' played the opponent. Cybersecurity adopted it, followed by AI development. In August 2023, at the DEF CON conference in Las Vegas, about two thousand participants publicly tested large language models from several companies, with support from the US government.

Where things stand

By the end of 2026, red teaming before release is standard practice at major model developers. The EU AI Act makes it mandatory for general-purpose models posing systemic risks. The focus is increasingly shifting to testing autonomous agents and automated methods.

Status: 2026 November

In the news

Appears in 2 Agent Age stories; the curve shows weekly occurrence in the news magazine.

METR red team broke OpenAI's agent monitoring in three waysOct 26, 2026 · Code
US cybersecurity job postings fell 26.4% while AI roles growOct 19, 2026 · Work & HR

Also known as

red teaming · red-teaming · AI red teaming · red team · red team · red team test · adversarial testing · adversarial testing

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.