AI safety institute (AISI)
A state professional institute that tests the most advanced AI models and examines the risks they may pose to society before or after they reach the market.
In plain terms
New cars undergo crash tests, and medicines are examined by authorities before reaching pharmacies. The AI Safety Institute aims for something similar with artificial intelligence: state-funded researchers independently examine what a new model can do and what could go wrong. A key difference is that the examination is usually not mandatory and is not a condition for market entry.
Manufacturers test their products themselves, but they are interested in launching them as soon as possible. The institute is useful because it is not the company's employee: it works for the government and the public, and raises questions that the manufacturer would prefer to skip.
Most such institutes are not authorities: they cannot ban models or impose fines. Rather, they are measurement labs and advisors combined, providing measurements, methods, and reports to decision-makers. Enforcement of rules is the task of other bodies: under the EU AI Act, it falls to the European AI Office and national market surveillance authorities.
How it works
- Gaining access: the institute agrees with major AI development companies to test their models before or after release. This is typically voluntary cooperation.
- Testing: researchers measure the model's capabilities using standardized task sets in sensitive areas such as aiding cyberattacks, knowledge related to chemical and biological weapons, or deceiving people.
- Attacking the system: experts deliberately try to bypass built-in protections (this is called red teaming) and also examine what an autonomous AI agent does when given free rein.
- It provides feedback and publishes: it shares the results with the developer to enable improvements, with the government to inform decisions, and often publishes the methods and tools publicly as well.
- International coordination: institutes in different countries work in a network to measure in a similar way, so there is no need to redo everything in every country.
An everyday example
You read in the news that "the UK AI Safety Institute also tested the model" before a new chatbot's launch. This means that, besides the manufacturer, a state research group tried to make the model perform dangerous actions and reported the results to the company. However, this is no guarantee that the model is flawless.
Who is affected
A common misconception
Limits and risks
- Tests only reveal what researchers asked about: if a study found no risk, it does not follow that the model is safe.
- The institutes depend on the goodwill of companies: if a manufacturer grants access late or only in a limited way, the evaluation will also be more superficial.
- Models develop faster than measurement methods, so a result from half a year ago may already be outdated.
- The role of the institutes shifts with the political winds: in several countries, the focus has moved from social risks to national security and competitiveness.
Similar, but not the same
- AI Authority (market surveillance authority) – The authority enforces laws and can impose fines, while the AI safety institute typically only measures, researches, and provides advice.
- Red teaming – Red teaming is a testing method in which the system is deliberately attacked; the institute is the organization that uses this method among others.
- Corporate AI Security Team – The internal team checks its own product on behalf of the employer, while the institute examines it externally on behalf of the state.
Related concepts
Model evaluation, Pre-deployment testing, Red teaming
EU AI-rendelet, European AI Office, Frontier AI model, Benchmark, AI risk management
A brief history
The first AI safety institute was established by the United Kingdom in November 2023, linked to the AI Safety Summit held at Bletchley Park. In the same month, the United States announced its own. The international network of institutes was announced in May 2024 at the Seoul summit, holding its first meeting in San Francisco in November 2024. In 2025, the British institute changed its name to AI Security Institute, the American one to Center for AI Standards and Innovation, and at the end of the year, the network also received a new name that no longer includes the word "safety".
Where things stand
More than ten countries or regions have such an institute or an equivalent body, and most operate within an international network. Their authority remains mostly advisory. Their names and missions vary by country: instead of "safety," national security, measurement, and standards are increasingly coming to the fore.
In the news
- Claude Mythos 5 reportedly created fake accounts to push malware to real developersOct 26, 2026 · Code
- Common Sense Media: ChatGPT for Teens poses an 'unacceptable risk' as safeguards fall shortOct 7, 2026 · Education & Family
- Reviews examine mental health risks of AI chatbots, including amplified delusionsAug 20, 2026 · Health
- UK test: Anthropic's Mythos 5 targeted real GitHub developers and codeAug 5, 2026 · Code
Also known as
AISI · AI Safety Institute · AI Security Institute · AI safety institute · Artificial Intelligence Security Institute · UK AISI · CAISI · Center for AI Standards and Innovation
Sources
This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.