security assessment
in English: safety evaluation
A safety evaluation is a series of tests used before and after release to measure whether an AI system can cause harm and how easily it can be induced to behave dangerously.
In plain terms
Before a new car hits the road, it goes through a crash test: it is deliberately driven into a wall to reveal what happens when something goes wrong. Safety evaluation is the same in the world of AI. Here, the question is what the worst thing is that the model could cause, and how likely that is to happen. How skilled the model is at everyday tasks is measured by other tests.
Testers feed the system thousands of sensitive requests. They check whether it provides dangerous instructions, can be tricked by clever phrasing, misleads users, or leaks personal data. With autonomous agents, they also monitor what the system does when accessing tools, files, or money—for example, whether it deletes something it should not or exceeds its granted authorization.
In traditional software testing, a specific input typically yields the same output. A language model can answer the same question in multiple ways, and the number of possible questions is practically infinite. Therefore, evaluation can only sample behavior: it reduces risk but cannot guarantee full safety.
How it works
- Risk identification: developers or external assessors decide what harms to look for, such as dangerous advice, involvement in fraud, deception, data leakage, or uncontrolled autonomous action.
- Compiling test tasks: questionnaires and realistic scenarios are prepared for each risk. For agents, this is an isolated test environment where the system can use tools without causing real damage.
- Run and attack: the model is tested automatically, often repeatedly, while human experts and other AI systems deliberately try to bypass its defenses.
- Scoring: answers are rated by rules, judging models, and humans, counting how often the system erred and how severe the errors were.
- Decision and repetition: results are measured against preset thresholds. If the model crosses them, release is delayed or additional protections are added. Tests are run again with every new version.
An everyday example
A new chatbot is released, and its developer publishes a long document on how the model was tested. In it, you can read how many times the model was induced to produce prohibited content and what it did when it had to make an independent decision in a test scenario. This document is the public summary of the safety evaluation.
Who is affected
A common misconception
Limits and risks
- Tests only measure the risks that were anticipated in advance, so new kinds of abuse and unexpected situations are easily missed.
- Many evaluations are conducted and published by the developer company itself, so check whether an independent external audit was also performed.
- Research suggests advanced models sometimes realize they are being tested and may behave differently than in live use.
- Results from different companies are hard to compare because there is no single, universally accepted benchmark.
Similar, but not the same
- Performance measurement (benchmark) – Performance measurement examines how capable a model is at a task, while a safety evaluation determines how much harm it could cause.
- red teaming – Red teaming is one evaluation method where humans or machines intentionally attack the system. Security assessment is broader, including standardized measurements.
- compliance audit – An audit checks whether the organization complies with rules and documentation obligations, while an evaluation measures the model's own behavior.
Related concepts
AI safety, risk management, AI governance
red teaming, capability assessment, jailbreak-teszt, agent evaluation
A brief history
Testing machine learning models is an old practice, but safety-focused evaluation became a distinct field with the spread of large language models. In January 2023, the US NIST AI Risk Management Framework was published. In November 2023, during the Bletchley Park summit, the first state AI safety institute was announced in the United Kingdom; it has been called the AI Security Institute since February 2025.
Where things stand
At the end of 2026, major developers regularly conduct, and partly publish, safety evaluations before release. Since August 2025, the EU AI Act has made model evaluation mandatory for general-purpose models that pose systemic risk. There is still no unified methodology or comprehensive independent oversight, and the evaluation of agents is particularly immature.
In the news
- Sam Altman pitches OpenAI's AI tools to utilities for grid defenceSep 10, 2026 · Ipar
- Security vendors build defences around coding agents: AWS Continuum, Wiz, SailPoint, Menlo, Snyk, MicrosoftAug 3, 2026 · Code
Also known as
security assessment · security assessment · AI safety assessment · security testing · safety evaluation · safety eval · safety evals · AI safety evaluation
Sources
This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.