Loading
AI red-teaming is the structured, adversarial testing of an AI system, by human experts, automated tools, or other AI models, to deliberately provoke failures such as jailbreaks, harmful or biased outputs, training-data leakage, and dangerous capability uplift before those failures reach real users. It extends the adversarial mindset of cybersecurity red-teaming to AI-specific attack surfaces, including prompt injection, jailbreaking, model evasion, and misuse for tasks like cyberattacks or weapons development. Red-teaming matters for governance because it has moved from informal best practice to hard law: the EU AI Act's Article 55 requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing, and national AI safety and security institutes in the US and UK now run their own pre-deployment red-team evaluations of frontier models, while Japan's AI Safety Institute focuses more on evaluation-methodology development and international coordination. For any organisation deploying or procuring AI, red-team results are becoming core audit evidence, proof that failure modes were actively sought out and mitigated, not just assumed away.
Run the free AI Health CheckAI Red Teaming, structured adversarial testing of an AI system, by humans or other AI systems, to identify vulnerabilities, failure modes, harmful outputs, and ways the system can be misused.
AI red teaming extends traditional cybersecurity red teaming to AI-specific risks: jailbreaks, prompt injection, training data extraction, model evasion, and emergent capability surfacing. It is mandated for systemic-risk GPAI under EU AI Act Article 55, and NIST AI RMF subcategory MEASURE 2.7 instructs the use of red-team exercises. Formal, well-documented red-team-style pre-deployment testing programmes exist at the US institute (reorganised in 2025 into NIST's Center for AI Standards and Innovation, CAISI) and the UK institute (renamed the AI Security Institute in 2025, retaining the AISI acronym, with lab access agreements and its Inspect testing platform); Japan's AI Safety Institute, by contrast, publishes comparatively little detail on a dedicated adversarial testing programme, focusing more on evaluation-methodology and standards work.
Source: EU AI Act, Article 55; NIST AI RMF MEASURE 2.7; US NIST CAISI (formerly US AISI)
Manual (human) red-teaming
Security researchers, linguists, or domain specialists hand-craft adversarial prompts and scenarios to surface jailbreaks, harmful content, and biased or unsafe outputs.
Automated / AI-assisted red-teaming
One model or a fuzzing pipeline generates and iterates attack prompts against a target model at scale, surfacing edge cases and failure clusters humans would miss. Anthropic's Ganguli et al. (2022) study on scaling red-teaming with language models is a foundational reference here.
Prompt injection and jailbreak testing
Probing whether instructions hidden in retrieved documents, images, tool outputs, or user turns can override a model's system prompt or safety training.
Data extraction and privacy probing
Attempting to make a model regurgitate memorised training data, personal information, or proprietary content it was never meant to disclose.
Dangerous-capability / uplift testing
Assessing whether a model can materially assist with cyberattacks, or with biological, chemical, nuclear, or radiological weapons development, the specific focus of government AI safety institute programmes.
Agentic and multi-turn testing
Evaluating whether an AI agent with tool or system access can be manipulated across a multi-step task into taking harmful real-world actions, not just producing one bad response.
The clearest legal mandate sits in the EU AI Act. Article 55(1)(a) requires providers of general-purpose AI (GPAI) models designated as posing systemic risk to 'perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks.' The same article also obliges these providers to assess and mitigate systemic risks at EU level (55(1)(b)), report serious incidents to the AI Office and national authorities (55(1)(c)), and ensure an adequate level of cybersecurity protection for the model and its infrastructure (55(1)(d)). These GPAI-model obligations became applicable on 2 August 2025.
The European Commission's voluntary General-Purpose AI Code of Practice, published 10 July 2025, gives providers a concrete way to demonstrate Article 55 compliance: its Safety and Security chapter, signed by Amazon, Anthropic, Google, IBM, Microsoft, Mistral AI, OpenAI and others (xAI signed only this chapter), sets out state-of-the-art adversarial testing and risk-mitigation practices for the small number of providers whose models are deemed systemic-risk.
In the United States, red-teaming is referenced rather than mandated. NIST's AI Risk Management Framework builds it into the MEASURE function: subcategory MEASURE 2.7 (AI system security and resilience) instructs organisations to 'use red-team exercises to actively test the system under adversarial or stress conditions, measure system response, assess failure modes' and to document the results as part of continuous improvement. The AI RMF is voluntary, so this operationalises red-teaming as recommended practice rather than binding law.
Government-run red-teaming has also become institutionalised through national AI safety bodies, though their names have shifted. The US AI Safety Institute was reorganised in 2025 into NIST's Center for AI Standards and Innovation (CAISI), which conducts security evaluations of frontier models focused on cyber, biosecurity, and chemical-weapons risk and has run joint assessments with counterpart institutes. The UK's AI Safety Institute was renamed the AI Security Institute (retaining the AISI acronym) in 2025; it holds pre-deployment testing access agreements with labs including Anthropic, Google, and OpenAI and built the open-source 'Inspect' evaluation platform. Japan's AI Safety Institute (AISI), established under Japan's IPA, focuses on developing AI safety evaluation methodologies and coordinating internationally, though it publishes less public detail on a dedicated red-team testing programme than its US and UK counterparts.
Traditional penetration testing targets deterministic software: a fixed set of code paths, known vulnerability classes, and reproducible exploits. AI red-teaming targets a probabilistic, natural-language system whose behaviour can vary between runs, whose 'vulnerabilities' include things that were never bugs in a coding sense, a model that is too persuasive, too compliant with a harmful request, or too willing to reveal its training data. This means AI red-teaming needs its own taxonomies of harm (jailbreaks, hallucinated legal or medical claims, biased outputs, unsafe agentic actions) and cannot simply reuse a cybersecurity vulnerability scanner. It typically sits alongside, not instead of, conventional application and infrastructure security testing for the systems an AI model is embedded in.
Independence
Testers who did not build the model or feature are more likely to find blind spots the development team has become numb to.
Documented scope and methodology
Article 55 and the NIST AI RMF both point to the same governance need: a written record of what was tested, how, and against which standardised protocol, not just a pass/fail claim.
Coverage matched to risk
High-stakes deployments (healthcare, finance, critical infrastructure, frontier general-purpose models) warrant deeper coverage of dangerous-capability and agentic testing than a low-risk internal tool.
Retesting triggers
Red-teaming is not a one-off gate: material model updates, new tool access, new deployment contexts, or newly disclosed attack techniques should trigger a fresh round.
A remediation and escalation path
Findings need an owner and a fix-or-accept decision, mirrored in the incident-reporting obligations that Article 55 attaches to systemic-risk GPAI providers.
What is AI red-teaming?
AI red-teaming is structured adversarial testing of an AI system, by human testers, automated tools, or other AI models, to deliberately find failure modes such as jailbreaks, harmful outputs, data leakage, bias, or dangerous capability uplift before the system is deployed or while it is in production.
Is AI red-teaming legally required?
Yes, in a specific case: the EU AI Act's Article 55 requires providers of general-purpose AI models designated as posing systemic risk to conduct and document adversarial testing, effective from 2 August 2025. Outside that category, red-teaming is generally a recommended practice (e.g. under the US NIST AI RMF) or a condition attached to government pre-deployment testing agreements, rather than a universal legal mandate on all AI systems.
How is AI red-teaming different from traditional penetration testing?
Penetration testing targets deterministic software vulnerabilities with reproducible exploits. AI red-teaming targets a probabilistic, natural-language system where the 'vulnerability' can be a model that is too persuasive, too compliant with harmful requests, or prone to leaking memorised training data, failure modes that don't map onto conventional vulnerability classes.
Which government bodies run official AI red-teaming programmes?
In the US, NIST's Center for AI Standards and Innovation (CAISI, the successor to the US AI Safety Institute) runs security evaluations of frontier models. In the UK, the AI Security Institute (formerly the AI Safety Institute, still using the acronym AISI) holds pre-deployment testing agreements with major labs and built the open-source Inspect evaluation platform. Japan's AI Safety Institute (AISI) focuses on evaluation methodology development and international coordination.
Can AI red-teaming be automated?
Yes. Alongside manual, human-led red-teaming, organisations increasingly use one AI model or an automated fuzzing pipeline to generate and iterate adversarial prompts against a target model at scale, an approach documented in Anthropic's 2022 research on scaling language-model red-teaming. Automated methods find volume and edge cases; human red-teamers bring domain judgment and creativity that automation still struggles to replicate fully.
Does passing a red-team exercise mean an AI system is safe?
No. Red-teaming is one input into a broader risk-management process, it demonstrates that specific attack scenarios were tested and, ideally, mitigated, but it cannot prove the absence of undiscovered failure modes. Frameworks like the EU AI Act and NIST AI RMF treat it as one required or recommended control alongside risk assessment, incident reporting, and ongoing monitoring.
Last reviewed July 2026