An artificial intelligence system can pass every planned test and fail the moment it meets someone willing to misuse it. This is why adversarial testing exists: to deliberately expose an AI model or application to manipulative, ambiguous, extreme or malicious requests, in order to identify vulnerabilities, unexpected behaviour and potential for abuse before they emerge in real-world use.
Ordinary acceptance testing checks that the system performs the task it has been given. Adversarial testing instead tries to push it beyond its limits: to circumvent policies, obtain confidential information, provoke hallucinations. From the recent case of Google Earth and Nano Banana 2 to the various scandals that have surrounded Grok, this practice is proving increasingly central to the Governance of artificial intelligence, and the leading AI agencies are placing it at the heart of their operations because, when it is missing, it shows.
The test can focus on the model, but its most useful scope covers the entire application: system prompts, documents retrieved through RAG, databases, APIs, connectors, permissions, agents and business procedures. An apparently secure model can become fragile when it is authorised to access an email inbox, a CRM or a tool capable of taking actions. This is why adversarial testing belongs to AI Governance.
And because models, prompts, data and integrations change, testing does not end with a single session: it becomes a periodic, documented process linked to the organisation’s decisions.
Let’s explore it together.
What is adversarial testing?
The term derives from the logic of the adversary. The tester adopts the perspective of a hostile user, an attacker or simply someone who uses the system in unpredictable ways.
Google describes adversarial testing as a systematic evaluation designed to observe how a model reacts to harmful or potentially dangerous inputs. NIST defines AI red teaming as a structured effort to find flaws and vulnerabilities, often in a controlled environment and in collaboration with those who developed the system.
“Adversarial testing” refers to the general category. “AI red teaming”, by contrast, describes a more structured activity, carried out by an internal or external team that builds realistic scenarios, explores different attack surfaces and documents the results. The two terms are frequently used interchangeably, although red teaming tends to involve a more formalised mandate, scope, objectives and success criteria.
The quality of the work depends on the variety of the tests. Explicitly malicious questions are only part of the problem. Some inputs look harmless but contain biased assumptions, culturally sensitive references or instructions capable of producing a dangerous result after several steps. It is precisely in these grey areas that standard benchmarks often leave room for unexpected behaviour.
The difference between acceptance testing, penetration testing and AI red teaming
Each check examines a different dimension. Functional testing verifies that the application responds correctly to its intended task. A traditional penetration test looks for vulnerabilities in infrastructure, code, configurations and access controls. Adversarial testing applied to AI also examines behaviour: it tries to manipulate the meaning of instructions, the model’s priorities and the relationship between language, data and actions.
The distinction becomes clear in generative systems. A chatbot may run on secure infrastructure and still disclose information through the prompt. An agent may respect its technical permissions and still treat an instruction hidden in a document as legitimate. An assistant may refuse a direct request and accept the same request when it is reframed as a simulation, a translation or a chain of small steps.
OWASP includes within the scope of contemporary red teaming risks such as prompt injection, model misuse, agent privilege escalation, data poisoning, hallucinations and emergent behaviours. Technical verification remains essential, but on its own it does not cover an attack surface that also runs through language and context.
What risks can emerge
The best-known case is prompt injection. The user enters an instruction that attempts to change the system’s priorities or override the rules set in the initial prompt. In jailbreaks, the manipulation aims to bypass filters and obtain content the model should refuse. In systems connected to data and tools, the risk grows: the attack may target the extraction of information, the modification of records, the sending of messages or the execution of unauthorised actions.
There are also less spectacular but equally significant failures. A model can amplify cultural bias, apply different criteria to different groups or produce responses that are formally credible and factually invented. In our article on AI hallucinations we show how even linguistically confident output can contain false information. Adversarial testing builds questions and contexts specifically designed to measure that fragility, rather than waiting for the error to reach a customer, an employee or a public communication.
The assessment can also cover the system’s reputational behaviour. An assistant that adopts a tone incompatible with the brand, makes unauthorised promises or responds in a discriminatory way creates a risk that involves security, identity and trust at once.
How an adversarial test is carried out
The work starts from the context of use. Before designing attacks, you need to know what the system is supposed to do, who can use it, what data it processes, which tools it can reach and what consequences an error would have. The same response may be acceptable in a creative assistant and unacceptable in a system that supports legal, financial or healthcare decisions.
The scope is then translated into scenarios. Policies define what the system must refuse; failure modes describe the behaviours to be guarded against; edge cases take the verification beyond ideal conditions. Testers build inputs that vary in language, tone, complexity and sequence, alternating manual and automated tests. Every useful result must be reproducible: the attack, the path followed, the output obtained and the technical conditions are all recorded.
The next phase concerns classification. An unpleasant response carries a different weight from a path that allows data to be extracted or a tool to be triggered. Severity, likelihood, impact and ease of reproduction help set priorities. The test then leads to mitigation: changes to prompts, restricted permissions, filters, data separation, server-side controls, human review or a redefinition of the use case.
After the fix, regression testing is required. The correction must close the vulnerability without compromising legitimate functions. The cycle continues with monitoring, because new attack techniques and new behaviours emerge while the system is in use.
Why you need to test the system, not just the model
Companies rarely use a model in isolation. They connect it to documents, search engines, internal archives, CRM, email, productivity tools and operational applications. Each integration adds capability and introduces a new dependency.
NIST has observed this problem in its evaluations of agents. Malicious instructions can be planted indirectly in websites, emails or repositories that the AI consults while it works. In March 2026, a competition analysed by the Center for AI Standards and Innovation gathered more than 250,000 attempts against thirteen frontier models: at least one successful attack was found for every model evaluated. The result does not prove that every agent is inevitably vulnerable. Rather, it shows how necessary it is to keep updating test scenarios as capabilities, tools and defences change.
The Fincantieri website offers a useful example of this evolution. At the centre of its digital ecosystem is captAIn, the Group’s proprietary AI agent. The presence of an agent transforms navigation and, at the same time, widens the perimeter to be governed.
The relationship with the AI Act
Adversarial testing has also taken on regulatory significance. The AI Act requires providers of general-purpose models with systemic risk to carry out evaluations in line with standardised protocols and state-of-the-art tools, including documented adversarial testing. The European Commission links these activities to the assessment and mitigation of systemic risks, cybersecurity and incident management.
The obligation applies to a specific category of provider and does not automatically extend to every company that uses a generative assistant. The distinction matters. For organisations adopting AI systems, testing can nonetheless become a measure proportionate to risk, a procurement requirement, a check before deployment to production and evidence of the diligence exercised.
Regulation points in a broader direction. Reliability, safety and oversight must be demonstrated through evidence. A policy describes the expected behaviour; adversarial testing verifies how well that behaviour holds when someone deliberately tries to break it.
From technical report to Governance decision
The value of the test depends on what happens next. A list of successful or failed prompts interests specialists, whereas management needs to understand which risks remain open and which decisions they call for.
The report should link each piece of evidence to the use case, the potential impact, the person responsible for mitigation and the date of the next verification. Some findings lead to a technical fix. Others require limiting the system, introducing a human step or giving up a function. Governance sets the threshold beyond which a risk becomes incompatible with the intended use.
This is where the link with Human Oversight comes in. Human control becomes meaningful when the person knows the failure modes, has the authority to intervene and receives understandable signals. Adversarial testing helps define precisely those signals: it shows where the system can fail, which outputs deserve attention and when an escalation should be triggered.
A test that must continue over time
An updated model may respond differently to the same inputs. A new RAG source may contain hostile instructions. A connector may extend the agent’s permissions. A seemingly marginal change to the prompt may weaken a refusal or create a contradiction between rules.
This is why red teaming becomes part of the system’s life cycle. It should be repeated before launch, after significant changes, following incidents and at a frequency consistent with the level of risk. The results must feed into policies, training, controls and investment decisions.
Artificial intelligence is often assessed by asking it to show what it can do. Maturity comes when an organisation also tries to discover where it can be led astray. Adversarial testing makes that fragility observable before it turns into harm, and makes security a concrete part of AI Governance.
New connections (FAQ)
Are adversarial testing and AI red teaming the same thing?
The terms are often used interchangeably. Broadly, adversarial testing refers to any test designed to provoke failures or undesired behaviour. AI red teaming usually describes a more structured programme, with defined objectives, scenarios, teams, evaluation criteria and reporting.
Does an adversarial test replace a penetration test?
No. The two activities examine different, complementary surfaces. Penetration testing looks for traditional technical vulnerabilities; AI adversarial testing also checks for behavioural manipulation, prompt injection, jailbreaks, misuse of tools, bias and responses that conflict with policy.
When should it be repeated?
Before going into production, after changes to the model, prompt, data, permissions or integrations, and following an incident. For high-impact or highly exposed systems, testing should form part of a periodic programme of monitoring and continuous improvement.
Sources
Google for Developers · NIST AI RMF and glossary · European Commission, GPAI obligations · OWASP GenAI Red Teaming
BLISS®
© 2026. ALL RIGHTS RESERVED