Is AI red teaming legally required?

There is no general statute obliging an ordinary business to red-team an AI feature. Obligations appear at the edges: the EU AI Act imposes testing and adversarial-evaluation expectations on providers of general-purpose models with systemic risk and on high-risk system providers, sector regulators expect model risk management in finance and health, and federal contracting can flow testing requirements down. For everyone else the pressure is contractual — enterprise security reviews now ask for it by name, and 'no' is a slow answer to give.

How is AI red teaming different from a penetration test?

A penetration test attacks the system: authentication, infrastructure, application logic. AI red teaming attacks the behaviour: it treats the model's willingness to be persuaded as the vulnerability. The two barely overlap in technique and do not substitute for each other. A perfectly hardened application will still exfiltrate data through a completion if the model can be talked into it, and your penetration testing vendor probably did not test that unless you asked specifically.

What should a small team actually test?

Five families cover most real exposure: direct jailbreaks that push the system past its instructions, indirect prompt injection through content the model reads such as documents, emails or web pages, data leakage where the model reveals system prompts or other users' data, harmful or defamatory output about real people and companies, and — if your feature can take actions — tool abuse, where a manipulated model performs an operation the user never authorized. Run these against your actual deployed configuration, not a bare model.

What does a usable red team record contain?

Scope and date, the exact system configuration tested including model version and system prompt revision, the attack families attempted, findings with severity, what was fixed, what was accepted and why, and the date of the next scheduled run. The two fields reviewers actually read are the accepted risks and the date. A record showing three accepted findings with reasoning reads as competence; a record showing zero findings reads as a test that was not really run.

Do we need to red team again after a model version change?

Yes, at least in reduced form. A model version change alters exactly the property you tested — the behaviour under adversarial pressure — and mitigations tuned to one version can fail silently against the next. The practical arrangement is a small regression set of your previously successful attacks that you re-run on every version change or system prompt revision, with a fuller exercise on a fixed cadence.

AI Red Teaming as Evidence: What Buyers and Regulators Ask For in 2026

Why This Became a Document Request

Three unrelated pressures converged. Regulation created explicit adversarial-testing expectations at the top of the stack — for providers of the most capable general-purpose models and for high-risk system providers under the EU AI Act. Sector supervisors in finance and healthcare extended existing model risk management expectations to cover generative components. And enterprise security teams, having been burned by prompt injection demonstrations against their own pilots, added a question to the vendor questionnaire.

Only the first two are law, and most companies reading this fall under neither. The third one is what actually shows up in your pipeline, and it does not care about your regulatory classification.

Your Penetration Test Did Not Cover This

The most common misconception is that an annual application penetration test already handles it. It does not, and the distinction is not pedantic — the two disciplines look at different objects.

Penetration Test — Attacks the System

•Authentication and authorization flaws
•Injection into queries, templates and parsers
•Infrastructure and configuration exposure
•Session handling and access control gaps
•Dependency and supply chain weaknesses

Red Teaming — Attacks the Behaviour

•Instructions the model can be persuaded to ignore
•Content the model reads and obeys as if it were a user
•System prompt and context disclosure through completions
•Harmful, biased or defamatory generations about real parties
•Actions an agent takes on behalf of an attacker's text

The Five Families Worth an Afternoon

Direct Jailbreak

Straightforward attempts to push the deployed system past its instructions: roleplay framing, claimed authority, incremental escalation, encoding tricks, and instruction-override phrasing. Test against your production configuration with your real system prompt, because a bare model's behaviour tells you almost nothing about your product's.

Indirect Prompt Injection

The one that surprises teams. Put attacker-controlled instructions inside content your product ingests — an uploaded document, a scraped page, an email body, a support ticket, a code comment — and see whether the model follows them. If your feature summarizes or acts on third-party content, this is your highest-severity family and the one your users cannot defend against themselves.

Data Leakage Through Output

Attempts to extract the system prompt, internal instructions, retrieved context belonging to another tenant, or training data that should not surface. Multi-tenant retrieval is where this gets expensive: a filter applied in the application layer but not enforced at the retrieval layer will pass every functional test and fail this one.

Harmful and Defamatory Generation

Push for confident false statements about identifiable people and companies, unqualified professional advice, and biased characterizations across groups. This family maps directly onto legal exposure rather than security exposure, which is why it belongs in the same record — a defamation claim and a data breach arrive through the same feature.

Tool and Action Abuse

Only applies if your feature can do things: send, write, purchase, delete, call an API. Test whether injected instructions can trigger an action the user never asked for, whether confirmation gates can be bypassed by framing, and whether the action's blast radius matches what a compromised session should be able to reach. Agentic features move this to the top of the list.

The Record Is the Deliverable

Testing that leaves no artefact is indistinguishable from testing that never happened, and the artefact is short. Write it as you go rather than reconstructing it when a buyer asks.

Scope and date. Which feature, which environment, who ran it, when.
Exact configuration. Model and version, system prompt revision, retrieval sources, tools enabled, guardrails in place.
Families attempted. Name them, including the ones that found nothing.
Findings with severity. What worked, how reliably, and what an attacker gains.
Remediation and acceptance. What you fixed, what you accepted, and the reasoning for each acceptance.
Next scheduled run. A date, plus the trigger conditions — model change, prompt change, new tool.

The counterintuitive part: a report with findings is more credible than a clean one. Every deployed LLM feature has adversarial weaknesses, so a reviewer reading zero findings concludes the test was shallow. Three findings, two fixed, one accepted with a stated rationale describes a team that looked honestly — which is the actual thing being assessed.

Frequently Asked Questions

Can we run this ourselves or do we need a specialist firm?

Start in-house. Two engineers and a structured afternoon against the five families will find real issues in almost any deployed feature, and the record is worth more than its sophistication. Bring in a specialist when a customer contractually requires independence, when the feature makes consequential decisions about people, or when it can take actions with financial or physical consequences.

Our model provider red-teams their models. Isn't that enough?

It covers their layer, not yours. Your system prompt, your retrieval sources, your tool permissions, and your tenant isolation are all yours, and they are where the exploitable configuration lives. Provider assurance is a useful attachment to your record and not a substitute for testing the thing you actually shipped.

How do we handle a finding we cannot fix?

Accept it explicitly, in writing, with a reason and a compensating control. Indirect prompt injection in particular has no complete fix at present, so the credible posture is constraint — limit what the model can reach, require confirmation for consequential actions, keep a human in the loop where the blast radius is large, and log enough to reconstruct what happened. Documented acceptance is a decision; silence is negligence.

How often should a small team run this?

A full pass twice a year, plus a short regression set on every model version change, system prompt revision, or new tool grant. The regression set is the part that pays for itself — it is your previously successful attacks, re-run in minutes, catching the case where an upstream change quietly reopened something you fixed.

Do we have to show the report to customers?

Rarely in full, and you generally should not — it is a map of your weaknesses. The normal arrangement is a summary: scope, date, families tested, count of findings by severity, remediation status, and cadence. Offer the detailed version under NDA if a customer's own regulatory position requires it, and treat a demand for unredacted detail with no such basis as a negotiation, not an obligation.

Look Before Somebody Asks Whether You Did

The question arriving in questionnaires is not "is your AI secure" — nobody expects yes. It is whether you have examined your own feature adversarially, written down what you found, and decided deliberately what to do about it.

That is an afternoon and a one-page record. The companies that fail this are not failing on capability; they are failing because nobody has ever sat down and tried to break their own product on purpose.

This article is general information about AI testing practice and the obligations that reference it, not legal or security advice. Regulatory testing duties depend on your role and risk classification — confirm what applies to your deployment with counsel and a qualified security practitioner.