What is AI red teaming?
Skip it and the first person to find the jailbreak is a user, on a screenshot, in public.
Red teaming
adversarial testing
AI red teaming means deliberately attacking your own model or agent before launch, using the same tricks a bad actor would. That covers jailbreaks, prompt injection, and attempts to pull out training data or system prompts. Each attempt is logged against a scope you wrote down first.
The name comes from military exercises, where a red team plays the attacker so the blue team can find gaps before a real one does. Applied to AI, the attacker is a person or a script trying to make the system say, do or reveal something it should not.
That is different from a normal QA pass. QA checks that the system does what you built it to do. Red teaming checks what happens when someone tries to make it do something else entirely.
A pre-launch gate, not a compliance ritual
Some teams treat red teaming as a box to tick before an audit, run once, filed away. Treated that way, it catches nothing new. The same five prompts get tried, the report says "passed", and the system ships with whatever holes nobody thought to probe.
A useful pass sits before every launch, and before every meaningful change: a new system prompt, a new tool, new data the model can reach. A model update from your provider counts too.
NIST's AI Risk Management Framework treats red teaming as an ongoing practice, not a one-time sign-off. OWASP's Top 10 for LLM Applications ranks prompt injection as risk number one. The attack surface moves every time the system changes, so the practice has to repeat.
Write the scope down before you start. What is in bounds, what tools the agent can actually call, what counts as a failure worth logging. Without a scope, two testers argue about whether a result even matters instead of fixing it.
- ScopeWhat is in bounds and what counts as a failure.
- AttackJailbreaks, injection, extraction attempts.
- LogEvery attempt and result, pass or fail.
- FixPatch the prompt, the tool permissions, or the filter.
- RetestSame attacks, confirm the fix holds.
Skip Retest and you have a patch you believe works, not one you know works.
Four categories cover most of what gets found
Jailbreaks try to get the model past its own instructions, often by asking it to roleplay a character with no rules. Prompt injection hides instructions inside content the model reads, such as a document or a webpage. The attack arrives through data the system was supposed to trust.
Data extraction tries to pull out the system prompt, training data, or another user's information from a shared context. Tool abuse is specific to agents. Can the model be talked into calling a tool with arguments nobody authorised, like issuing a refund it was only meant to look up?
Related questions
01How is red teaming different from a security penetration test?
A penetration test checks infrastructure: servers, networks, access controls. Red teaming an AI system checks the model's behaviour itself, through language rather than exploit code. Most production systems need both, run against different layers.
02Who should run the red team, us or an outside firm?
Either can work, and the strongest results usually combine both. Your own engineers know the system's real attack surface; an outside tester brings prompts and techniques your team has not seen yet. What matters more than who runs it is whether the findings get fixed and retested.
03How often does red teaming need to happen?
Before launch, and again after any change to the system prompt, the tools available to the model, or the model itself. A provider's silent model update can reopen an issue a previous test closed. That is why this is a recurring gate, not a one-time report.
04What does a findings log actually contain?
Each row names the attempt, the exact input used, what the system did, and whether that counts as a pass or a fail. Skip the exact input and the log is not reproducible, so nobody can verify the fix later.
