AI Red Teaming
Attacking an AI system on purpose, before someone else does, to find how it fails.
What it is
Structured adversarial testing of a model or AI application: injections, jailbreaks, data leakage, harmful output and abuse of its tools. It borrows the red team mindset and adds failure modes specific to models.
Why it matters
Model behavior changes with every update, prompt edit and new tool. Red teaming turns vague risk into concrete findings an engineering team can fix and retest.
In practice
Before launch, a team runs hundreds of injection attempts through an internal agent's ticket-reading tool and finds that a crafted ticket can make it reassign other tickets.
What to look for
- A test plan mapped to a framework such as MITRE ATLAS.
- Findings retested after each model or prompt change.
- Coverage of tool use and data access, not only harmful text.
Common mistake
Testing the chat box and stopping there. The serious findings are usually in what the system can reach and do.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Red Team: An authorized team that emulates real adversary tactics to test detection and response.
- Jailbreak: A prompt that gets a model to produce output its safety training is meant to refuse.
- Prompt Injection: Text crafted to make a language model ignore its instructions and follow the attacker's instead.
- MITRE ATLAS: MITRE's ATT&CK-style knowledge base of tactics and techniques used against AI systems.