Jailbreak
Also called LLM jailbreak, AML.T0054.
A prompt that gets a model to produce output its safety training is meant to refuse.
What it is
An attempt to bypass a model's built-in safety behavior, usually through role-play, hypotheticals, encoding or long chains of setup. Where prompt injection hijacks an application's instructions, a jailbreak targets the model's own restrictions. MITRE ATLAS tracks it as AML.T0054, LLM Jailbreak.
Why it matters
A jailbroken model in a business application can produce content that creates legal or safety exposure in the company's name. It is also often the first step before a more targeted injection.
In practice
A user asks a customer-facing assistant to pretend it is a character with no rules, then asks the character for instructions the assistant would normally refuse.
What to look for
- Prompts built around role-play, fiction or a claimed test mode.
- Requests encoded in Base64, another language or split across turns.
- A refusal followed by many rephrasings of the same request.
Common mistake
Treating a model's refusal as a security control. Refusals are probabilistic; anything that must never happen needs a control outside the model.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Prompt Injection: Text crafted to make a language model ignore its instructions and follow the attacker's instead.
- Guardrails: Checks around a model that filter what goes in, what comes out, and what it is allowed to do.
- AI Red Teaming: Attacking an AI system on purpose, before someone else does, to find how it fails.