CyberKit

Prompt Injection

Also called Direct prompt injection, AML.T0051.

Text crafted to make a language model ignore its instructions and follow the attacker's instead.

What it is

An attack where input to a language model application is written to override the instructions the developer gave it. In the direct form, the attacker types the payload into the application themselves. MITRE ATLAS tracks it as AML.T0051, LLM Prompt Injection.

Why it matters

It is the AI equivalent of an injection flaw, with one difference: there is no reliable escaping. Defenses reduce the odds and limit the damage, so what the application is allowed to do matters more than how well the prompt is worded.

In practice

A support bot is told to answer only billing questions. A user writes that the previous rules no longer apply and asks for the internal refund policy, and the bot quotes it.

What to look for

  • Inputs that address the model directly, such as "ignore the above" or a fake system message.
  • Responses that reveal configuration, switch persona, or step outside the application's job.
  • Tool calls or data access that the user's request does not explain.

Common mistake

Relying on a stronger system prompt as the fix. Wording helps a little; limiting what the model can reach and checking its output before acting on it helps far more.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • Indirect Prompt Injection: Instructions hidden in content a model reads on someone else's behalf, such as a web page, an email or a document.
  • Jailbreak: A prompt that gets a model to produce output its safety training is meant to refuse.
  • System Prompt Leakage: Getting a model application to reveal the hidden instructions its developer gave it.
  • Excessive Agency: An AI application that can do more than its job needs, so a manipulated model can do real damage.

Sources