CyberKit

Indirect Prompt Injection

Instructions hidden in content a model reads on someone else's behalf, such as a web page, an email or a document.

What it is

A prompt injection where the payload sits in data the application retrieves rather than in what the user typed. The model reads a page, a file or a message as part of its task and meets the attacker's instructions there. NIST's adversarial machine learning taxonomy treats it as its own class because the victim never sees the payload.

Why it matters

Any assistant that reads mail, browses or searches a document store can be steered by whoever wrote that content. When the assistant can also act, the injected text can turn into an action the user never asked for.

In practice

An assistant summarizing a shared folder opens a document with white-on-white text telling it to send the folder's contents to an outside address. The assistant has a mail tool, and it tries.

What to look for

  • Hidden or out-of-place text in retrieved content: tiny fonts, matching colors, comments, alt text.
  • An agent action that follows reading external content rather than a user request.
  • Outbound requests or messages to destinations that appear only in retrieved data.

Common mistake

Assuming internal content is safe. A shared document, a ticket or an inbound email is written by someone else, and the model cannot tell their words from yours.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • Prompt Injection: Text crafted to make a language model ignore its instructions and follow the attacker's instead.
  • Retrieval-Augmented Generation: Fetching relevant documents at question time and handing them to a model so it answers from them.
  • Excessive Agency: An AI application that can do more than its job needs, so a manipulated model can do real damage.
  • AI Agent: A language model given tools and allowed to decide, step by step, which ones to use to reach a goal.

Sources