Indirect Prompt Injection
Instructions hidden in content a model reads on someone else's behalf, such as a web page, an email or a document.
What it is
A prompt injection where the payload sits in data the application retrieves rather than in what the user typed. The model reads a page, a file or a message as part of its task and meets the attacker's instructions there. NIST's adversarial machine learning taxonomy treats it as its own class because the victim never sees the payload.
Why it matters
Any assistant that reads mail, browses or searches a document store can be steered by whoever wrote that content. When the assistant can also act, the injected text can turn into an action the user never asked for.
In practice
An assistant summarizing a shared folder opens a document with white-on-white text telling it to send the folder's contents to an outside address. The assistant has a mail tool, and it tries.
What to look for
- Hidden or out-of-place text in retrieved content: tiny fonts, matching colors, comments, alt text.
- An agent action that follows reading external content rather than a user request.
- Outbound requests or messages to destinations that appear only in retrieved data.
Common mistake
Assuming internal content is safe. A shared document, a ticket or an inbound email is written by someone else, and the model cannot tell their words from yours.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Prompt Injection: Text crafted to make a language model ignore its instructions and follow the attacker's instead.
- Retrieval-Augmented Generation: Fetching relevant documents at question time and handing them to a model so it answers from them.
- Excessive Agency: An AI application that can do more than its job needs, so a manipulated model can do real damage.
- AI Agent: A language model given tools and allowed to decide, step by step, which ones to use to reach a goal.