System Prompt Leakage
Getting a model application to reveal the hidden instructions its developer gave it.
What it is
The exposure of the instructions, rules and sometimes secrets that a developer places ahead of every user conversation. Attackers ask for them directly, ask the model to repeat everything above, or have it translate or summarize its own setup.
Why it matters
The leak itself is usually minor. The damage comes from what developers put there: API keys, internal URLs, business rules, or the exact wording of the filters an attacker now knows how to step around.
In practice
A tester asks a chatbot to print its instructions inside a code block. The output includes a database connection string the developer pasted in for convenience.
What to look for
- Requests to repeat, translate or summarize "everything above" or "your instructions".
- Output that contains configuration language rather than an answer.
- Credentials or internal hostnames appearing in chat transcripts.
Common mistake
Keeping secrets in the prompt and trusting the model not to say them. Assume the system prompt is public, and keep credentials in the application layer where the model cannot read them.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Prompt Injection: Text crafted to make a language model ignore its instructions and follow the attacker's instead.
- Jailbreak: A prompt that gets a model to produce output its safety training is meant to refuse.
- Large Language Model: A model trained on large amounts of text that predicts the next token, which lets it write, summarize and answer.