Insecure Output Handling
Also called Improper output handling.
Passing a model's output straight into a browser, shell, query or API without checking it.
What it is
A flaw where an application treats generated text as trusted and hands it to another system: rendered as HTML, run as a command, used as a SQL query or sent as an API call. The model becomes a path for classic injection, steered by whoever controls its input.
Why it matters
It turns a prompt injection from a strange answer into code execution or data loss. The fix is the one web developers already know: treat model output like user input.
In practice
A reporting tool asks a model to write SQL from a plain-English question and runs the result directly. A crafted question produces a query that drops a table.
What to look for
- Model output rendered as markup or executed without an encoding or allow-list step.
- Generated queries or commands that touch objects the user never mentioned.
- Web pages that show script or markup coming back from an AI feature.
Common mistake
Validating the user's question but not the model's answer. The answer is the thing that reaches the database.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Prompt Injection: Text crafted to make a language model ignore its instructions and follow the attacker's instead.
- Cross-Site Scripting: Injecting attacker script that runs in other users' browsers.
- SQL Injection: Injecting SQL through input to manipulate a database query.
- Command Injection: Injecting attacker commands into a shell call the app builds.