OWASP LLM Top 10 · LLM05
Improper Output Handling
Treating model output as safe and passing it straight into a browser, shell, query, or API.
What it is
This is the flip side of prompt injection. Instead of trusting input, the app trusts output. Model responses get passed downstream and rendered as HTML, run as code, dropped into a database query, or placed in a shell command or HTTP request, all without validation. The model becomes a confused deputy: it produces a payload, and your own system runs it.
How it shows up in real apps
- Rendering model output as HTML/Markdown without escaping, which gives you stored or reflected XSS.
- Feeding generated SQL or commands to a database or shell, which gives you injection.
- Letting the model construct URLs the backend then fetches, which gives you SSRF.
- Auto-executing generated code in a privileged context.
A concrete example
Scenario
A chat widget renders assistant replies as HTML to show links and formatting.
Attack
A prompt-injected reply contains an <img src=x onerror=...> tag.
Result
The script runs in the next user's browser. Classic XSS, delivered through the model.
How we test for it
We craft inputs that make the model emit active payloads (script, SQL, shell, URLs), then check what your application does with them downstream. Does it escape, validate, and least-privilege the output, or does it just run it? This is the point where an LLM bug turns into an ordinary web or app vulnerability.
How to reduce the risk
- Treat model output as untrusted user input: encode and escape on render, parameterise queries, sandbox any execution.
- Allow-list formats and destinations. Never let generated text directly choose a URL the server fetches.
- Apply least privilege to anything the output drives (DB role, shell, API scope).
- Add a validation layer between the model and any executor.
EU AI Act: commonly maps to Art. 15 (cybersecurity). Redproof reports findings as independent testing evidence, not a conformity verdict.
Test this on your own AI before someone else does
Redproof is independent red-teaming for LLM and AI-agent products. We probe your system for improper output handling and the rest of the OWASP LLM Top 10, hand you severity-ranked findings with reproductions, fixes, and EU AI Act mapping, and re-test after you patch. That is the evidence your self-assessment needs, before a regulator or customer asks.