Most teams spend their security effort on what goes into a model. They scrub the prompt, filter the user input, and watch for jailbreaks. Then they take whatever the model says and drop it straight into a browser, a shell, or a database. That second move is insecure output handling, and it is one of the OWASP Top 10 risks for large language model apps. The bug is not in the model. It is in the code that trusts the model’s words as if they were a safe command.
What insecure output handling actually is
A model returns text. Your app then feeds that text into another system: it renders it as HTML, runs it as a shell line, builds a SQL query from it, passes it to an HTTP fetch, or hands it to eval. The flaw is treating that text as trusted just because a model produced it. It is still data, and it can be steered. An attacker who controls any content the model reads can shape the output, so the model’s reply is best understood as untrusted user input wearing a friendly voice.
Model output is data, not a command. The instant you run it, render it, or query with it without escaping, you have handed the next system over to whoever could influence the model.
The classic sinks
Insecure output handling is a confused deputy problem. The model has no malice, but it relays instructions into a system that grants them weight. The damage depends on where the raw text lands:
- Into a browser as HTML. Render the reply without escaping and a returned script tag executes. That is stored or reflected cross site scripting, delivered by your own assistant.
- Into a shell. Pass the text to a command line and a returned
;or backtick becomes command injection on your server. - Into SQL. Concatenate the reply into a query and you get SQL injection, the same class of bug as trusting a raw form field.
- Into an HTTP fetch. Let the model name a URL and call it, and a returned internal address turns into server side request forgery, reaching a metadata endpoint or a private service.
- Into
eval. Run the output as code and you have arbitrary code execution. There is no boundary left to cross.
A scenario: the chatbot that scripts the dashboard
Picture an invented support tool, call it Acme Desk. A chatbot answers staff questions, and its replies appear in an internal admin dashboard. The frontend takes the model’s answer and writes it into the page with innerHTML, because answers sometimes include simple formatting and that was the quick way to keep it. The model also reads customer tickets to write its replies. One ticket carries a planted instruction telling the assistant to end every answer with a specific line of markup. The model obliges. The answer that reaches the dashboard is no longer plain text:
Here is the account status you asked about.
<img src=x onerror="fetch('/api/admin/export').then(...)">When an admin opens that conversation, the browser parses the answer as HTML, the broken image fires its handler, and code runs in the admin’s session. The model never attacked anything. It wrote text. The app’s choice to render that text as live markup is what turned a poisoned ticket into cross site scripting against a privileged user. The same poisoned input pointed at a shell sink or a SQL sink would produce command injection or SQL injection instead.
Why developers fall for it
Model output reads like natural language, so it feels like a result rather than input. A raw form field looks suspicious by default. A polite paragraph from your own assistant does not. Teams that would never run eval on a query string will happily render a model reply as HTML, because the reply came from a system they built and the text looks helpful. The output looks like an answer, so it gets the trust an answer would earn from a human.
How it differs from prompt injection
Prompt injection and insecure output handling are two ends of the same pipe. Prompt injection is the input side: an attacker plants instructions in content the model reads and bends what it produces. Insecure output handling is the output side: your app takes whatever the model produced and trusts it into the next system. One steers the model, the other delivers the result. They chain cleanly. The poisoned ticket above is prompt injection; the innerHTML render is the output handling failure that cashes it in. We walk the browser leg of that chain in detail in prompt injection to XSS, and the same trust gap shows up when a model relays a tool’s response in tool output injection. Both are part of the wider AI agent attack surface.
Detecting insecure output handling
You find this by tracing data flow, not by scanning for known payloads. Follow the model’s output to every place it lands.
- Map the sinks. List every spot where model text reaches a browser, a shell, a query builder, an HTTP client, or an interpreter. Each one is a place to check.
- Check for escaping at each sink. A reply rendered with
innerHTMLor built into a query with string concatenation is the tell. Look for the missing encode step, not for a bad string. - Diff intent against effect. The user asked a question. The reply contained a script tag or a URL pointing inward. That mismatch flags the problem without recognizing any specific exploit.
Preventing insecure output handling
The fix is the same discipline you already use for user input. Treat the model’s output as hostile and handle it on the way out.
- Encode for the destination. Use context aware output encoding. HTML escape before rendering, so a script tag shows as text instead of running. Set
textContentrather thaninnerHTMLwhen you only need to show words. - Parameterize queries. Never build SQL by pasting model text into a string. Use bound parameters so the output can only be a value, never structure.
- Keep output away from shells and
eval. Do not pass model text to a command line or an interpreter. If an action is needed, map the reply to a fixed set of allowed operations. - Constrain tool arguments. When the model fills in a tool call, validate every field against an allowlist. A fetch tool should accept only approved hosts, which closes the server side request forgery path.
- Add a content security policy. A strict policy is a backstop. Even if a script slips into the page, it limits what that script can load or reach.
The assumption that breaks
One assumption holds the whole risk up: that text from your own model is safe to use directly. The attacker’s move is to control what the model reads, so the output serves them, and your trusting sink delivers it. You catch this by asking what each piece of output is trusted to do and what could steer it, not by matching payloads. An autonomous researcher that tests assumptions instead of signatures is built to find exactly this gap. As an early signal, a frontier model drove the full methodology on its own and identified and verified real access control and injection issues in test applications it had not seen before. You can read more on our about page.
This attack is one entry in our AI Agent Security Field Guide, a map of how AI agents get attacked and how to defend each one.
Frequently asked questions
What is insecure output handling?
It is an OWASP Top 10 risk for large language model apps where the code downstream of the model trusts the model’s text output as if it were safe, then feeds it into another system. The app renders the reply as HTML, runs it in a shell, builds a SQL query from it, passes it to an HTTP fetch, or sends it to eval. Because an attacker can steer the model through poisoned content, that output is really untrusted input, and trusting it turns the model into a confused deputy that delivers cross site scripting, SQL injection, command injection, or server side request forgery.
How is insecure output handling different from prompt injection?
They are two ends of the same pipe. Prompt injection is the input side: an attacker plants instructions in content the model reads and bends what it produces. Insecure output handling is the output side: your app takes whatever the model produced and trusts it into the next system without escaping. One steers the model, the other cashes in the result, and they chain. A poisoned ticket that makes the model emit a script tag is prompt injection; rendering that tag as live markup is the output handling failure.
What kinds of attacks come from insecure output handling?
It depends on where the raw text lands. Rendered as HTML in a browser it becomes stored or reflected cross site scripting. Passed to a shell it becomes command injection. Concatenated into a query it becomes SQL injection. Used to pick a URL for an HTTP fetch it becomes server side request forgery against internal services. Run through eval it becomes arbitrary code execution. The same poisoned model reply can hit any of these sinks.
How do you detect insecure output handling?
Trace data flow rather than scan for known payloads. Map every place model text reaches a browser, a shell, a query builder, an HTTP client, or an interpreter. At each sink check whether the output is encoded or escaped, since a reply written with innerHTML or built into a query by string concatenation is the tell. The clearest signal is a mismatch between intent and effect, like a question that returns a script tag or a URL pointing at an internal host.
How do you prevent insecure output handling?
Treat model output as hostile and handle it on the way out, the same way you handle user input. Use context aware output encoding and HTML escape before rendering. Set textContent instead of innerHTML when you only need to show words. Parameterize SQL queries so output can only be a value. Never pass model text to a shell or eval, validate and constrain tool arguments against an allowlist, and add a strict content security policy as a backstop.
Put an autonomous researcher on your own systems
UnboundCompute is an autonomous security researcher that reasons about how an application fits together and proves the access control and injection bugs it finds. We are opening a small number of founding design partner seats: private early access pointed at a staging target you choose, a say in what it looks for, and founding pricing. If your team ships software worth pressure testing, apply to the design partner program.