Prompt injection is an architecture problem, not a filtering problem
No classifier reliably separates instructions from data inside a language model’s context. Teams that accept this design their AI systems differently, limiting what an injected instruction could ever achieve.
- Published
- September 2, 2026
- Reading time
- 12 minutes
- Written by
- MarkTillson AI Security Practice
Every security engineer who first meets prompt injection reaches for the same analogy: it looks like SQL injection, so it must have a similar fix: escape the input, validate it, filter out the malicious patterns. That analogy is the most expensive misconception in AI application security today.
SQL injection was solved by a structural change: parameterized queries keep code and data in separate channels, so data can never be executed as code. A large language model has no such separation. System instructions, user messages, retrieved documents, tool outputs and web pages all arrive as tokens in a single context, and the model decides, probabilistically, which of them to treat as instructions. There is no parameterized prompt.
This is why the OWASP Top 10 for LLM Applications (2025 edition) still lists prompt injection first (LLM01), and why the NIST taxonomy of adversarial machine learning attacks (NIST AI 100-2 E2025) treats it as a class of attack to be managed rather than a bug to be patched. The practical conclusion is uncomfortable but liberating: assume injection will sometimes succeed, and design so that a successful injection cannot do much harm.
Direct and indirect injection
Direct injection is the attack most people picture: a user types instructions intended to override the system prompt, such as “ignore your previous instructions and…”. It matters, but its blast radius is usually limited to what that user could already see and do.
Indirect injection is the more serious problem. The attacker never talks to the model. Instead, they plant instructions in content the model will later process on someone else’s behalf: a web page an agent browses, an email an assistant summarizes, a support ticket, a PDF in a retrieval index, a code comment, or the output of another tool. When a legitimate user asks the system to work with that content, the planted instructions arrive inside the context with the same standing as everything else. MITRE ATLAS catalogs both forms under its LLM prompt-injection technique.
Input classifiers and “guardrail” models reduce the success rate of known attack patterns, and they are worth having. But they are themselves models, they can be evaded with paraphrase, encoding or multi-step setups, and they cannot see intent. Treat them as a layer that lowers likelihood, never as the control that bounds impact.
The dangerous combination
The security researcher Simon Willison has described a useful test, which he calls the “lethal trifecta”. An AI system is at serious risk of data exfiltration through prompt injection when it combines three capabilities:
- access to private data: the user’s mailbox, documents, customer records, internal knowledge bases;
- exposure to untrusted content: anything an outsider could have written: web pages, inbound email, uploaded files, third-party tickets; and
- the ability to communicate externally: sending email, calling APIs, fetching URLs, or even rendering an image whose URL carries data in its query string.
Any two of these can usually be made safe. All three together create a path by which an instruction hidden in untrusted content can direct the system to read private data and send it out. The value of the framing is that it turns an open-ended model-behavior problem into a design review question that architects can answer.
Design patterns that bound impact
A growing body of work, including research published in 2025 on design patterns for securing LLM agents, converges on the same idea: constrain what the model is allowed to do after it has seen untrusted input. The patterns we apply most often are below.
| Pattern | How it works | Trade-off |
|---|---|---|
| Capability separation | Workflows that read untrusted content run without access to private data or outbound tools; privileged workflows never see untrusted content directly. | Some “do everything” agent experiences are no longer possible. |
| Plan before reading | The system fixes its plan and tool sequence from the trusted user request before it processes untrusted data, so injected text cannot add new actions. | Less adaptive to surprises in the data. |
| Quarantined processing | A model with no tools summarizes or extracts from untrusted content into a constrained schema; a privileged component acts only on that structured output. | Extra latency and engineering. |
| Deterministic egress control | Outbound calls, links and image fetches are restricted to allow-listed destinations by code outside the model. | Requires maintaining allow-lists. |
| Human confirmation for consequential actions | Sending, paying, deleting or sharing requires explicit user approval, showing exactly what will happen. | Friction; confirmation fatigue if overused. |
| Least-privilege tool credentials | Tools act with the user’s own, narrowly scoped permissions, never a broad service account. | Integration complexity. |
Retrieval pipelines deserve special attention
Retrieval-augmented generation is where indirect injection most often enters enterprise systems, because the index quietly accumulates content from many authors. OWASP’s 2025 list added a separate entry for vector and embedding weaknesses (LLM08) for this reason. Three controls make the biggest difference:
- Permission-aware retrieval. The retriever must enforce the requesting user’s access rights at query time. If the index can return a document the user could not open directly, the model can disclose it.
- Provenance on every chunk. Record where content came from and who could have authored it, so that untrusted sources can be handled differently or excluded from privileged workflows.
- Ingestion review for high-trust corpora. Content that will be treated as authoritative (policies, clinical guidance, procedures) should enter the index through a controlled process.
How to test for it
Prompt-injection testing is closer to penetration testing than to model evaluation. A useful test plan starts from the system’s capabilities and data flows, not from a list of jailbreak strings:
- Map every source of untrusted content and every tool or output channel.
- For each path from untrusted content to a consequential action or external channel, attempt indirect injection through that source.
- Test exfiltration channels specifically: links, image rendering, tool parameters, file creation and logs.
- Check whether retrieval respects permissions under adversarial queries.
- Re-run after every change of model, system prompt, tool or data source. Model upgrades change behavior.
Report findings by impact rather than by the cleverness of the payload: “an inbound email can cause the assistant to forward the user’s last ten messages to an external address”. Executives can act on that.
Where governance fits
Management-system standards do not tell engineers how to stop prompt injection, but they make sure someone has to decide. Under ISO/IEC 42001, an AI risk assessment and treatment process should identify injection as a risk for any system that processes untrusted content, record the chosen treatment and its owner, and verify that the treatment works. The NIST AI Risk Management Framework’s Map, Measure and Manage functions ask for the same discipline. The architectural patterns above are what a credible treatment plan looks like.
Key takeaways
- There is no parameterized prompt. Instructions and data share one channel inside a language model.
- Indirect injection is the enterprise risk. Anything an outsider can write may reach your model.
- Check for the trifecta: private data, untrusted content and external communication in the same workflow.
- Bound impact with deterministic controls outside the model: egress allow-lists, scoped credentials and human confirmation.
- Test by data flow and report by impact.
Sources
- OWASP, Top 10 for LLM Applications, 2025 edition, LLM01 Prompt Injection; LLM08 Vector and Embedding Weaknesses.
- NIST, AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, March 2025.
- MITRE, ATLAS knowledge base, LLM prompt injection technique.
- Simon Willison, “The lethal trifecta for AI agents”, June 2025.
- ISO/IEC 42001:2023, Information technology, Artificial intelligence, Management system.
This is how our AI red teaming engagements work: we map every data flow, attack every path to a consequential action and report what an attacker could actually achieve. Test your assistant or agent before it reaches customers.
