Where does untrusted data enter your AI pipeline, and how far can it travel? That question should shape the security review. Input validation, API authentication, and rate limiting still matter, but the AI-specific attack surface needs its own examination before launch.
Pre-launch threat modeling is not an attempt to find every flaw. It helps you locate high-risk nodes before an incident does. This article presents a fast, structured method based on traditional threat modeling and adapted to LLM pipeline topology.
Map Data Flow, Not Just Components
Component diagrams show architecture and relationships. Security reviews need data flow: where untrusted content enters and how far it can move through the pipeline.
AI agent pipelines usually have more untrusted entry points than conventional services. Common examples include:
- Direct user input
- RAG content fetched from external or partly external stores
- Tool responses from external APIs
- Web content scraped or retrieved
- Database data writable by users or third parties
- Email or message content read by the agent
For each entry point, record who controls the data, whether an attacker can inject content and how easily, and what trust level you assign when placing it in the context window.
Every context item receives implicit trust. The model does not label retrieved text as "potentially untrusted." It processes the context as one input, so your threat model must reflect that.
Mark Every Trust Shift
A trust boundary is where content moves between higher- and lower-trust sources or contexts. In web security, the classic boundary is the HTTP request: the server is trusted, while the client is not.
LLM pipelines contain several trust boundaries, and they can be easy to miss:
System prompt and user message boundary Operators control the system prompt, so it should be high-trust. Users control messages, so they should be low-trust. Check whether the pipeline enforces that separation, or lets user input shape the system prompt through dynamic template variables, for example.
Internal content and external retrieval boundary A curated internal knowledge base is not the same as a vector store where users can contribute. If user-influenced content enters the RAG corpus, this boundary is weaker than it appears.
Agent output and tool execution boundary The model creates a tool call, then your infrastructure runs it. That handoff is a trust boundary. Model output becomes tool-layer input, and injection-compromised output can create a malicious call that infrastructure executes faithfully.
At each diagrammed boundary, ask what happens if an attacker controls the content crossing it. Identify the worst case.
Rank Tools by Access and Impact
Pipeline nodes do not carry equal risk. Rank them by the chance adversarial input reaches them and the blast radius if a node is compromised.
Tool access largely determines blast radius. A read-only agent that retrieves and summarizes data is less exposed than one that sends email, writes databases, or calls external APIs. Begin with tools capable of irreversible side effects.
A simple ranking model:
| Tool type | Blast radius | Priority |
|---|---|---|
| External communication, email, webhooks, Slack | High: exfiltration, spam, social engineering | 1 |
| Database writes | High: data corruption, unauthorized changes | 1 |
| External API calls | Medium-High: determined by API capabilities | 2 |
| Internal read operations | Medium: information disclosure | 3 |
| Computation-only, no side effects | Low | 4 |
For every Priority 1 tool, trace backward: which injection route could trigger a malicious call? Focus detection and prevention on that route.
Add Visibility at High-Risk Nodes
Threat modeling matters only when it informs instrumentation. For each high-priority node, ask whether you can currently see what happens there.
Visibility requires the right logs at the right detail. For email tools, log each recipient, subject, and body, not only the send attempt. For database writes, log the complete query, not merely the table name.
If that visibility is missing before launch, make it a launch requirement. Running an agent with high-blast-radius tools and no detailed call logs means operating blind. Secondary effects may appear hours or days after an attack.
Limits of This Pre-Launch Review
Pre-launch threat modeling identifies your riskiest nodes and guides defensive investment. It does not remove the attack surface or replace runtime detection.
Threat modeling happens before production. It reflects the architecture designed, the planned agent tools, and documented data flows. Production can introduce surprises: tools added under pressure, data sources expanded without review, and unexpected user behavior.
Effective pre-launch modeling creates a documented baseline of the expected attack surface. If something fails, that baseline helps explain what changed. It is a foundation, not a guarantee.
Teams we work with consistently find that mapping data flows and trust boundaries exposes at least one significant overlooked risk. Common examples include a RAG source users can indirectly influence or a tool parameter an injection payload can change. Finding it before launch is worth the two-hour exercise.