AI Application Threat Modeling: How to Identify Risks Before Deployment

Threat model an AI application by first drawing the system: components, data flows, assets and actors. Mark trust boundaries, treating user input, retrieved documents, web content, tool outputs and model providers as untrusted or external. For each boundary and component, list threats using STRIDE plus AI references such as the OWASP Top 10 for LLM Applications, write concrete abuse cases, choose layered mitigations with owners, and record residual risk for an accountable person to accept. Revisit whenever tools, data, models or autonomy change.

Threat modelling comes before red teaming and security testing, which verify that mitigations work. An overview of AI threats is in AI security for business applications, and governance of residual risk in AI governance framework.

Draw a data flow diagram covering users, front ends, backend services, the model provider or self-hosted model, retrieval stores, tools and the systems they call, logs and traces, and administrators. Note what data moves along each flow and in which direction. Many risks become obvious once the diagram shows, for example, that an assistant reading customer emails also has a tool that sends email.

Mark every point where data crosses from less trusted to more trusted context. In AI applications the critical boundaries are: user input entering prompts; retrieved documents, web pages, emails and tool outputs entering the model's context; model output turning into tool calls, rendered HTML or database queries; and data leaving to model providers and third-party tools. Treat everything crossing into the model's context from outside as potentially containing instructions, and everything leaving the model as untrusted output.

Use STRIDE (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege) for the conventional view, then walk through AI-specific risks from the OWASP Top 10 for LLM Applications and tactics in MITRE ATLAS. Turn them into concrete abuse cases written from the attacker's perspective, such as: 'As a customer, I paste instructions into the chat so the assistant approves a refund above policy' or 'As an outsider, I email instructions that make the inbox assistant forward a summary to me.'

For each abuse case, choose layered mitigations: prevention (least privilege, validation outside the model, isolation of untrusted content), detection (monitoring tool calls, anomaly alerts), response (kill switches, rollback) and limits (budgets, approval thresholds). Prompt instructions can be one layer, never the only one. Assign owners and target dates, then record residual risk and have an accountable owner accept it explicitly.

A threat model is a hypothesis about where the system is weak. Verify mitigations with targeted tests, red team exercises and automated regression cases tied to threat IDs. Update the model when tests reveal unexpected paths.

Threat modelling finds design-level risks when they are cheapest to fix, and it gives security, product and engineering a shared picture. It depends on participants' knowledge and imagination, and it ages as the system changes. Keep it lightweight, tied to the architecture diagram and updated on every significant change.

Agents add threats around autonomy: goals being hijacked by injected content, tools combined in unintended ways, permissions inherited across agents, runaway loops and actions taken without the user's awareness. In multi-agent systems, messages between agents are another trust boundary; one compromised or confused agent can pass harmful instructions to others. Model each agent's identity, tools, data access and the channels between agents explicitly. See AI agent architecture and agent-to-agent communication.

Tie threat model updates to change triggers: adding a tool, connecting a new data source, changing the model or provider, increasing autonomy or exposing the feature to new users. Keep the model next to the architecture diagram in the repository, review it in design reviews for these changes and link each threat to its tests. A short, current threat model is far more useful than a long one written once.

A practical session for one AI feature takes two to three hours. Before it, the engineering lead prepares the data flow diagram and a list of tools, data sources and providers. In the session, walk the diagram from user input to final action, stopping at each trust boundary to ask what could enter, what could leave and what the model could be persuaded to do. Capture abuse cases on the diagram itself. Spend the last part agreeing mitigations, owners and which threats need tests.

Keep the group small but varied: the engineers who build it, a security specialist, the product owner and someone who knows the users and data. Afterwards, write up the results in the repository next to the architecture, and schedule a short review when the next significant change lands. Testing that follows is covered in AI red teaming.

An illustrative scenario, not a client case: a property management company designs a tenant assistant that can read lease documents and log maintenance requests. The threat model shows the assistant could retrieve other tenants' leases because documents are indexed by building, not by tenant. The design changes to per-tenant document filters before any code is written, and a canary test is added to the release suite.

Most weak threat models share the same gaps.

Threat modelling makes AI risks visible while they are still design decisions. Draw the system, mark trust boundaries, write abuse cases, layer mitigations, accept residual risk explicitly and verify with tests.