AI Threat Modeling: Boundaries and Evidence for Abuse Cases
AI threat modeling starts with assets and trust boundaries, then traces attacker paths across data, model, retrieval, agent, tool, identity, and application components. This guide combines NIST adversarial-machine-learning terminology with MITRE ATLAS techniques and turns the result into testable abuse cases with control owners and evidence requirements.
An AI threat model should begin with a data-flow diagram, not a list of model risks. Draw the user, application, retrieval store, model endpoint, agent loop, tool servers and identity provider, plus outbound destinations. Mark every place where data or authority changes custodian.
The model is useful once each arrow has an abuse case and a control owner, with evidence that the control ran. A generic box labeled "AI" gives a security review nothing to test.
TL;DR
- Draw components and trust boundaries before selecting a threat taxonomy.
- Identify assets and attacker access. Record goals and model influence. Add tool authority and expected impact for each abuse case.
- Use NIST terminology to describe adversarial-machine-learning attacks and MITRE ATLAS to map observed tactics and techniques.
- Assign prevention, detection, recovery, and evidence to named owners. DeepInspect covers only routed authenticated HTTP traffic between callers and LLMs.
Start with assets and decision authority
List what the system can expose or change. Assets may include prompts, retrieved documents, customer records, model outputs, API credentials, tool permissions, audit records, embeddings, training data, plus business decisions. Rank them by confidentiality and integrity, then operational consequence.
Decision authority deserves its own row. A summarization service and an agent that can issue a refund may call the same model, yet compromise produces different outcomes. Record the maximum amount, dataset, system, plus workflow each identity can affect. Include delegated authority passed from a person to an agent and onward to a tool.
The AI agent security guide covers that delegated chain. For threat modeling, make the chain visible. Put the authenticated principal above the agent box, write the permitted scope beside the tool arrow, and mark where the scope is checked.
Draw trust boundaries around actual components
A useful diagram distinguishes the browser, application backend, prompt builder, retrieval service, vector store, model gateway, provider endpoint, MCP host, tool server, plus target system. Add the identity and logging paths to the same diagram, then label each arrow with protocol, credential, data class, plus owner.
Next, mark every source of external influence. User prompts are one source. Email, web pages, uploaded files, database rows, tool descriptions, retrieved passages, plus model output can also carry attacker-controlled content. The indirect prompt injection guide shows how retrieved data can influence a model after entering through a trusted application.
Concrete diagrams expose missing controls quickly. A yellow sticky note beside "retrieval result" may reveal that provenance disappears before the prompt builder. A red line from "model output" to "payment API" may reveal that one service account authorizes every transaction.
Describe attacker capability with NIST terminology
NIST's Adversarial Machine Learning taxonomy, published as NIST AI 100-2e2023, organizes attacks by machine-learning method, lifecycle stage, attacker goal, capability, plus knowledge. It covers evasion, data poisoning, privacy breach, trojans, backdoors, generative models plus LLMs.
Use those dimensions to replace vague threat names. "Model attack" becomes a scenario with a stage, access path, attacker knowledge, plus intended effect. A poisoned retrieval document differs from training-data poisoning. A prompt designed to reveal context differs from stealing model weights. The control owner and observable evidence change with the mechanism.
For each scenario, write preconditions explicitly. State how the attacker reaches the component, what information they possess, and what input they can modify. Then state which outcome they seek. This turns taxonomy into an engineering test rather than a decorative appendix.
Map paths with MITRE ATLAS
MITRE ATLAS is a living knowledge base of adversary tactics and techniques involving AI. It includes attacks against AI-enabled systems and manipulation or abuse of AI capabilities, along with harmful autonomous behavior supported by real incidents and realistic demonstrations.
Use ATLAS after drawing the system. Select techniques that match exposed components and attacker access, then trace a path across the diagram. A route might begin with external content, influence retrieved context, alter model behavior, collect sensitive output, then use an allowed channel for exfiltration. Another might abuse an agent's tool permission to modify a business record.
Technique coverage is useful only when tied to a boundary. Record the ATLAS identifier, affected asset, precondition, expected telemetry, prevention control, detection signal, recovery action, plus owner. The architecture decides where each mitigation can operate.
Turn each threat into a testable abuse case
Write abuse cases as concrete sequences. Name the initial actor, supplied input, component behavior, security decision, then impact. Then define a pass condition that can be observed.
For indirect prompt injection, a test might place hostile text in a retrieved document and request a summary through an authenticated session. Verify that the application preserves provenance, limits tool authority, applies model-request policy, plus prevents disallowed output handling. For excessive agency, request an action outside delegated scope and verify that server-side authorization blocks execution before the target system changes.
I reject threat registers that end with "apply guardrails." That phrase hides the control location and gives the tester no expected result. Name the component, policy, decision, and resulting artifact. The prompt injection benchmark guide explains why a model-level score also needs containment for the cases that pass through.
Separate prevention, detection, and recovery
Prevention controls include source restrictions, authorization checks, prompt-level classification, route policy, tool allowlists, transaction limits, output encoding, plus network egress rules. Place each control immediately before the action or data transfer it governs.
Detection needs telemetry from the same path: principal, session, source provenance, model route, tool request, policy version, decision, destination, plus timestamp. Alerts become more useful when they link to the event rather than to a monthly aggregate.
Recovery covers revoking credentials, disabling a route, quarantining a retrieval source, reversing a business action, notifying affected owners, plus preserving evidence. Assign the person who can perform each step. A control with no operator during a weekend incident exists only on the diagram.
Build the evidence plan beside the control plan
Every abuse case should name the record that proves the control ran. A gateway decision can prove that a routed model request was permitted or denied under a policy version. An MCP server record can prove that a tool invocation was authorized. An identity provider can prove authentication. A target application can prove the final business action.
Use a correlation identifier across those records while preserving separate events. The AI compliance logging guide lists fields for reconstructing an individual model decision. Threat-model evidence also needs test version, input fixture, expected result, actual result, reviewer, date, plus remediation owner.
Review the model after changes to providers, prompts, retrieval sources, agent tools, permissions, data classes, plus output handlers. A new arrow in the production diagram deserves a new abuse-case review before release.
DeepInspect
DeepInspect operates on authenticated HTTP traffic deliberately routed between users or agents and LLM endpoints. At that boundary it can evaluate supplied identity and role context, classify request content, enforce model and data policy, inspect responses, plus produce a per-decision audit record.
That makes DeepInspect relevant to threats whose decisive request or response crosses this specific boundary. Examples include a disallowed data class sent to an external model and an unauthorized model route. They also include a prompt and response that violate an identity-aware policy. The resulting record can support testing and incident reconstruction for that traffic.
Training-data poisoning, model-weight theft, local MCP stdio, credential theft, browser rendering, tool execution, plus downstream business actions require controls at their own boundaries. A credible AI threat model shows those limits in ink. Book a technical deep dive at deepinspect.ai.