Effective threat modeling traces real attack chains across inputs, context, tools, and permissions — not just category labels.
Defences most commonly fail at trust boundaries, not within a single component.
Frameworks: OWASP vs ATLAS
OWASP
Describes what can go wrong — a risk category reference for LLM applications.
MITRE ATLAS
Describes how attackers do it — enumerating adversary goals, tactics, and techniques specific to AI systems.
Covers the full AI lifecycle: training, inference, and development.
Enables systematic threat coverage and cross-team coordination between red teams, SOC, and engineering.
How LLMs Change Threat Modeling
Blurred Trust Boundaries
LLMs blur traditional trust boundaries because all inputs arrive as text — user input, system instructions, and retrieved data are treated the same way.
Control flow becomes data-driven — application behaviour is influenced by natural language, not fixed logic.
Where ATLAS Fits
Apply ATLAS after system architecture and asset identification to identify AI components, map realistic attack goals per lifecycle stage, and prioritise threats by impact and exposure.
Typical Outputs
An AI-specific threat list with mapped mitigations and security controls.
Identified gaps in existing controls.
Trust Boundaries and Control Points
Key Trust Boundaries
Most AI failures occur when data or control crosses a boundary without enforcement. The critical boundaries are User → Orchestrator, Orchestrator → Retrieval, Orchestrator → Tools, and Orchestrator → LLM.
Primary Control Points
Input and instruction hierarchy enforcement — enforce the distinction between system prompt and user input.
Retrieval authorisation — verify that the data being returned is permitted for the user who requested it, preventing exfiltration.
Tool allowlist and approval gates — tools can modify file and database state; the orchestrator must maintain an allowlist and enforce human-in-the-loop where appropriate.
Output filtering — model output is unpredictable and untrusted (e.g. XSS); the orchestrator must filter all output before it reaches downstream systems.
MITRE ATLAS Tactics
1. Reconnaissance
Attackers attempt to enumerate all tools and documents the system has access to.
Detection: Deploy honeytokens — fake tools (e.g. get_all_secrets) or RAG documents containing fake credentials. Trigger a detection alert if a user attempts to enumerate tools or documents.
2. Resource Development (Arming Phase)
Attackers prepare offline before striking, developing AI artifacts (poisoned models, datasets, weights), infrastructure (cloud workspaces, domains), compromised accounts on model hubs, and adversarial capabilities such as deepfake pipelines.
3. Initial Access
Attackers influence or evade the model to gain their first foothold in an AI-enabled system across cloud, mobile, or edge environments.
4. AI Model Access
Most incidents involve access mediated through an API or product interface.
5. Execution
Execution is often indirect — attacker-controlled intent or logic is acted upon by the model.
LLMs blur the line between code generation and code execution.
6. Persistence
Attackers embed long-term influence into AI artifacts or agent state through agent context poisoning, model backdoors, training data poisoning, RAG poisoning, and public-facing prompt infiltration.
7. Privilege Escalation
Attackers act with permissions beyond what was originally granted — typically through tool invocations that execute without user-context authorisation checks, or by exploiting agents running with system or service-account privileges.
8. Defence Evasion
Attackers avoid detection by AI-enabled security systems.
9. Credential Access
If an LLM has access to a repository with committed secrets, attackers can extract API keys, tokens, and passwords — including credentials copied into RAG stores.
10. Discovery
Adversaries map the AI environment to understand its components, tools, and data sources.
11. Lateral Movement
Attackers pivot from AI applications and agents into underlying tools and systems.
In AI environments, lateral movement is typically API-to-API, not host-to-host.
12. Collection
Adversaries gather high-value AI artifacts and data including RAG databases, agent tool outputs (API responses and logs), model weights and prompts, and training or inference data from repositories and object stores.
13. Exfiltration
Inference API abuse — model extraction via repeated queries.
Membership inference — reveals whether specific data was used in training.
Prompt and system prompt leakage — guardrails and instructions revealed through probing.
Agents as exfiltration tools — agents are used to extract and transmit sensitive data.
14. Impact
Denial of service — technical disruption of AI systems.
Integrity attacks — poisoned outputs undermine trust in AI-generated content.
Business manipulation — workflows are manipulated through adversarial inputs.
Intellectual property theft — model extraction and weight exfiltration.