AI Incident Response
The Four Phases
- Prepare — establish tracing infrastructure and assign playbook ownership before an incident occurs.
- Detect and Analyse — monitor across prompt, tool, and RAG layers to identify anomalous behaviour.
- Contain — kill the agent, roll back actions, and isolate affected components.
- Key dependency: IR depends entirely on telemetry — without AI turn traces, you are blind.
AI Incident Types
What Constitutes an AI Incident
- Unauthorised data exposure.
- Unauthorised actions via tools such as deletions or refunds.
- Model, prompt, or RAG poisoning.
- Supply chain compromise.
Severity Assessment Questions
- Did data cross a trust boundary?
- Is the behaviour persistent across sessions?
- Do you have sufficient telemetry to prove scope?
Detection: Huntable Indicators of Compromise (IOCs)
What to Hunt For
- Jailbreak and policy breach triggers detected by runtime filters.
- Unusual retrieval scores from the vector store.
- Spikes in sensitive content hits.
- Abnormal usage patterns across sessions or users.
Telemetry and Control Evidence
AI and LLM Platform Logs
- Model audit logs and tool invocation logs including actual API calls, parameters, retrieval queries, and actions taken.
- Guardrail decision logs with reason codes.
- Model version information.
SOC Telemetry
- Identity and access logs tied to UserID.
- Backend audit trails.
- Network and session logs.
Logging Architecture
Hot Log (Per AI Turn)
Minimal fields retained for hunting and correlation:
| Field | Description |
|---|---|
| trace_id, session_id | Turn and session linkage |
| user_id | Identity binding |
| model + param_hash | Model version and config |
| policy_outcome + reason_code | Guardrail decision |
| index_build | Retrieval context reference |
| tool_name + args_hash | Tool invocation record |
| output_class | Output classification |
| tokens_in / tokens_out | Usage metering |
Vault Log (Break-Glass)
Raw prompts and raw tool input/output — access controlled and audited:
- Encrypted with separate access controls; access requires a ticket with who, why, and when recorded.
- Sensitive fields are redacted or tokenised.
- Retention policy aligned to compliance requirements.
Evidence Collection
Minimum Evidence Set per Incident
- User and system prompt (hashed).
- Model type and version.
- Tool usage record.
- Retrieval queries and results.
Triggers to Start an IR Effort
Escalate When You Observe
- Model ignoring expected rules or policies.
- Input containing classic injection cues.
- Unexpected tool behaviour — wrong target, strange parameters.
- Multi-turn obfuscation techniques such as base64 encoding, encoded text, or language switching.
- Outputs that suggest data leakage.
Approval Workflow
Gates Required Before and During Deployment
- Security and legal review of risk before any AI system goes live.
- Model selection approval — vendor assessment, technical evaluation, and addition to the approved model registry.
- Deployment approval — pre-deployment checklist, AI Bill of Materials (AI-BOM), and a defined rollback and monitoring plan.
- Runtime approval — defined triggers for escalation, all decisions logged for audit trail.