RAG pipelines introduce a new prompt injection surface — external content retrieved from a vector store enters the model's trusted context window without being verified.
Vector and embedding weaknesses arise when authorisation is missing at the vector layer — semantic search ignores traditional access boundaries, and shared or multi-tenant embedding stores without isolation can expose data across tenants.
Mitigations
Apply access controls at the agent level — each agent should have access only to the tools and data it requires.
Scope all similarity searches to the authenticated user or tenant to prevent cross-tenant data exposure.
Misinformation and Output Risk
When the Model Gets It Wrong
Misinformation risk occurs when the AI produces incorrect or misleading outputs that are trusted and acted upon — particularly dangerous in internal assistants, decision-support tools, and policy guidance.
Contributing factors include missing context, incomplete or low-quality retrieval data, and no validation or human review layer.
Key Principle
LLMs generate plausible-sounding information that may be false. Always verify AI-generated facts, citations, and policies.
Supply Chain and Model Loading Attacks
What We Can Attack
The model itself is an attack surface — attackers can target the files and dependencies that make up the model, including weights, tokenizer configuration, and embedded Python code.
Model loaders can execute code in two ways: directly when loading or importing code, and indirectly during deserialisation routines.
Backdoor and Data Poisoning
Poisoning at Scale
As few as 200–250 malicious documents on the internet is sufficient to backdoor a model during training.
Data poisoning is not easily reversible — once a corrupted sample enters the training process, the only reliable fix is to remove the data and retrain the model from scratch.
Mitigations
Implement data filtering pipelines to screen training data before ingestion.
Apply backdoor detection techniques to identify anomalous patterns introduced during training.
Unbounded Consumption
When Models Consume Too Much
Unbounded consumption occurs when a model consumes excessive compute, tokens, or external resources — allowing users to trigger costly operations through unrestricted tool calls or API usage.
Mitigations
Enforce quotas and per-user budgets to cap resource usage.
Apply rate limits and timeouts to prevent runaway processes.
Set hard caps on the number of agent steps and tool calls per session.
