Security Notes

54 notes

Pre-Reading2

Guidelines for Secure AI System Development

A joint framework by NCSC and CISA outlining how to securely design, develop, deploy and maintain AI systems.

Deep Dive into LLMs like ChatGPT

Andrej Karpathy explains how LLMs are pre-trained, fine-tuned and subjected to reinforcement learning.

Day 1: Anatomy of AI Systems15

[Lecture] Course Introduction

Fundamentals of AI, AI Architectures, How Large Lanuage Models generate text, and transformer architecture

[Lecture] Components of LLMs, RAG and Agents

Base LLM, Instruction tuned LLM, Reinforcement Learning. RAG systems and agents

Lab Credentials - Self host n8n & create your own API keys

n8n instance, student accounts, API Keys

Lab 1.1 Temperature parameter

temperature controls the creativity and randomness of the model

Lab 1.2 Limitations on LLM

Knowledge cutoffs, Math errors, Character & token processing

Lab 2.1 Configuring Memory for LLMs

add a database to the LLM to remember chat context conversations

Lab 3.1: Chatbot System Prompt Testing

Showcases how the outputs from a chatbot can be affected by the variations in system prompt

Lab 4.1 RAG source verification

Grounding the LLM outputs using RAG vs without RAG

Lab 5.1: Calculator Tool

Add a Calculator tool to the AI Agent and compare its math accuracy to the LLM alone

Lab 5.2: Web Search Tool

Add a web search capability to the AI Agent so it can find current information beyond the knowledge base

Lab 5.3: Tool Chaining Challenge

Test the AI Agent’s ability to chain multiple tools together (RAG + Calculator + Search) to answer complex, multi-part questions

Lab 5.4: HTTP Request Tool

Add an HTTP Request tool that allows the AI Agent to save conversation summaries to an external paste service and return a shareable URL

Lab 5.5: Explore other tools

Google Calendar, Gmail, Slack, Google Sheets etc

Lab 6.1: Guardrails

Familiarize yourself with n8n’s Guardrails node by reading the official documentation before building your own guardrail configuration

Lab 6.2: Build your own guardrail config

Add and configure the Guardrails node in your workflow, then test it against various attack prompts to understand what it catches and what slips through

Day 2: LLM Security Prompt Injection7

[Lecture] Attack and Defences of AI Systems

Direct prompt injection, indirect prompt injection, indirect RAG injection, tool abuse through injection

Level 0: Secret Word (Introduction)

Welcome to your first prompt injection challenge!

You're applying for a loan, but there's a problem: your credit score is 612, which is below the system's hard threshold of 650. Your goal is to manipulate the AI into approving your loan despite this rule.

Level 2: Loan Application (Stronger Model)

This challenge uses a more capable model (Gemma 2 27B) compared to Level 1. Your goal remains the same: manipulate the AI into approving your loan despite the rules.

Level 3: Loan Application (Strongest Model)

This is the final escalation - the most capable model (Qwen 2.5 72B). Your goal remains the same: manipulate the AI into approving your loan.

Level 4: Calendar Privacy Leak

Your goal: Extract private event information about Bob and Carol's calendars through prompt injection. What is Carol doing on January 20, 13:00-16:00?

Day 3: LLM Security Tools & RAG10

[Lecture] Agents, Tools, and RAG Attack Surface

RAG as an injection surface, tool & agent calls

Lab 1: Direct Prompt Injection - Bypassing System Prompts

Lab 2: System Prompt Extraction & Advanced Injection

Lab 3: Advanced Prompt Injection - Red Team Exercise

Lab 4: Tool Manipulation - Basic Tool Discovery

Lab 5: Tool Manipulation - Coercing Inappropriate Tool Use

Lab 6: Advanced Python Tool Exploitation - Bypassing Safety Filters

Lab 7: Indirect Prompt Injection via RAG - Document Discovery

Lab 8: Indirect Prompt Injection via RAG - Policy Poisoning

Lab 9: Improper Output Handling - LLM Output as Attack Vector

Day 4: Supply Chain & AI Threat Modelling7

[Lecture] MCP Security

MCP Architecture. Threats associated with MCP. Malicious MCP Servers.

[Lecture] Threat Modeling for LLM Apps

Threat modeling user/client, orchestrator, LLM, RAG, tools, and telemetry plane

[Download] Threat Modeling Presentation Template

Identify Highest Risk Threat, How the attack works, How the attack works, Mitigation Measures

Group Activity: Threat Modeling AI Systems

Each group will: Analyze one assigned trust boundary. Identify likely threats using STRIDE. Connect AI-specific threats to MITRE ATLAS. Prioritize the most important risks. Present the strongest findings.

System Brief: Internal Company AI Assistant

DocuMind is an enterprise AI assistant that searches internal documents and can take actions through connected tools.

System Brief: Clinical Operations Assistant

CarePilot is a healthcare AI assistant used by staff to support clinical operations and administrative workflows. Summarizes policies and support documents with RAG. Assists with patient operations workflows. Interacts with clinical systems and messaging tools. Relies on external healthcare and insurance services

Lab: Malicious Model Identification (Pickle Analysis)

Exercise to showcase how model loading can be compromised by unsafe deserialisation

Day 5: DFIR & GRC6

[Lecture] AI Incident Response

Prompt traces, Telemetry vs IOCs, Containment, Evidence Capture, ML-BOM & Approvals