Lesson 92 — Intermediate

AI Threat Model

LLM app mein attack surface tab tak dikh hi nahi jab tak koi naksha na banaye. Is lesson mein tum ek LLM application ka threat model banaoge — assets, entry points, trust boundaries aur har boundary par sawaal. Defenses andhe tareeke se mat lagao, pehle batayo kya defend kar rahe ho.

⏱ 10 min• Intermediate• Basic computer usage

Threat model kya hai

WHAT

Threat model ek likha hua naksha hai — app ke assets kaunse hain, attacker kahan ghus sakta hai, aur har entry point par kaunsi boundary chahiye. Opinion nahi, checklist hai jo discussion ko replace karti hai.

WHEN

Naya feature aaya (tool, data source, integration) — threat model update karo. Sirf launch se pehle ek baar banakar rakhna stale documentation hai, security nahi.

WHERE

Design docs, security reviews, aur sprint planning mein. Ek A4 page kaafi hota hai: components, boundaries, top risks, aur har risk ka owner.

Funda ek line mein: defense se pehle diagram. Kaunsa data kahan se aata hai, kahan parse hota hai, kaunse action bahar jaate hain — yeh likh lo, risks khud saamne aa jaate hain.

Default instinct: Pehla sawaal hamesha yeh ho — "is feature mein sabse bura kya ho sakta hai?" Phir decide karo rokna hai, limit karna hai, ya document kar ke accept.

Attack surface kaise badalta hai

Traditional web app mein input HTTP request tak simit tha. LLM app mein input ki definition hi phool gayi — jab model untrusted content parse karta hai, private data tak pahunchta hai, tools call karta hai aur actions leta hai, toh har woh jagah entry point ban jaati hai.

User prompts: sabse obvious raasta — chat input, form fields, file uploads jo seedha prompt mein jod diye jaate hain.
Retrieved data: RAG corpus, web pages, emails, tickets — koi aur likhta hai, model padhta hai. Indirect injection ka raasta.
Tool outputs: API responses, file contents, search results — bhi attacker control kar sakta hai.
Memory: purane conversations aur stored facts jo har call par dobara jaate hain — poison ek baar, effect lamba.
Model output: model galat, biased ya manipulated output de sakta hai — yeh bhi entry point hai, trust ka nahi.

Rule of thumb: Jo bhi data model tak pahunche — chahe user bheja ho ya kisi third party ne — woh untrusted hai. Yeh line threat model ki neev hai.

Trust boundaries — naksha banao

USER / WEB PAGE / EMAIL / DOC     (untrusted)
              │
              ▼
   [PROMPT ASSEMBLY]  ── boundary 1: untrusted ko tagged block mein daalo
              │
              ▼
        [LLM CALL]     ── boundary 2: model output bhi untrusted
              │
              ▼
  [TOOL / DB / VECTOR STORE]  ── boundary 3: args validate, tiers, ACL
              │
              ▼
    EXTERNAL SYSTEMS   ── boundary 4: egress allowlist, human gate

Boundary ka matlab: yahan data ka trust level badalta hai — us par ek check padta hai, warna woh andar nahi jaana chahiye.
Har boundary ka ek owner ho: prompt assembly ka harness developer, tool calls ka service layer, egress ka platform team.
Checks tumhare code mein baithenge — delimiters, validators, ACL filters, approval gates — model ke mood par nahi.

Yeh naksha banane ka sabse tez tareeka: app ke flow ko upar se neeche likho aur har jagah line khincho jahan data ka trust badalta hai. Line ke upar wala hissa untrusted, neeche wala trusted — aur har line par ek control likho.

STRIDE — har component par sawaal

STRIDE = {
    "S": "spoofing — koi user/agent ko pretend toh nahi kar raha?",
    "T": "tampering — prompt, chunk ya tool output badla ja sakta hai?",
    "R": "repudiation — kaunne action liya, logs mein pata chalega?",
    "I": "info disclosure — PII, secrets ya private docs leak ho sakte hain?",
    "D": "denial of service — token bomb, unbounded loop, cost spike?",
    "E": "elevation — model ko zaroorat se zyada tools/permissions toh nahi?",
}

def audit_component(name):
    print(f"component: {name}")
    for cat, q in STRIDE.items():
        print(f"  {cat}: {q}")

# Har component (assembly, LLM call, retrieval, tool runner) par yahi
# checklist chalao. Khali jawab = wahi gap risk register mein likho.

Pehle assets likho: user data, API keys, vector store, email/CRM jaise external systems, aur tumhara khud ka reputation.
Phir STRIDE har component par — jawab sirf haan/naan nahi, "kaise rokenge" ke saath.
Sabse pehle high-impact + easy boundary fix karo: least privilege tools, egress allowlist, output validation — yehi teen sabse zyada cover karte hain.

Honest limit: Threat model complete list nahi hota — naye attacks aate rehte hain. Iska kaam hai blast radius aur priorities set karna, zero risk ka wada nahi.

Exercise

Question: Model ka output jab tak validate na ho, use kya maana chahiye? (do shabd, English)

Question: Threat modeling ki classic 6-category methodology — 6 akshar (Spoofing, Tampering se shuru)

Question: LLM app ke trust boundaries — prompt, retrieved data, tools, memory, model ___ aur external systems? (ek shabd)

Question: Defenses lagane se pehle kya banao? (do shabd)

Common mistakes

Defenses pehle, planning baad mein: Bina threat model ke tum wahan taala lagate ho jahan darwaza hi nahi. Pehle assets, entry points aur actors likho — uske baad hi controls choose karo.
Sirf user input ko untrusted maanna: Retrieved docs, tool outputs, memory aur model ka khud ka output bhi attacker ke raaste hain. Naksha mein har source ko untrusted zone mein daalo.
Threat model ek baar banakar bhool jaana: Naya tool, naya data source, naya integration — app badalta hai toh model stale ho jaata hai. Har bade change par revisit karo.

Desi example: Ghar ki security audit

Pehle ghar ka naksha: kaunse darwaze, kaunsi khidkiyaan, andar kya keemti. Tab taale lagte hain. Ulta kiya toh taala us darwaze pe lag jaata hai jo kisi khulta hi nahi, aur khuli khidki reh jaati hai:

ghar = assets + darwaze + khidkiyan
# 1. naksha banao   2. weak points chuno   3. tab taala lagao
LLM app = data flow + boundaries + controls — order same hai

Interview angle — yahan se sawaal aate hain

Q: LLM app ka attack surface traditional web app se kaise alag hai? (model untrusted content parse karta hai, retrieval bhi entry point hai, output action ban sakta hai)
Q: Trust boundary kya hoti hai aur app mein kitni hongi? (jahan data ka trust level badalta hai — user→prompt, doc→retrieval, LLM→tool, tool→external)
Q: Model output ko untrusted kaise treat karoge practically? (schema validation, allowlist, irreversible action par human gate, audit log)
Q: STRIDE se LLM app par mapping kaise karoge? (har component par 6 sawaal, jawab risk register mein owner ke saath)

Lesson complete?

Threat model tayyar. Agla lesson — Prompt Injection & Data Exfiltration — isi attack surface ka sabse common exploit hai.