Lesson 95 — Intermediate

Agent & Tool Security

Jab tak model sirf baat karta tha, galti shabd tak simit thi. Agent mein galti action ban jaati hai — ticket update, email, query, code execution. Is lesson mein tum seekhoge useful autonomy deni kaise hai bina unrestricted control diye: tool tiers, LLM-se-bahar validation, human gates, run limits aur sandboxing.

Agent security kya hai

Core principle: model tumhara colleague nahi, untrusted planner hai. Woh plan banata hai; plan ko execute karne se pehle tumhara code decide karega ki plan allowed hai ya nahi.

Least privilege: read vs consequential

TOOL_TIERS = {
    "search_kb":     "read",
    "get_ticket":    "read",
    "update_ticket": "write",
    "send_email":    "consequential",
    "run_shell":     "consequential",
}

def run_tool(name, args, approved=lambda n: False):
    tier = TOOL_TIERS.get(name)
    if tier is None:
        raise PermissionError(f"unknown tool: {name}")   # default deny
    if tier == "consequential" and not approved(name):
        return {"status": "needs_approval", "tool": name}  # ruko, insaan dekhe
    return dispatch(name, args)

• Read: padhna — search, get, list. Inhe auto-run karao, yehi agent ki speed banati hai.
• Write: badalna — update, create. Yahan scope check: sirf us resource par jo task mein hai, aur rate limit.
• Consequential: email bhejna, purchase, delete, money, code execution — hamesha gate, hamesha log.
• Naye tool ka sawaal: "yeh galat chala toh kya bigadta hai?" Jawab bada hai? Toh tier higher karo aur gate lagao.

Validation jo LLM se bahar hai

ALLOWED_COLUMNS = {"status", "priority", "assignee"}

def validate_sql(sql: str) -> str:
    head = sql.strip().split()[0].upper()
    if head != "SELECT":
        raise ValueError("read-only endpoint: sirf SELECT")
    if ";" in sql.rstrip(";") or " union " in sql.lower():
        raise ValueError("suspicious SQL shape")
    return sql

# Har tool argument tumhare validator se guzaro:
# parse karo → allowlist check → schema match → phir hi execute.
# LLM ke "trust me" par kabhi nahi. Simple string checks fragile hain —
# real systems mein proper parser, parameterized queries aur DB-level
# permission layer use karo; yeh sirf shape dikhane ke liye hai.

Model arguments ko hostile input maano — hallucinated column, galat ID, injected SQL, ya manipulated JSON. Tumhara dispatch layer untrusted ko typed, validated, scoped call mein badalta hai.

• Schema validation: type, range, enum — arguments pehle check, phir dispatch.
• ID/ownership check: resource user ka hai ya nahi — yeh check model ke pass karne se independent hai.
• Output bhi validate: tool result se bhi malicious content aa sakta hai (indirect injection) — usse bhi tagged data maano.

Step, time aur cost limits

LIMITS = {"max_steps": 12, "max_seconds": 300, "max_cost_usd": 1.0}

def agent_loop(task, limits=LIMITS):
    spent, t0, steps = 0.0, time.time(), 0
    while True:
        steps += 1
        if steps > limits["max_steps"]:
            raise RuntimeError("step limit — run band, log alert")
        if time.time() - t0 > limits["max_seconds"]:
            raise RuntimeError("time limit")
        spent += estimate_cost(last_call())
        if spent > limits["max_cost_usd"]:
            raise RuntimeError("cost cap hit")
        step = model_next_step(task)
        result = run_tool(step.tool, step.args)
        if result.get("status") == "needs_approval":
            log_audit(step)
            result = ask_human(step)      # gate: approve / reject, record ke saath
        plan.observe(result)

• Step cap: runaway loop ka sabse common shape — ek hi tool call baar baar. Cap + repetition detector (same call 3 baar = stop).
• Time aur cost cap: bade context aur retries chupke se kharcha badhate hain. Per-run budget default ON rakho.
• Retry cap: tool fail hone par model retry karta rehta hai — bounded retries, phir human ko de do.

Sandboxing — generated code ke liye

• Container jail: model-generated code alag container/process mein — no host FS mounts, no privileged mode, default-deny network.
• Egress allowlist: bahar sirf whitelisted domains/hosts — warna code exfiltration ka naya exit ban jaata hai.
• Resource caps: CPU, memory, disk, wall-clock time — fork bomb aur zip bomb dono isi se marte hain.
• No credentials: sandbox env mein API keys, tokens, cloud creds nahi. Kaam chahiye toh scoped, short-lived token do.
• Generated SQL/commands/URLs bhi untrusted: parse + allowlist + DB permissions — code likhna model ka kaam hai, chalana tumhara control.

Exercise

Common mistakes

• Model ke tool arguments par bharosa: Model hallucinated ya manipulated arguments de sakta hai. Har argument validator/allowlist se guzaro — dispatch layer tumhara hai, LLM ka iraada nahi.
• Consequential tools bina gate ke dena: Read/write/irreversible ko alag karo. Irreversible par human confirmation + audit log — aur gate sirf UI mein nahi, dispatch layer mein enforce ho.
• Bina limits ke agent chalana: Runaway loop ek chhoti galti se shuru hota hai aur cost bill ban jaata hai. Steps, time, cost aur retries — chaaron caps default ON rakho.

intern_ki_chabiyan = [reception, store_room]
# tijori = manager_approval + register entry (audit log)
# Agent bhi wahi: pehle minimum, phir prove karo, tabhi bada access