Lesson 170 — Advanced

Agentic AI interview scenarios

Agent ka sawaal interview mein aksar ulta chalta hai — interviewer tumse autonomy badhwata hai aur tumhe decision defend karna hota hai: workflow ya agent, kaunsi tool, kitni autonomy, kahan human checkpoint. Is lesson mein tum ek business task ka agent design karoge (planning, tools, memory, multi-step execution), ek failure-modes ka menu pakadoge (runaway loops se lekar over-powered tools tak), HITL guardrails rakhna seekhoge, aur trajectory se debug karna — kyunki test mein success aur production mein failure ka farq hi asli interview question hai.

Workflow ya agent — autonomy pehle, capability baad mein

Defence ka frame do line mein hai: "is task ka raasta pehle se likh sakte hain?" Haan toh workflow — sasta, testable, tez. Nahi toh agent — mehnga, unpredictable, par flexible. Interview mein ye hi do line decision hai; framework ka naam baad mein aata hai.

Agent design — planning, tools, memory, multi-step

agent loop (single agent, ek business task):
  goal    = user ka job (e.g. "is ticket ka root cause dhoondho")
  plan    = LLM chunta hai: kaunsi tool pehle, kab stop (think-act-observe)
  tools   = search_kb(), get_logs(ticket_id), query_db(sql), create_ticket()
  memory  = short-term: trajectory (abhi tak kya kiya) + running summary
            long-term: user prefs, past resolutions (vector store se retrieval)
  stop    = budget (max steps) + confidence check + ya tool ne final answer diya

har step ka log: tool, args, result, tokens, latency -> trajectory = debugging fuel

• Planning: interview mein poochho — plan ek baar banta hai ya har step pe? ReAct style (think-act-observe loop) flexible hai; fixed plan predictable task pe sasta aur testable.
• Tools: 3-6 sharply defined tools behtar hain 20 overlapping se. Har tool ka clear schema, useful error message aur dry-run mode — model ko error se hi sikhna padta hai.
• MCP: tool integrations ko standardized plug banaata hai — faida naya tool add karna, darr versioning aur permission surface ka. MCP se tool aasaan hota hai, permission model automatic nahi.
• Memory: short-term trajectory context mein; long-term store mein retrieval se. Dono alag problems hain — long-term memory galat ho toh agent confidently galat path pe jaayega.
• Multi-agent: tab jab genuinely alag concerns ho (researcher vs writer). Warna ek agent ke andar hi kaam hota hai — communication overhead zyada, benefit zero.

Failure modes ka menu — signal pakdo, fix batao

failure                  signal                      fix
runaway loop             same tool repeat, budget     max steps, progress check, hard stop
poor tool selection      wrong tool, phir retry       fewer tools, descriptions, few-shot
tool failure             timeout/5xx/empty result     idempotent retries, clear error to model
nondeterministic run     do runs, alag raaste         evals on outcomes, guardrails on paths
token cost blowup        long context, deep loops     summarise trajectory, cap context, cache
context degradation      purani steps bhool gaya      running summary, key facts pinned
over-powered tool perms  delete-all / prod scope      least privilege, read-only default

• Runaway loops: model ek hi tool baar baar leta hai (search, search, search). Max steps + "kya progress ho rahi hai" check + hard stop — budget agent ka nahi, tumhara design decision hai.
• Poor tool selection: aksar tool hi galat hain — overlapping descriptions, bhayaanak error messages. Model ko dosh dene se pehle tool ka naam, schema aur error text dobara padho.
• Nondeterministic trajectories: do runs ne alag raaste liye. Eval hamesha outcome pe rakho, path pe nahi — path badalna agent ki freedom hai, failure nahi.
• Context degradation: lambi trajectory mein beech ke steps bhool jaate hain. Running summary har N steps + critical facts (ticket id, decision) pinned rakho.
• Over-powered permissions: sabse khatarnak. Ek read-only tool aur ek delete-all tool — blast radius ka farq poore system ka hai.

Human-in-the-loop checkpoints aur guardrails

approval_policy:            # frame — apne product ke risk se bharo
  read_only:    agent khud chala sakta hai, log ke saath
  low_risk_write: agent chala sakta hai + undo window + audit log
  high_risk:    human approval mandatory (refund, delete, external email, prod deploy)

guardrails:
  scope:        approval per RUN, na ki per user (warna session reuse se leak)
  timeout:      approval ka time na mile -> default DENY, na ki default allow
  budgets:      max_steps, max_tokens_per_run, max_cost_per_day
  filters:      input (prompt injection check) + output (PII, harmful) checks

• Checkpoint ka placement: consequential action se PEHLE, uske baad nahi. Blast radius wahi kam hota hai — response ke baad approval ka koi matlab nahi.
• Blast radius define karo: har tool pe likho "ek run mein kitna nuksaan ho sakta hai". Yahi number decide karta hai approval chahiye ya nahi.
• HITL ko fast banao: single-click approve with context (kya hoga, kisne maanga, agent ka reasoning). Dheema approval = log mein approve dabana — guardrail ka dushman khud team hai.
• Input bhi ek attack surface hai: user ke text mein chhupa hua prompt-injection style instruction agent ko galat tool pe le ja sakta hai. Isliye untrusted input aur tool instruction alag rakho.

Trajectory se debug karna — test success vs prod failure

trajectory (debug fuel) — step by step padho, output nahi:
[
  {step: 1, tool: search_kb,     args: {q: "refund policy"},   ok: true},
  {step: 2, tool: search_kb,     args: {q: "refund policy"},   ok: true},  <- repeat: loop smell
  {step: 3, tool: query_db,      args: {sql: "update ..."},    blocked: by_policy},
  {step: 4, tool: create_ticket, args: {...},                  ok: true}
]

test mein success, prod mein failure ke aam kaaran:
  test queries clean the, prod queries messy/indian-english/spanglish
  prod tools ke paas rate limits/latency hain jo test stubs mein nahi the
  prod memory mein purane, galat resolutions bhi retrieve ho rahe hain

• Pehle tool sequence padho: "kya kiya" se pata chalta hai "kya socha" — args aur order hi sabse tez signal hain, final output baad mein.
• Tool errors bhi design hain: error message vague ("something went wrong") toh model guess kar leta hai — aur guess galat. Clear, structured error do model ko bhi.
• Test aur prod ka farq: stubbed tools vs real tools, clean inputs vs messy inputs, chhota corpus vs bada + stale memory. Ye teeno cheezein mock mein nahi dikhti.
• Kab nahi jaana autonomous: deterministic compliance rules, high-stakes irreversible actions, ya kam data jahan eval karke prove hi nahi kar sakte. Wahan workflow hi rakho.
• Multi-agent kab: jab ek agent ka context window ya role separation genuinely fail ho raha ho. Fashion ke liye multi-agent = debugging ka nightmare.

Exercise

Common mistakes

• Har task ko agent bana dena: Predictable raasta wale task pe agent = mehnga, unreliable, untestable. Fix: workflow first; agent sirf jab raasta genuinely unknown ho — aur autonomy ladder pe sabse neeche se shuru.
• Bahut zyada tools dena: 20 overlapping tools = model confuse hota hai, galat chunta hai, tokens bhi udata hai. Fix: 3-6 sharply defined tools, distinct schemas, useful error messages.
• Agent ko sabse powerful tool de dena: Delete-all ya prod-deploy access — 99 sahi runs ke baad ek galti explosive. Fix: least privilege, read-only default, high-risk actions ke liye mandatory HITL.
• Trajectory log na rakhna: Production mein fail hua aur sirf "model ne galti ki" — debugging possibility hi nahi bachi. Fix: har step ka tool, args, result aur tokens log karo; wahi asli evidence hai.

menu (tools)      -> 3-6 items, alag-alag, clearly named
refund policy     -> waiter khud nahi, manager (HITL) approve
tip jo kabhi nahi -> delete-all wali dish menu mein hi nahi
kitchen ticket    -> trajectory log: step, order, status
busy night (prod) -> running tally (summary) bhoolta nahi