Lesson 172 — Advanced

LLMOps & production debugging interviews

Interview ka sabse imandar sawaal ye hai: "kal raat system down tha — tum kya karte?" Is lesson mein tum ek incident-response frame seekhoge jo har AI system pe lagta hai — model APIs, self-hosted inference, retrieval services, queues, caches, rate limits aur downstream tools — phir har symptom (latency spike, outage, cost explosion, capacity limit, degraded quality) ka suspect tree, resilience patterns (fallback, graceful degradation, retries, load balancing, autoscaling), aur wo line jo decide karti hai ki alert chahiye, rollback chahiye ya investigation.

⏱ 20 min• Advanced• Fine-tuning & model adaptation interviews

Incident ka frame — detect, triage, mitigate, fix, postmortem

Kya hai?

AI incident ka matlab sirf 5xx nahi — quality drop, cost spike, latency tail, capacity limit sab incident hain. Frame hai: detect (kaise pata chala) → triage (kaunsa stage) → mitigate (abhi ruko) → fix (root cause) → postmortem (dohrana na pade).

Kab kaam aata hai?

Pahla sawaal hamesha: "kya affect ho raha hai — users ya andar ki metric?" Doosra: "kab se, kis change ke baad?" Ye do line har debugging round ki shuruaat hain.

Kahan milta hai?

On-call rotation, incident review meetings, aur har technical interview ka debugging round — jahan tumhara tarika poochha jaata hai, sirf answer nahi.

LLM systems mein failure compound hota hai: queue backlog → cache stale → retrieval timeout → fallback model chala → quality drop. Alag-alag symptoms, ek root. Isliye pehle system map saamne rakho (kaun kis pe dependent hai), phir symptom ko us map pe trace karo — warna har symptom alag-alag "fix" karoge aur incident khatam hi nahi hoga.

On-call rule: Blast radius jitna bada, investigation ka acceptable time utna kam. Sev1 mein mitigation pehle, root cause baad mein — purana known-good wapas, phir socho. "Main abhi root cause dhoond raha hoon" sev1 ka jawab nahi hai.

Symptom → suspect tree — guess nahi, list

symptom -> suspects (order mein; sab ek saath mat badlo):
latency spike   -> provider slowness, queue backlog, cache miss storm, long prompts, cold autoscale
outage          -> provider incident, quota/capacity hit, cert/DNS, deploy bug, dependency down
cost explosion  -> retry storm (exponential amplification), traffic spike, prompt bloat, cache miss
capacity limit  -> rate limit 429s, GPU queue depth, embedding backlog, per-user quota
quality drop    -> model/prompt change, data/index drift, cache serving stale, truncation,
                   retrieval failure, upstream tool returning empty

har suspect ka verification step likho: "check karo" nahi, "is query se check karo"

Latency spike: pehle per-stage p50/p95 dekho. Sirf provider slow ya poora pipeline — farq pata chalega ki fallback chalana hai ya capacity add.
Outage: "sab down" aksar ek dependency hai jisse sab juda hain (provider, quota, ek shared cache). Ek hi check se poora tree traverse karo.
Cost explosion: sabse aam self-inflicted kaaran — retry storm. Retries load ko exponentially amplify karte hain: 1x request, timeout pe 3x, uske retries pe 9x.
Capacity limit: 429s ka rate aur GPU queue depth dekho — autoscale configured hai par trigger galat signal pe (CPU instead of queue depth)?
Quality drop: change-correlated aksar. Pehle deploy/prompt/model/index ka timeline, phir golden set se stage-wise isolate.

Tree ka faida: Suspect list likhne se do cheezein hoti hain: kuchh suspects turant eliminate ho jaate hain (metrics se), aur kuchh verify hote hain. Interview mein ye list hi dikhata hai ki tumne pehle socha, phir bola — panic nahi.

Resilience patterns — fallback se autoscale tak

resilience:                # frame — numbers apne SLO se
  retries:      exponential backoff + jitter, max 2-3, idempotent ke liye hi
  fallback_chain: primary model -> alternate provider -> smaller model -> cached answer -> honest error
  degradation:  sabse zaroori feature zinda (short answer), optional off (citations, tools)
  load_balance: per-provider quotas, weighted routing, health-check se route hatao
  autoscale:    queue depth (tokens/sec) pe scale, warm pool taaki cold start na dikhe
  circuit_breaker: error rate threshold -> provider ko break, recovery probe ke baad wapas

Retries sirf idempotent ke liye: LLM call dobara chalana sasta hai (read), par uske tool side-effects ke saath nahi. Retry policy per-action decide hoti hai, global nahi.
Graceful degradation: "honest error" bhi ek fallback stage hai. Users ko partial value do (chhota answer bina citations) — zero se behtar hai.
Load balancing: health checks se route hatao — dead provider ko bhejte rehna sabse tez way hai outage ko lamba banana.
Autoscale signal: LLM workloads CPU pe nahi, queue depth (ya tokens/sec) pe scale karte hain — CPU toh inference servers pe hamesha high rehta hai.
Circuit breaker: error rate threshold cross -> provider ko break, dheere-dheere probe. Warninga tum usi down provider ko load dete rahoge.

Jitter kyun zaroori hai: Bina jitter ke sab clients ek saath retry karte hain — synchronized burst, thundering herd. Provider ki capacity phir usi waqt hit hoti hai jab sabko zarurat thi. Exponential backoff + jitter = retry ka sabse sasta insurance.

Monitoring signals ko kaise padhte hain

AI system ka dashboard (per stage):
per_stage:  p50/p95 latency, error rate, throughput (req/s aur tokens/s)
spend:      tokens per request, cost per day, cost per successful answer
retrieval:  hit-rate, empty-retrieval rate, index freshness (last ingest age)
quality:    rolling eval score, guardrail trigger rate, human feedback thumbs
capacity:   queue depth, 429 rate, GPU utilization, cache hit rate

signal ka sawaal: ye number aaj pichhle hafte se alag kyun hai?
count vs rate:   100 errors/ghanta traffic kam pe bhi alarm, rate se pata chalta hai
lagging vs leading: eval score lagging indicator hai, 429 rate leading

Per-stage, kabhi aggregate nahi: "latency 2s" se pata nahi chalega retrieval 1.8s le raha hai ya generation. Stage ke p95 alag-alag graph pe.
Token usage = cost ka predictor: tokens per request ka trend dekho — prompt bloat chupke se aata hai aur pehle cost pe dikhta hai, phir budget alert pe.
Retrieval health: hit-rate aur empty-retrieval rate — quality drop ka sabse tez leading indicator, kyunki generation toh phir bhi confident jawab dega.
Eval scores lagging hain: rolling eval ka score girta hai toh damage ho chuka. Isliye 429 rate, timeouts aur retrieval metrics pehle se alert chahiye.
Cost per successful answer: per-request cost se behtar — failed requests ka kharcha bhi isi mein aata hai, wahi asli meter hai.

Dashboard ka test: Poochho: "kya main is dashboard se 5 minute mein bol sakta hoon system healthy hai ya nahi?" Agar jawab ke liye 3 tools kholne padte hain toh dashboard nahi, archive hai.

Alert, rollback ya investigation — line kahan hai

decision frame:
alert chahiye   -> user-facing SLO breach, error rate spike, cost/day budget cross, 429 storm
rollback        -> change-correlated regression (deploy/prompt/model ke baad hi gira), fix fast chahiye
investigate     -> gradual drift (data/index), intermittent, ya abhi tak cause unknown

rollback pehle, root cause baad mein: purana known-good wapas, phir soch

API vs self-hosted incident ka farq:
  API:      provider ka status tumhara monitoring nahi; fallback chain hi mitigation
  self-hosted: tumhare paas logs/metrics hain par capacity, patching, scaling bhi tumhare

Alert ka matlab action: sirf notification nahi — har alert ke saath runbook link aur owner. Warna alert fatigue, phir sab alerts ignore, phir asli incident bhi miss.
Rollback ka trigger: correlation with change. Deploy ke 10 minute baad gira — rollback karo, debug baad mein. Bina correlation ke rollback andhera chhodna hai.
Investigation ka trigger: drift — hafte mein dheere-dheere girta hai, kisi ek change se juda nahi. Yahan rollback ka koi fayda nahi, data audit chahiye.
API incidents mein: tumhara options limited hain — fallback chain, caching, degradation. Isliye multi-provider design pehle se hona chahiye.
Self-hosted incidents mein: observability tumhare paas hai (bada faida), par mitigation bhi tumhara kaam hai — patch, restart, scale, sab on-call ka burden.

Postmortem ka rule: Blameless postmortem: trigger, detection gap, time-to-mitigate, action items — insaan ka naam nahi, system ka gap. Aur ek replay drill: fix ko test karne ke liye incident ko staging mein dobara chalao.

Exercise

Question: Cost explosion ka sabse aam self-inflicted kaaran kya hai — retry storm ya cache hit?

Question: Retries mein synchronized bursts se bachne ke liye kya lagate hain? (ek shabd)

Question: Change-correlated regression aaye toh pehle kya karo — investigation ya rollback?

Question: LLM workloads ko scale karne ke liye CPU ke bajaye kis signal pe autoscale karte hain? (do shabd)

Question: Quality drop ka sabse tez leading indicator kaunsa hai — rolling eval score ya retrieval hit-rate?

Common mistakes

Retries bina jitter ke: Sab clients ek saath dobara try karte hain — thundering herd, provider ki capacity usi waqt hit hoti hai. Fix: exponential backoff + jitter + retry cap, aur retries sirf idempotent actions pe.
Symptom pe turant code badalna: Bina stage isolate kiye fix — do changes, zero knowledge, incident repeat. Fix: pehle timeline + per-stage metrics, phir ek change, phir measure.
Cost alerts na lagana: Retry storm raat bhar chalta hai aur subah budget ka bomb — quality incident ke roop mein mehnga. Fix: daily budget alert + tokens-per-request trend + cost per successful answer.
Postmortem bina fix kiya: Same incident mahine baad dobara. Fix: blameless postmortem — trigger, detection gap, action items, aur ek replay drill jo fix ko prove kare.

Desi example: Bijli ka bill — month-end surprise

AI system bijli ke connection jaisa hai: meter (metrics) hamesha chalna chahiye, load-shedding switch (circuit breaker) time pe girta hai, backup generator (fallback) ready rehta hai, aur billing alert (budget) month ke beech mein hi aata hai — month-end pe nahi:

meter (dashboard)     -> per-stage p95, tokens/s, queue depth
load-shedding switch  -> circuit breaker: error rate threshold pe provider break
backup generator      -> fallback chain: alternate provider -> chhota model -> cached
billing alert         -> cost/day budget cross -> turant notification + owner
month-end surprise    -> alert hi nahi tha = monitoring design ka failure

Interview angle — yahan se sawaal aate hain

Q: "Kal raat system down tha" — shuruaat kahan se? — Detect (kaise pata chala), triage (kaunsa stage), mitigate (known-good wapas), fix, postmortem. Pehle blast radius, phir stage isolate.
Q: Retry storm kaise banta hai aur kaise rokte ho? — Timeout pe sab clients ek saath retry, load exponentially amplify. Exponential backoff + jitter + cap; retries sirf idempotent pe.
Q: Alert vs rollback vs investigation? — SLO breach = alert; change-correlated = rollback pehle; gradual drift = investigation (data audit), rollback ka fayda nahi.
Q: API vs self-hosted incident mein farq? — API: mitigation limited (fallback chain), provider status external. Self-hosted: observability zyada, par patching/scaling ka burden bhi zyada.
Q: Dashboard mein kya hoga? — Per-stage p50/p95, error rate, tokens/s, cost/day, retrieval hit-rate, rolling eval score, queue depth, 429 rate — sab rate mein, counts nahi.

Lesson complete?

Incident response ka frame ready — ab tum ek down system ko stage-by-stage traverse kar sakte ho. Agla lesson — AI System-Design Interviews — mein isi operating discipline ko design-time pe lagayenge.