Lesson 120 — Advanced

Incidents, Rollbacks & Change Management

Production mein kuch tootega hi — sawaal ye hai ki raat 2 baje kya hoga. Is lesson mein hum LLM-specific incidents pehchanna, har cause ke liye sahi lever, chhoti runbooks, provider model change par re-evaluation, aur blameless reviews dekhenge.

⏱ 10 min• Advanced• Monitoring & Observability

Incident kya hai — aur kyun practise karna

Kya hai?

Incident matlab user-facing problem jise turant contain aur recover karna hai — page ya alert se shuru, service restore hone tak chalta hai. Post-mortem alag hota hai, incident ke baad.

Kab kaam aata hai?

Jab bhi alerts fire ho ya user complain kare — par asli taiyaari usse pehle hoti hai: runbooks aur rehearsals. Pehli baar raat ko plan banane se behtar hai ki plan pehle se likha ho.

Kahan milta hai?

Runbooks repo mein, alert channels par, aur rollback commands ke roop mein — incident management wahi jagah jahan monitoring khatam hota hai aur action shuru hota hai.

LLM incidents ka khaas dard: kuch chal raha hai, par galat. Server down ho toh 500 dikhta hai — prompt regression mein app normal dikhta hai, sirf jawab bekaar hain. Isliye signals (L10.9) aur levers (yahan) ka rishta pehle se likha hona chahiye.

Golden rule: Restore pehle, root cause baad mein. Service wapas laao (rollback, fallback, rebuild), phir socho kyun hua. Incident ke waqt investigation karne se sirf aur lamba incident banta hai.

Har cause ka apna lever

signal                          -> cause           -> lever
-------------------------------------------------------------------
provider 5xx / timeouts spike   -> provider outage -> model fallback
quality complaints + pichhla rel -> prompt regression-> label rollback (L10.3)
retrieval mein purani info      -> stale index     -> index rebuild
agent steps/spend explode       -> runaway agent   -> spend cap + kill loop
spend graph neeche gir gayi     -> silent update   -> re-pin + re-eval
usd/din spend limit paar        -> cost spike      -> spend cap + alert

# rule: pehle cause pehchano, phir lever — sabko "restart" karna
# incident ko lamba banata hai

Provider outage: fallback model endpoint config mein pehle se rakha ho — switch ek env change.
Prompt regression: production label wapas previous version — ek call, deploy nahi.
Stale index: rebuild job documented ho, kabhi bhi chal sakta ho.
Runaway agent: per-request spend cap (L10.5) hi bachaav hai — alert ke saath auto-kill bhi ho sakta hai.
Cost spike: daily budget alert + kill switch — bill aane se pehle rokna zaroori.

Lever = command: Har lever ke liye exact command runbook mein likhi ho — "rollback karo" nahi, "registry set-label ... to=v3" likha ho. Raat ke 2 baje koi bhi soch nahi sakta, sirf chala sakta hai.

Runbooks — chhoti, sharp, tested

# runbook: provider-outage.md (chhota example)

DETECT: alert "provider_5xx_spike" OR readyz provider=false
FIRST : fallback model env set -> rolling restart
VERIFY: healthz + ek real request (stream check)
DECIDE: fallback se theek nahi? -> status page + escalation (oncall lead)
CLOSE : primary wapas? -> env revert -> blameless review schedule

# per incident type ek page ka runbook:
# signal | first action | verification | who decides

Har runbook mein teen cheezein: detect kaise (signal), pehla action (command), aur kaun decide karta hai (name/role).
Runbook ko aksar chalao (game day) — likha hua aur chala hua runbook mein farq raat wala hota hai.
Har incident ke baad runbook update — jo step kaam nahi kiya, wo fix hua, wo document hua.
Runbook repo mein, alert ke paas link — alert firing, link ready.

Size rule: Runbook ek page se lamba nahi. Lambi runbook raat ko koi nahi padhega — chhoti, command-ready list hi use hoti hai. Detail chahiye toh appendix mein.

Model changes aur blameless reviews

provider announcement: "model X updates next month"
  -> staging pin update
  -> FULL eval suite chalao (L10.4 wala)
  -> report: kaunse metric, kaunse cases bade
  -> decide: promote / adjust prompt / pin purana
  -> phir production

incident ke baad (blameless review):
  timeline -> what/when, naam nahi
  factors  -> system kaise allow kiya
  actions  -> eval case, alert, runbook, fix
  blame    -> koi nahi — banda nahi, system tha

Provider model update ek planned release hai — eval suite chalao before it reaches users, surprise banne se pehle.
Blameless review mein sawaal "kisne" nahi, "system kaise allow kar gaya" — warna log incidents chhupayenge.
Har incident ka outcome: ek naya eval case (L10.9 loop) ya ek naya alert ya ek runbook step.
Incident ka data (MTTR, kis type ka zyada) — usse dekh ke kaunsa lever sabse pehle banana hai, wahi roadmap hai.

Culture line: Blameless ka matlab accountability khatam nahi — accountability SYSTEM pe hai (runbook, gate, alert). Insan pe nahi. Jab log darr ke incidents chhupate hain, tab organisation sabse zyada vulnerable hoti hai.

Exercise

Question: Prompt regression ka sahi lever — image rollback ya production label move?

Question: Provider outage ke liye pehle se rakha backup model endpoint kya kehlata hai? (fallback / reranker)

Question: Provider ke model update announce kare toh users tak usse pehle kya chalao? (full eval suite / marketing)

Question: Blameless review ka central sawaal — kisne kiya ya system kaise allow kiya?

Common mistakes

Root cause dhundte rahe service restore ki jagah: Har minute down-time user ka hai. Fix: restore pehle (rollback/fallback), investigation baad mein — timeline ke saath.
Runbook sirf likh kar rakhna: Kabhi chalayi nahi — raat ko step kaam hi nahi karta. Fix: periodic game day; har incident ke baad update.
Provider update par kuch na karna: Chupke se model badal gaya, quality giri — koi gate nahi. Fix: announcement = scheduled re-eval, pin badlo sirf report ke baad.
Blameless ka matlab "kuch nahi hota": Bina actions ke review = post-mortem theatre. Fix: har incident se ek concrete outcome — case, alert, ya runbook step — owner ke saath.

Desi example: Doctor ki emergency kit

Doctor har emergency ke liye tayyar nahi "sochte" — uski kit mein har symptom ka fix pehle se rakha hai: pehle stabilize (restore), phir diagnosis (root cause). Kit mein jo nahi, wo emergency mein nahi banega:

symptom pehchaan -> alert signal (detect kit)
stabilize        -> fallback / rollback (restore)
diagnosis        -> timeline + trace (baad mein)
naya case file   -> eval case + runbook update
follow-up        -> blameless review

Interview angle — yahan se sawaal aate hain

Q: LLM apps ke 4 aam incident types? — Provider outage, prompt regression, stale index, runaway agent/cost spike — har ek ka alag lever.
Q: Restore aur root cause ka order? — Restore pehle (rollback/fallback), root cause baad mein; time-to-restore hi first metric hai.
Q: Runbook mein kya hota hai? — Detect signal, first action command, verification, aur decide karne wala — per incident type, ek page.
Q: Blameless review kya hai? — Naam nahi, system factors; har incident se concrete outcome (case/alert/runbook) banta hai.

Lesson complete?

Incident response ka framework ready — practise karne ke liye sab kuch hai. Ab final lesson — Capstone: LLMOps Project — mein poore level ko ek project mein jodenge.