Lesson 108 — Advanced

Monitoring, Feedback & Continuous Improvement

Launch ke baad static product kabhi nahi sudharta. Evals, traces, metrics aur user feedback ko architecture mein wire karo — tabhi system deploy → observe → improve → redeploy loop par chalta hai aur har bug se naya test case banta hai.

⏱ 10 min• Advanced• AI Pipeline Orchestration

Kya kya monitor karo

WHAT

Production AI system ka instrumentation: operational metrics, request traces, sampled evals aur feedback signals — sab architecture ka hissa, baad mein add hone wala nahi.

WHEN

Day one se. Pehla incident aane ke baad instrumentation add karna hamesha mehnga padta hai — chhote app mein bhi basic logs pehle se.

WHERE

Traces (har request ka journey), metric dashboards, feedback buttons/implicit signals, periodic eval jobs (L7) aur version registry — ek feedback loop mein jude hue.

Quality: sampled eval scores, citation/hallucination checks, refusal rate — trend dekho.
Latency: per stage (retrieval, TTFT, total), p95.
Cost: tokens per request per feature, cache savings — dashboard (L10) par.
Errors: provider errors, validation failures, tool failures, empty retrieval rate.
Retrieval behaviour: zero-hit queries, kaunsa source top par, rerank drop.
Drift: same eval set periodic run — provider/model update se quality silent drop ho sakti hai.

Sab kuch nahi: Pehle wo jo user ko dukhti hai (latency, errors, quality) aur wo jo bill ko dukhata hai (tokens, cache miss). Baad mein add karte jao — par pehla set day one se chahiye.

Explicit aur implicit feedback

Explicit: thumbs, rating, "wrong answer" report, user ka khud sahi jawab likhna — kam volume, high signal.
Implicit: regenerate, edit-and-resend, abandon, follow-up correction — bina bole batata hai jawab nahi jama.
Context ke saath store karo: request snapshot + response + trace id — warna feedback ka debugging value nahi.
Bias yaad rakho: sirf gussa wale feedback dete hain — implicit signals se balance karo, warna picture adhuri rahegi.

Surface banao: Feedback ko product experience ka hissa banao — sirf survey nahi. "Is jawab mein kya galat tha?" wala seedha data eval set jaata hai (neeche).

Failures bane evaluation cases

def on_bad_response(trace, issue, fixed_answer):
    dataset.append({
        "id": f"prod-{trace.id}",
        "input": trace.request,
        "expected": fixed_answer,          # user correction / reviewer fix
        "source": "production",
        "issue": issue,                    # hallucination, wrong chunk, refusal
    })
    return dataset                         # agla eval run isi par -> regression gate

Yahi loop ka dil hai: production failure → eval case → fix ka proof. Bina iske har bug fix ek guess hai aur wahi bug chupke se wapas aata hai.

Har failure triage karo: Issue classify karo: prompt bug, retrieval bug, routing bug, model limit. Galat classification = galat fix — aur eval case bhi galat banega.

Versioning — regressions traceable rakho

Prompt: version tag + diff — chupke se prompt badalna band.
Model: pin version, upgrade explicit — "latest" kabhi bhi badal sakta hai.
Retrieval config: chunking, k, reranker, index snapshot — config bhi version.
App changes: deployments trace mein — regression kaunsa change laaya yahi pata chalta hai.
Response snapshot: har answer par config id log karo — "kal tak theek tha" ka answer version history se.

Regression chain: Alert aaya → versions diff dekho → eval set replay → culprit pin → rollback. Ye chain tabhi chalti hai jab har cheez pinned aur logged hai.

Deploy → observe → improve → redeploy

deploy -> observe (traces/metrics) -> collect feedback
  -> evaluate (offline + online) -> improve (prompt/config/model)
  -> version bump -> deploy

har lap par SAME eval set — taaki comparison fair rahe

Loop ka discipline: improvement bhi change hai, aur har change par gate (L7 evals) aur version (upar wala section). Sirf deploy-deploy se system bina compass ke chalta hai.

Cadence: Roz: metrics/feedback scan. Hafte: eval set run + triage. Release: gate + version bump. Cadence team ki capacity se rakho — perfect process jo chalta nahi, process nahi hai.

Exercise

Question: Thumbs/rating jaisa saaf feedback — do shabd, English?

Question: User ke behaviour se nikla feedback (retry, edit) — do shabd, English?

Question: Prompt, model aur retrieval config ka record rakhna — ek shabd, English?

Question: Deploy → observe → improve → redeploy ka loop — do shabd, English?

Common mistakes

Launch ke baad instrumentation: Pehla incident aane ke baad logs add karoge toh us incident ka data hi nahi bacha. Day one se basic metrics + traces chalu rakho.
Sirf explicit feedback: Ratings kam aati hain aur gussa wale zyada dete hain. Implicit signals (regenerate, edit, abandon) se volume aata hai — dono ko saath lo.
Bina version ke badalna: Prompt/model/config chupke badal diya — regression ka culprit pakda hi nahi ja sakta. Pin + snapshot + diff, har change par.

Desi example: Dukaan ka roz ka register — monitoring ka desi analogy

Dukaan har roz bikri likhti hai — mahine mein pata chalta hai kaun sa saman chala aur kaun nahi. AI system ka register wahi hai, bas reaction column ke saath:

row = {trace, request, response, latency, tokens, user_reaction}
weekly review = kaunsi row BAAR BAAR galat -> wahi naya test case

Interview angle — yahan se sawaal aate hain

Q: AI system mein kya monitor karoge? (quality, latency, cost, errors, retrieval behaviour, provider drift — per stage)
Q: Explicit vs implicit feedback — kaise capture karoge? (ratings vs behaviour signals, context/trace ke saath)
Q: Production bug ko eval set kaise convert karoge? (input + expected + issue label — agla run regression gate)
Q: Regression aaya toh trace kaise karoge? (pinned versions + config snapshot → diff → eval replay → rollback)

Lesson complete?

Observe-improve loop ready. Agla lesson — AI System Design Case Studies: teen real systems whiteboard karke defend karna.