Lesson 119 — Advanced

Monitoring & Observability

Deploy ke baad sabse bada khaali-paana ye hota hai ki system bolta nahi. Is lesson mein hum Prometheus + Node Exporter se infra metrics, backend se apne application metrics export karna, Grafana dashboards aur alerts, Langfuse se per-request traces, aur online evals + feedback ko eval suite tak wapas bhejna dekhenge.

⏱ 30 min• Advanced• Deployment on AWS

Observability kyun — chhota system bhi

Kya hai?

Observability matlab system ke andar se sawaal ka jawab mil paana — "abhi kya ho raha hai", "kal kya hua tha", "wo user kyun roya". Metrics (numbers), logs (events) aur traces (per-request story) — teeno milkar dete hain.

Kab kaam aata hai?

Pehle production request ke saath hi. Uske baad har sawaal — spike kyun, slow kyun, mehnga kyun — isi pe tickta hai.

Kahan milta hai?

Metrics ke roop mein Prometheus mein, dashboards ke roop mein Grafana mein, per-request traces ke roop mein Langfuse jaisi service mein, aur alert routes mein (Slack/email/webhook).

LLM apps ki khaas baat: har request ke andar teen alag systems hain — retrieval, model provider, tools. Kabhi kabhi sirf ek slow hota hai, aur bina per-step visibility ke tumhe lagta hai "app slow hai". Observability isi liye chahiye ki system bol sake kaunsa hissa doob raha hai.

Rule: Jab tak "kya dikhega" likha nahi, koi tool lagaana kaam nahi karta. Pehle 5 sawaal likho (requests, latency, errors, tokens, spend), phir unhe dikhane wale panels banao.

Prometheus aur Node Exporter

scrape model:
  har instance pe app /metrics endpoint
       ^
       | scrape (interval)
  Prometheus <---- Node Exporter (CPU, mem, disk, net)
       |
       +-> retention (kitne din data)

prometheus.yml (conceptual):
  scrape_configs:
    - job_name: llm-backend
      targets: ["host1:8000", "host2:8000"]
    - job_name: node
      targets: ["host1:9100", "host2:9100"]

Prometheus pull model pe chalta hai: tumhara app ek /metrics endpoint expose karta hai, Prometheus usse periodic scrape karta hai.
Node Exporter instance ka hardware/OS metrics deta hai — CPU, memory, disk, network. Ye infra ki health hai, app ki nahi.
Counters (badhta hai hamesha) aur gauges (up-down dono) — errors counter, in-flight gauge.
Alert rules wahi Prometheus query se bante hain — dashboard aur alert ek hi source of truth se.
Retention set karo — LLM metrics aksar daily spend tak zaroori hain, infinite storage ki zarurat nahi.

Small start: Shuru ke liye ek Prometheus + Node Exporter kaafi hai — baaki sab baad mein. Chhote stack pe ye do component kaafi sawaalon ka jawab dete hain, aur onboarding bhi aasaan hai.

Backend se apne metrics export karna

from prometheus_client import Counter, Histogram, Gauge

REQS = Counter("llm_requests_total", "requests", ["route", "status"])
LAT  = Histogram("llm_request_seconds", "latency", ["route"],
                 buckets=[0.5, 1, 2, 5, 10, 30, 60])
TOKS = Counter("llm_tokens_total", "tokens", ["direction"])  # in/out
BLOCKS = Counter("guardrail_blocks_total", "blocks", ["check"])
INFLIGHT = Gauge("llm_inflight_requests", "current requests")

def observe(route, status, seconds, tokens_in, tokens_out):
    REQS.labels(route, status).inc()
    LAT.labels(route).observe(seconds)
    TOKS.labels(direction="in").inc(tokens_in)
    TOKS.labels(direction="out").inc(tokens_out)
# histogram -> percentiles (p95) aasaan; average se decision nahi banta

Har request pe: count, latency, status — aur LLM-specific: tokens in/out, model call count, guardrail blocks, retries.
Histogram use karo, average nahi — p95 batata hai ki 5% users ko kya dikhta hai; average usse chhupa deta hai.
Labels (route, model, prompt version) se slice kar sakte ho — par zyada labels = zyada series = zyada storage. Sirf decision wale labels.
In-flight gauge scaling wali signal hai (L10.8 wala concurrency proxy) — isi metric pe ASG policy banti hai.
Token counter se spend nikalta hai: tokens x per-token rate — daily budget alert isi se banti hai.

Metric naming: Naam aise rakho jo baad mein dashboard query likhne mein madad kare: llm_requests_total, llm_tokens_total. "_total" suffix counters ke liye convention hai — follow karo, baad mein khud fayda milega.

Grafana dashboards aur alerts

dashboard (pehla page — 5 panels):
  requests/min       |  p95 latency
  error rate (5xx + timeouts)  |  tokens/min + spend trend
  guardrail blocks (window)

alert rules (wo jo insaan ki zaroorat rakhte hain):
  - error rate spike (window-over-window)  -> page
  - p95 latency budget ke upar laga raha   -> page
  - scaling fail / instances unavailable   -> page
  - daily spend limit ke 80% par            -> notify

alert nahi karo (dashboard dekh lo):
  normal traffic variation, chhoti metric wiggles

Alert sirf un signals pe jo insaan ki karta maangte hain — har metric pe alert = koi bhi alert nahi sunta.
Har alert ka owner aur runbook link ho — "kya karna hai" alert ke saath hi mile.
Deploy markers dashboards pe overlay karo — spike aksar usi ke saath dikhta hai, post-mortem aasaan.
Alert ko tune karte raho: jo alert hamesha ignore hota hai, usse band karo ya threshold theek karo.
Notification routing: page (phone) sirf customer-impacting ke liye, baaki chat channel.

Signal selection: Sawaal yahi hai — "is alert ko dekh ke kya action loge?" Action clear nahi toh alert nahi, panel hai. Alert fatigue sabse bhai ka enemy hai; kam alerts jo sunne hain, wahi asli safety net hain.

Langfuse se per-request tracing

ek request ki trace (Langfuse mein):

chat POST /chat/stream                       1.8s   12,400 tokens
├─ guardrail: input_scan                     12ms
├─ retrieve: topk=8                          90ms   8 chunks
│    └─ rerank                               60ms
├─ model call: provider/model@v3             1.6s   12,380 tokens
│    ├─ prompt: rag/system-prompt@v4
│    └─ guardrail: output_scan               15ms
└─ usage: in=240 out=12,140  cost≈...

# har span pe latency + cost — dobara sawaal: "mehnga kisne banaya?"

Trace = ek request ka poora raasta — retrieval, model calls, tool calls, har step ke latency aur tokens ke saath.
Prompt version har trace mein linked hona chahiye — regression ki investigation seedha us version tak pahunchti hai.
Trace search se "kal ki sabse mehnga request" jaise sawaal seconds mein hal hote hain.
Sampling: high traffic pe sab trace karna mehnga — percentage ya error pe 100% trace rakho.
Traces mein user data ka dhyan — PII hataao ya redact karo, debugging aur privacy dono ke liye.

Triage tool: Support ticket aaya "jawab galat tha" — request ID se seedha trace kholo: kya retrieve hua, kaunsa prompt, kya guardrail laga. Ye 5-minute triage bina traces hoga hi nahi.

Online evals, feedback aur loop band karna

Live traffic ki quality offline eval se alag hoti hai — sawaal mix badal jaata hai. Isliye sampled online evals chalao: traffic ka ek chhota percentage jiske paas gold comparison nahi, par judge-model ya rubric se score ho sakta hai. Saath mein user feedback (thumbs up/down + comment) bhi isi sample ke paas jaata hai.

production failure -> naye eval case ka journey:

ticket/trace  ->  case bana (question + expected + reason)
             ->  case review (insaan)
             ->  eval suite mein add (regression)
             ->  gate ab isse hamesha protect karega

# rule: har customer-finding kam se kam EK naya case chhodta hai
# warna wahi galti dobara hogi — aur tab koi ticket nahi aayega

Online eval score girne laga toh pichhla release check karo — deploy markers isi kaam ke hain.
Feedback rate bhi ek metric hai: thumbs-down ka % per feature, per prompt version.
Judge model bhi apne bias ke saath — usko human-reviewed samples pe periodically calibrate karo.
Loop tab band hua jab production ki har finding eval suite mein pahunch gayi — yahi "improve" step hai lifecycle ka.

Closing the loop: Dashboard dekhna monitoring hai; jo dekha usse eval suite mein wapas daalna improvement. Doosra step aksar chhoot ta hai — use ritual banao: weekly review mein "kaunse production finding naye ban gaye cases?" ek fixed sawaal ho.

Exercise

Question: Prometheus kis model pe chalta hai — pull (endpoint scrape) ya push?

Question: Average ki jagah latency ka wo summary jisme bura 5% dikhta hai — kya? (p95 / sum)

Question: Instance ka CPU/memory dene wala exporter kya kehlata hai?

Question: Har customer finding ko kahan daalte hain taaki wahi galti dobara na ho? (eval suite / wiki)

Common mistakes

Average latency par decision karna: Bure 5% users average mein gayab ho jaate hain. Fix: histograms se percentiles (p95) dekho — budget usse set karo.
Har cheez pe alert laga dena: Alert fatigue — notifications ka dhakkan band, asli incident miss. Fix: sirf action-demanding signals pe alert, owner aur runbook ke saath.
Traces bina prompt version ke: Regression mila par kaunse prompt se — pata hi nahi. Fix: har trace prompt/model/config version ke saath link karo.
Monitoring aur improvement alag samajhna: Dashboards dikha rahe hain, par galtiyan baar baar ho rahi hain. Fix: har production finding = naya eval case, weekly ritual se.

Desi example: Hospital ka ward monitoring

Har patient ka bed number (request id), har vaital sign ka chart (metrics), aur case sheet jismein har test ka record (trace). Doctor sawaal karta hai — "raat mein kyun badha?" — aur chart jawab deta hai. Bina chart ke sab guess hain:

bed no.   -> request id (trace tak link)
vitals    -> metrics (latency, errors, tokens)
case sheet-> spans (retrieve, model, guardrail)
rounds    -> alerts (sirf zaroori pe)
dawa change-> release marker (spike ka suspect)

Interview angle — yahan se sawaal aate hain

Q: Metrics, logs aur traces ka role? — Metrics = trends aur alerts; logs = events; traces = ek request ka poora raasta with per-step latency/cost.
Q: LLM app ke key metrics kaunse? — Requests, p95 latency, errors, tokens in/out, guardrail blocks, in-flight concurrency, spend.
Q: Prometheus kaise kaam karta hai? — Pull model: app /metrics expose kare, Prometheus scrape kare; alerts wahi query language se.
Q: Monitoring se improvement kaise? — Online evals + feedback se failures nikaalo, insaan se review karke eval suite mein add karo.

Lesson complete?

System ab bolta hai — metrics, traces aur alerts sab live hain. Agla lesson — Incidents, Rollbacks & Change Management — mein dekhenge jab kuch toot hi jaaye toh practise response kaise chalti hai.