Lesson 36 — Beginner

Cost, Usage & Observability

Print statements chhote script mein chalte hain — pipeline badi hote hi nahi chalti. Is lesson mein hum har request par kya record karein, multi-step interaction ko trace kaise samjhein, failing step kaise dhoondein aur callbacks se logging kaise lagate hain — dekhenge.

⏱ 10 min• Beginner• Robustness — Retries, Timeouts & Fallbacks

Observability kyun chahiye

WHAT

Observability matlab har LLM request ka record — model, tokens, latency, retries, errors, tool calls, estimated cost — taki bina guess ke pata chale kya chal raha hai.

WHEN

Pehle production deploy se hi. Debug karte waqt nahi — tab tak data bacha hi nahi hoga, aur rona bekaar hai.

WHERE

Dashboards, alerts, cost reviews, postmortems — aur roz ke debugging jab "kyun slow / kyun mehnga / kyun galat" poocha jaata hai.

Funda: jitna observable, utna debuggable. "Kal theek chal raha tha" wali debugging sirf print se nahi chalti — structured records chahiye jo filter aur correlate ho saken.

Har request par kya record karein

import time, logging

log = logging.getLogger("llm")

def tracked_call(model, messages):
    t0 = time.perf_counter()
    resp = client.chat.completions.create(model=model, messages=messages)
    dt = time.perf_counter() - t0

    usage = resp.usage
    record = {
        "model": resp.model,
        "input_tokens": usage.prompt_tokens,
        "output_tokens": usage.completion_tokens,
        "latency_s": round(dt, 2),
        "finish_reason": resp.choices[0].finish_reason,
        "retries": 0,                  # retry wrapper se aayega
        "error": None,
        "est_cost": cost_of(resp.model, usage),   # apna rate table (config)
    }
    log.info("llm_call", extra=record)
    return resp, record

Minimum set: model, input/output tokens, latency, retries, error type, finish_reason, tool calls — yeh har request par.
Cost estimate ke liye apna rate table config mein rakho — rates badalte rehte hain, code mein hardcode mat karo.
Token counts usage se aate hain (estimate se behtar); estimate tab jab usage na mile (kuch proxies/streams mein).
Har record ko request_id se jodo — user complaint aaye toh seedha wahi request dhoondh lo.

Print mat: logging module + structured fields (JSON lines) use karo. Plain text grep karne se JSON filter karne mein farak hota hai, aur log ship karne mein bhi wahi structure chahiye.

Interactions as traces — failing step dhoondo

trace_id = "req-9f2c"
spans = [
  {"step": "extract_fields",    "model": "small", "tokens": 410,  "ms": 620,  "ok": true},
  {"step": "route",             "model": "small", "tokens": 35,   "ms": 210,  "ok": true},
  {"step": "tool:lookup_order", "tokens": 0,      "ms": 480,  "ok": false, "error": "timeout"},
  {"step": "draft_reply",       "model": "large", "tokens": 1890, "ms": 3100, "ok": true},
]
# Teen sawal: kahan fail? kahan tokens? kahan time?

Multi-step interaction = trace, ek API call nahi. Har step ek span: naam, status, tokens, duration, error.
Failing step = ok: false wala span. Cost driver = sabse zyada tokens wala span. Slow step = sabse zyada ms wala span.
Tool selection bhi span hai — kaunsa tool chuna, kitni der lagi, kya result aaya.
Trace id har log line mein — teen alag files ke logs ko jodne ka yehi ek zariya hai.

Debug question: Jab bhi "ye galat kyun" mile, teen sawal poocho: kaunsa step fail, kaunse step ne sabse zyada tokens khaaye, aur time kahan ruka. Trace teeno ka jawab deta hai.

Callbacks aur debuggable logging

from langchain_core.callbacks import StdOutCallbackHandler

chain.invoke({"doc": text}, config={
    "callbacks": [StdOutCallbackHandler()]
})

# Production mein: on_llm_start / on_llm_end / on_tool_start hooks par
# apna logger lagao — har model aur tool call automatically span ban jaayega

Callbacks har run ke start/end/tools par fire hote hain — apna logging sink ek jagah hook karo, saare chains covered.
Aisa karne se scattered print() gayab — ek hi destination, jahan JSON lines ko kisi bhi tool se dekha ja sake.
Retries aur fallbacks ko bhi wahi sink jodo — reliability ka health isi se dikhta hai (M2.9 ke metrics).
Logs mein user text redact ya mask karo — poora prompt har line mein daalna PII leak hai.

Balance: Metadata (model, tokens, latency, step) hamesha log karo; payload (prompt ka text) sirf zaroorat par aur chhota/masked. Debug aur privacy dono sambhali rehti hain.

Exercise

Question: Ek LLM call ko isolated event na samajh kar, poore multi-step run ko kis roop mein dekhte hain? (ek word)

Question: Input aur output tokens response ke kis field se aate hain?

Question: LangChain mein har step par hook lagane wali cheez kya kehlati hai? (ek word)

Question: Cost estimate ke liye rate table code mein hardcode karni chahiye ya config mein rakhni chahiye?

Common mistakes

print() se debugging: — production mein print gayab, level filter nahi, request_id correlate nahi. Structured logging + request_id lo.
Sirf final answer log karna: — beech ka step, tokens, retries nahi dikhte. Har span ka record rakho, tabhi pata chalega kya bigda.
Rate table code mein hardcode karna: — rates badalte hain, estimate galat aata hai. Config/table mein versioned rakho, change review ke saath ho.
Poora prompt har log line mein daal dena: — user data (PII) log systems mein phail jaata hai. Metadata hamesha, payload masked aur limited.

Desi example: Dukaan ka roz ka khata

Dukaan ka khata roz likha jaata hai — tabhi mahine ke end mein pata chalta hai kahan paisa gaya. LLM app ka logging wahi khata hai, aur spans har kaam ki entry hain:

Khata entry (log line):
time | kaam       | kharcha (tokens) | deri (ms) | result
10:01 | extract    | 410 tok           | 620ms     | ok
10:01 | draft      | 1890 tok          | 3100ms    | ok     <- yahan bill phat raha hai
10:02 | lookup     | 0 tok             | 480ms     | FAIL   <- yahan kaam atka

Bina khate ke: "pata nahi kahan kharch hua" — wahi print() debugging hai.

Interview angle — yahan se sawaal aate hain

Q: Har request mein kya record karna chahiye? — Model, input/output tokens, latency, retries, errors, finish_reason, tool calls, est. cost, request_id.
Q: Trace aur single log line mein farq? — Trace ek workflow ki poori chain of spans; single call ka record. Debugging trace se hoti hai.
Q: Sabse zyada tokens kis step par lage — kaise pata chalega? — Span-level token counts se; wohi cost driver hota hai.
Q: LangChain steps ko ek trace mein kaise jodoge? — Callbacks se — har model/tool call apne aap span ban jaata hai, sink tumhara ek hi hai.

Lesson complete?

Observability ka framework tayyar. Agla lesson — Other Orchestration Frameworks — mein dekhenge LangChain ke alawa aur kya options hain.