Lesson 111 — Advanced

What LLMOps Adds

Ek LLM application tab tak toy project hai jab tak usse chalane ka koi system na ho. Is lesson mein hum LLMOps kya hai, classic MLOps se kahan alag hai, ek release mein actually kya jaata hai, aur RAG versus agent ko operate karna alag kyun hai — sab mental-model level pe dekhenge.

⏱ 10 min• Advanced• Basic computer usage

LLMOps kya hai

Kya hai?

LLMOps matlab LLM applications ko ship, operate aur improve karne ki practice — prompts, models, retrieval aur tools ko ek system ki tarah version karna, test karna aur nazar rakhna.

Kab kaam aata hai?

Jab pehla user aa jaaye ya team badh jaaye — jab "kal wala jawab aaj jaisa kyun nahi" ka kisi ke paas jawab na ho. Notebook se nikal kar production jaate hi yahi layer chahiye.

Kahan milta hai?

Tumhare repo, prompt registry, eval suite, CI pipeline, dashboards aur runbooks mein — koi single tool nahi, system of record hai ye.

MLOps ka core sawaal tha "model kaise train aur serve karein". LLMOps mein sawaal badal jaata hai: model toh kisi aur ne train kiya hai — ab uske upar bana product kaise safe badale? Tumhara application code chhota ho sakta hai, par behaviour badalne wale inputs (prompt, data, model endpoint) bahut hain.

Yaad rakho: LLMOps ka matlab model training nahi hai. Application LLMOps = existing models ke upar product chalana; model LLMOps = model khud train aur host karna. Zyada product teams pehli wali karti hain — is level mein bhi wahi hum operate karenge.

Classic MLOps se kya alag hai

Non-determinism: classic ML model same input pe same output deta hai; LLM temperature ke saath alag-alag likh sakta hai — testing ka definition hi badal jaata hai.
Prompt hi code hai: ek line prompt badli, behaviour badla — aur git diff mein kuch nahi dikha. Isliye prompts ko bhi versioned hona padta hai.
Provider-side updates: tumhara model name same hai par endpoint peeche upgrade ho sakta hai bina tumhare deploy ke — "release" kabhi hua hi nahi.
Har request ka kharcha: per-instance compute nahi, per-request tokens — cost knob latency aur quality ke saath saath chalta hai.
Output khud evaluate karna padta hai: ground truth aksar nahi hoti, judge model ya rubric chahiye — single accuracy number kaafi nahi chalta.

Common confusion: "humne MLOps kiya hai, LLMOps bhi wahi hoga." Concepts — versioning, CI, monitoring — same hain. Farq sirf itna hai ki behaviour ke itne saare non-code sources hain ki unka record rakhna hi asli kaam ban jaata hai.

Ek release mein kya jaata hai

RELEASE = {
  "code"              : backend + frontend ka source,
  "prompts"           : system prompt, few-shots, tool descriptions,
  "model"             : provider + model name + version pin,
  "retrieval config"  : chunk size, top-k, reranker on/off,
  "index contents"    : vector index ka data snapshot,
  "tool definitions"  : agent ke tools ka schema,
}
# in mein se KISI ka bhi badalna = naya release
# in sab ka record hai? — yahi LLMOps ka pehla sawaal

Index contents sabse chhupa hua component hai — naya doc index mein gaya, behaviour badal gaya, koi deploy nahi hua.
Model pin na ho toh aaj ka build kal reproduce hi nahi hoga — same code, alag jawab.
Tool definitions bhi prompt hain — agent ka tool description badal diya toh tool-selection behaviour badal jaata hai.

Rule: Release = un saari cheezon ka set jo badalne pe behaviour badal sakti hain. Agar koi cheez bina record ke change ho sakti hai, wo tumhare kal ke mystery bug ki guarantee hai.

RAG operate karna vs agent operate karna

RAG ki failures              Agent ki failures
----------------             ----------------
stale index (data purana)    runaway loop (tool baar baar)
retrieval drift              tool error / partial failure
purani ya galat docs         per-request spend explode
embedding model mismatch     steps badhte jaate hain

RAG:  quality retrieval pe depend karti hai
Agent: quality poori trajectory pe — har step ek decision hai

RAG ka prime suspicion hamesha data hai: index kab update hua tha, kisne daala, kaunsa version chal raha hai.
Agent ka prime suspicion loop aur budget hai: steps aur spend cap hain ya nahi, tool timeouts set hain ya nahi.
Dono ka common missing piece: per-request record ki aaj kya hua — isi liye tracing pehle din se chahiye.

Decision rule: RAG mein sawaal karo "kya context sahi tha?"; agent mein "kya raasta sahi tha?". Eval aur monitoring dono ke alag honge — aage ke lessons mein dono banayenge.

Is level ka lifecycle — track se improve tak

track    -> har request ka record (trace, versions, cost)
evaluate -> eval set pe quality measure + CI gate
guard    -> input/output/agent checks runtime pe
serve    -> API + streaming + container
ship     -> CI/CD: test -> gate -> image -> registry
deploy   -> cloud pe health-checked release + scaling
monitor  -> live signals: errors, latency, tokens, spend
improve  -> production failures = naye eval cases -> track

# ek closed loop — improve hamesha track par wapas aata hai

Agle lessons isi order mein aage badhenge. Har step agle pe khada hai — bina tracking ke evaluate andhera hai, bina eval ke deploy gamble hai. Isliye order matter karta hai, fancy tools nahi.

Practical tip: Team ke saath ek baar ye aat words likho aur tick karo ki kaunsa step abhi bhi manual, ad-hoc ya missing hai. Poora L10 isi checklist ko complete karne ke liye hai.

Exercise

Question: Ek LLM application ka wo non-code component jo badalne se behaviour badal jaata hai aur aksar versioned nahi hota — prompt, compiler ya HTTP server?

Question: Application LLMOps aur model LLMOps mein zyada product teams kaunsa karti hain? (application / model)

Question: RAG application ka sabse aam stale component kaunsa hai — vector index ya load balancer?

Question: Is level ka lifecycle ka pehla step kaunsa hai — track, deploy ya fine-tune?

Common mistakes

Sirf code version karna: Prompts, model pin aur index contents git ke bahar reh gaye toh "same build" kabhi same nahi hoga. Fix: release = code + prompts + model + config + index, sab ka ek manifest.
Provider endpoint ko stable maanna: Provider kabhi bhi model update kar sakta hai, tumhara behaviour chupke se badal jaayega. Fix: model version pin karo aur provider announcement pe eval suite dobara chalao.
Prompt change ko chhota change samajhna: Ek line edit ne poora product behaviour badal diya par koi gate nahi hua. Fix: prompt change = release — code change ke bilkul same gate se guzrega.

Desi example: Restaurant ki kitchen vs LLMOps

Restaurant mein dish consistent chahiye toh recipe (prompt), chef ka standard (model), ingredient supplier (data/index) aur order padhne wala (code) — sab fixed aur recorded hote hain. Sirf waiter badalne se dish nahi badalti:

recipe badli (prompt edit)     -> naya dish version
supplier badal gaya (index)   -> wahi order, alag taste
chef upgrade (model pin)      -> taste change bina notice ke

isliye: har cheez ka record + taste-test (eval) before serving

Interview angle — yahan se sawaal aate hain

Q: LLMOps aur MLOps ka farq? — MLOps training/serving pe; LLMOps existing model ke upar product: prompts, retrieval, tools version karna, non-determinism handle karna.
Q: LLM system operate karna mushkil kyun hai? — Non-deterministic output, prompt se behaviour change, provider-side updates, per-request cost.
Q: Ek LLM release mein kya-kyaa hota hai? — Code, prompts, model version, retrieval config, index contents, tool definitions.
Q: RAG aur agent ki operations alag kyun? — RAG mein staleness/retrieval drift, agent mein runaway loops aur tool failures — signals hi alag hain.

Lesson complete?

LLMOps ka mental model ready. Agla lesson — Preparing Your Apps for Production — mein in saari cheezon ko explicit config, pins aur request-traceable structure mein badlenge.