Lesson 121 — Advanced

Capstone: LLMOps Project

Ab tak sab kuch alag-alag tha — ab ek project mein. Is lesson mein hum apne RAG application aur agent ko poore LLMOps loop se guzaraate hain: config aur pins se shuru karke, eval gate aur guardrails se hote hue, container stack aur CI/CD se deploy karke, aur monitoring + incident drill se finish karenge.

Capstone mein kya banaoge

Capstone ka success criteria simple hai: koi bhi machine pe clone karke ek command se stack up, aur har change ek gated release se guzarta hai. "Mere laptop pe chala" ka exit ho jaana hi capstone hai. L4 aur L5 ke apps hi base hain — naya banana zaroori nahi.

Release manifest — ek source of truth

release-manifest.yaml (repo mein, versioned):

app:
  rag:
    model: provider/model@version
    prompt: rag/system-prompt@v4
    retrieval: { chunk_size: 400, top_k: 8, rerank: true }
  agent:
    model: provider/model@version
    tools: [search_kb, fetch_doc, create_ticket]
    limits: { max_steps: 8, max_spend_usd: 0.05 }
eval:
  dataset: evals/v3          # git tag
  policy: gates/policy.yaml  # thresholds
deploy:
  images: { backend: tag, frontend: tag }
  stack: compose.yaml

# manifest badla = release hui — CI isi se check karta hai

• Manifest tumhare L10.1 wale "release = sab kuch" ka concrete roop hai — ek file mein saare behaviour components.
• CI isi file ko diff karta hai — manifest change toh eval gate mandatory.
• Config loader isi se banao — app start pe manifest padhta hai, scattered settings nahi.
• Release history = manifest ke git commits — "kya kab badla" timeline se dikhta hai.

Build order — ek sprint plan

sprint 1 (foundation):
  [ ] config + manifest + pinned deps + request tracing
  [ ] baseline experiment run (MLflow/Langfuse)
sprint 2 (quality):
  [ ] eval suite -> DeepEval tests + thresholds policy
  [ ] CI: tests -> eval gate -> image build + tag push
sprint 3 (runtime):
  [ ] guardrails: schema+reask, injection/PII scan, agent caps
  [ ] FastAPI + SSE + health/readiness + thin frontend
sprint 4 (ship & see):
  [ ] compose stack + secrets via env
  [ ] deploy (staging auto + prod approval) + ALB timeouts
  [ ] dashboards + alerts + trace links
  [ ] runbooks + ek game-day drill

• Har sprint ke end mein demo-able cheez ho — foundation ke baad bhi "gate fail karke dikhao" mumkin hai.
• Sprint 4 ke game-day drill mein se ek incident chuno (provider outage) aur fallback chalao — runbook ka pehla test.
• Har checklist item acceptance criteria ke saath likho: "clean machine pe ek command se up" jaisa, vague nahi.
• Jo skip hua uske liye reason — capstone mein "kyun nahi" ka jawab bhi part of work hai.

Demo checklist — kya dikhana hai

• Streaming chat — token by token, aur ek guardrail block bhi dikhao (scope check).
• Prompt rollout — staging v5 ko eval report ke saath promote karo, phir label se rollback bhi.
• Gate red karo — jaan-boojh kar kharab prompt push karo, PR red report ke saath dikhega.
• Clean machine start — clone + ek command; streaming har layer se guzarti hui.
• Dashboard tour — p95, tokens, spend; phir ek trace kholo prompt version ke saath.
• Incident drill — provider outage simulate, fallback switch, time bhi note karo.

Demo ka order hi story hai: pehle quality (eval), phir safety (guardrails), phir operability (deploy, dashboard, incident). Ye teen pillar interview mein bhi hote hain — interviewer ko ek structured walkthrough do, feature list nahi.

Interview story — 3 minute version

3-minute narrative (template):

1. context  : do apps the (RAG + agent), production mein the
2. problem  : bina system ke har change gamble tha
3. actions  : manifest+tracking -> eval gate -> guardrails
              -> container stack -> CI/CD -> dashboards
4. proof    : ek regression gate ne pakda; ek incident drill
              mein fallback ne bachaya (numbers apne daalo)
5. learning : sabse bada lever kaunsa nikla (eval gate ya
              prompt registry?) aur kyun

# har point ek artifact se linked — screenshot, log, report

• Numbers sirf apne measured daalo — "eval gate ne 2 bad prompts block kiye, ek tha jo slip ho gaya (kyun?)" zyada powerful hai than inflated claims.
• Trade-off wale sawaal ke paas ready raho: fast subset kyun, threshold kaise chune, guardrail latency kaise budget ki.
• Failure story bhi ready rakho — ek cheez jo plan se alag chali, aur usse kya seekha. Yahi seniority ka signal hai.
• Repo ka structure ek slide pe — modules ke naam, aur har module kis problem pe tha.

Exercise

Common mistakes

• Naye apps banaane mein lag jaana: Base L4/L5 ke apps hain — LLMOps loop banana goal hai, features nahi. Fix: scope chhota, loop poora.
• Loop ka koi step skip karna: Gate bana par dashboard nahi — incident ke waqt system phir bolta nahi. Fix: sprint plan ke har item ka acceptance criterion.
• Demo sirf happy path: Sab pass dikhaya — par kisi ne gate fail nahi dekha. Fix: jaan-boojh kar ek bad prompt, ek rollback, ek incident drill dikhao.
• Inflated ya borrowed numbers: Interview mein detail maanga jaata hai — borrowed claim turant pakdi jaati hai. Fix: apne measured numbers, apne trade-offs, honest gaps.

menu card      -> release manifest
quality check  -> eval gate in CI
bouncer        -> runtime guardrails
kitchen kit    -> containers + compose
delivery chain -> CI/CD + deploy
monthly audit  -> dashboards + traces
fire drill     -> incident game day

# yehi restaurant = production-ready LLM app