Lesson 62 — Intermediate

Capstone — Deploy an Advanced RAG Application

Aakhri capstone: tumhara hybrid search engine ab ek full RAG service banega — fusion + rerank + grounded generation + citations + abstain + eval + API. Yehi wo project hai jo resume pe aur interview mein tumhara flagship banega — L4 ke saare layers ek saath.

Kya banayenge

Scope: ek corpus, ek endpoint, ek eval set. Lekin quality guarantees real hon: har answer traceable (citations), out-of-scope pe abstain, aur har regression automatically pakri jaaye. Yehi "advanced" ka matlab hai — features nahi, guarantees.

Step 1 — Service skeleton

from fastapi import FastAPI
app = FastAPI()

@app.post("/ask")
def ask(q: str, k: int = 5):
    out = pipeline.answer(q, k=k)     # saara logic pipeline module mein
    return {"answer": out.text, "citations": out.citations,
            "abstained": out.abstained, "latency_ms": out.ms}

@app.get("/health")
def health():
    return {"index_size": index.count(), "model": MODEL_NAME}

• Pipeline ko app se alag module rakho — eval harness bhi wahi pipeline call kare, API wrapper sirf transport hai.
• Config (model names, thresholds, k) environment se — code mein hardcode nahi; health endpoint config dikhaye.
• Structured log: query, hit ids + scores, answer, abstained, latency — debugging ka poora record.
• Index rebuild job alag script mein — incremental (M4.1 ka fingerprint) + full rebuild ka raasta dono.

Step 2 — Advanced pipeline

def answer(query, k=5):
    variants = rewrite(query)                       # M4.11
    ranked = rrf([dense_rank(v, 20) for v in variants]
                 + [bm25_rank(v, 20) for v in variants])  # M4.6
    top = rerank(query, ranked, top=k)              # M4.10
    if not top or top[0]["score"] < MIN_SCORE:
        return Out(ABSTAIN, [], True)               # M4.12
    text, cits = generate(query, top)               # M4.9 + citations
    return Out(text, cits, False)

• Layered pipeline: rewrite → fusion (dense+sparse) → rerank → threshold → grounded generate → cite.
• Har layer ka apna on/off flag config mein — A/B ke liye, aur jab koi layer issue de toh turant disable.
• Citations chunk id se map, answer parse ke baad — format check + support check (M4.12).
• Abstain dono level pe: retrieval threshold aur prompt rule — dono ke rates log mein.

Step 3 — Eval, regression aur ship

EVAL = [
    ("refund policy damaged item", {"must_contain": "7 din", "gold": "doc-12#0"}),
    ("INV-4821 status",                     {"gold": "doc-7#2"}),
    ("kya kal cricket score batao",         {"abstain": True}),   # out of scope
]

def run_eval():
    hits = abst_ok = ans_ok = 0
    for q, exp in EVAL:
        out = pipeline.answer(q)
        if "gold" in exp:
            hits += exp["gold"] in [c["id"] for c in out.citations]
        if exp.get("abstain"):
            abst_ok += out.abstained
        if "must_contain" in exp:
            ans_ok += exp["must_contain"] in out.text
    print("retrieval", hits, "| abstain", abst_ok, "| answer", ans_ok)

• Eval set git mein versioned — har pipeline change ke baad wahi chalta hai (regression harness).
• Tin checkpoints: retrieval (gold chunk mila), abstain (out-of-scope pe ruk gaya), answer (key fact present).
• Latency budget bhi ek check: p95 latency limit se upar toh fail — quality aur speed dono guarantees.
• Ship checklist: config docs, index rebuild script, log schema, eval green, README setup steps.

Exercise

Common mistakes

• Bina eval set ke deploy: Demo chal gaya, par quality ka koi proof nahi — pehla regression pata hi nahi chalega. Fix: gold + abstain queries pehle likho, CI mein chalao.
• Threshold aur model names hardcode: Corpus badla, model badla — code edit karna pada. Fix: sab config/environment se, health endpoint se config verify.
• Logs mein sirf answer: Galat answer aaya, par retrieval theek thi ya nahi — pata hi nahi. Fix: query + hit ids + scores + latency hamesha log, structured format mein.
• Index update job na rakhna: Docs update hoti rehti hain, index stale — answers purane. Fix: incremental update job (fingerprint-based) scheduled, full rebuild ka raasta bhi.

guest (query)
  -> 2 guards poochhte hain (fusion: dense + sparse)
  -> supervisor filter (rerank top-k)
  -> register se page number (citations)
  -> "nahi pata" bhi ek jawab (abstain)
everything logged -> manager ka eval set = kal ka quality check