Lesson 161 — Advanced

Document Q&A Platform

Yeh project L4 (RAG) ko ek deployable product mein badalta hai: user PDF poochta hai aur jawab sources ke saath aata hai. Is lesson mein architecture se lekar retrieval eval tak ka poora build plan hai.

Project kya hai

Engineering ka asli kaam yahan scope control hai: pehle ek document type (jaise PDF manuals), ek language, ek deployment — phir format badhao. Har extra format (scanned pages, tables, mixed languages) apna naya failure mode laata hai.

Architecture — pipeline ke 7 stages

Pipeline ko alag-alag stages mein todo — har stage ka apna input/output aur apna test hota hai:

• Ingest: PDF/DOCX padho, metadata (filename, page number) ke saath text nikalo.
• Chunk: text ko400–800 token wale pieces todo,20–15% overlap ke saath (L4.2).
• Embed: har chunk ka vector banao (L4.3) — batch mein, cache ke saath.
• Store: vectors + text + metadata kisi vector store mein (pgvector, Chroma, Qdrant).
• Retrieve: sawaal embed karke top-k search — hybrid (BM25 + dense) best starting point (L4.6).
• Answer: retrieved context + sawaal se LLM se jawab banao, citation ids ke saath (L4.12).
• Eval: ek fixed question set se har change ka effect measure karo (neeche detail).

FastAPI skeleton

from fastapi import FastAPI, UploadFile
from pydantic import BaseModel

app = FastAPI()

class AskRequest(BaseModel):
    question: str
    top_k: int = 5

@app.post("/ingest")
async def ingest(file: UploadFile):
    doc = parse_pdf(await file.read())          # stage 1-2: parse + chunk
    store.upsert(embed_all(doc.chunks))         # stage 3-4: embed + store
    return {"doc_id": doc.id, "chunks": len(doc.chunks)}

@app.post("/ask")
async def ask(req: AskRequest):
    hits = store.search(embed_text(req.question), k=req.top_k)  # stage 5
    answer, cites = answer_with_citations(req.question, hits)   # stage 6
    return {"answer": answer, "sources": [h.cite() for h in hits]}

Retrieval eval — pehla deliverable

LLM answers har baar alag hote hain, isliye pehle retrieval evaluate karo (deterministic hai): kya relevant chunk upar aaya? Phir answer quality ka spot-check karo.

# Chhota sa golden set — har improvement ko isi se maano
eval_set = [
    {"q": "Refund kitne din mein milta hai?", "relevant_chunks": ["policy-7", "policy-8"]},
    {"q": "Escalation contact kaun hai?", "relevant_chunks": ["contact-2"]},
]

def hit_at_1(store, item):
    top = store.search(item["q"], k=1)
    return int(top and top[0].chunk_id in item["relevant_chunks"])

score = sum(hit_at_1(store, it) for it in eval_set) / len(eval_set)
print(f"hit@1 = {score:.2f}")   # har chunk-size, k aur hybrid toggle par yahi doharao

• Hit rate / precision@k: top-k mein kitne relevant the — retrieval ka core number.
• Context recall: jawab ke liye zaroori sabhi facts context mein aaye ya nahi.
• Faithfulness spot-check: jawab ke claims context se uthe ya model ne jod diye — 10–20 samples par manual/LLM judge (L7.8).

Ship checklist

• Dockerfile + README: ek command se chalne chahiye (L0.7 ka Docker skill yahan).
• Secrets env vars mein, .env.example repo mein — key kabhi commit nahi (L0.4).
• Rate limit + max file size + accepted formats explicitly check karo (L8.6).
• Errors structured: parse fail, empty doc, retrieval empty — har ek ka alag message.
• README: live demo link, architecture diagram (ASCII/mermaid), eval numbers (apne, real), limits likhi hui.

Exercise

Common mistakes

• Bina retrieval eval ke deploy karna: Sirf "acche answers aa rahe hain" se kaam nahi chalta —3–4 random tests mein luck chal jaata hai. Golden set banao, har change par doharao.
• Chunk size andaraz se chunna: Bohot chhote chunks = context toota; bohot bade = irrelevant shor. Eval hit-rate ke saath400/600/800 compare karo, guess nahi.
• Scanned PDF ko text PDF samajh lena: Image-based pages se extraction khaali string deta hai — ingestion ke waqt text density check karo aur zaroorat par OCR lagao.

sawaal -> index search -> passage uthao -> line-by-line padho
        -> jawab + page number   (bina index, andheri mein dhoondhna)