Lesson 169 — Advanced

Production RAG interview scenarios

Demo RAG ek weekend mein ban jaata hai; interview uske baad shuru hota hai — messy enterprise PDFs, hazaaron concurrent users, badalta hua corpus, aur ek din achanak giri hui quality. Is lesson mein tum ek production RAG ko services mein tuta hua dekhoge (ingestion, embedding, retrieval, inference, orchestration, storage), stateless orchestration aur stateful stores ka farq seekhoge, queues/fallbacks/observability/IAM wala infra layer padhoge, aur wo systematic debugging frame jo "quality ek hafte baad gir gayi" jaisa sawaal solve karta hai.

⏱ 20 min• Advanced• Interview strategy & role positioning

Demo se production tak — har arrow pe ek sawaal

Kya hai?

Production RAG = ingestion + retrieval + generation + evals + ops, jismein corpus badalta rehta hai aur users hazaar hain. Demo keval ek retriever → LLM pipeline hai; production mein har stage ka apna failure mode, apna metric aur apna rollback hota hai.

Kab kaam aata hai?

Jab documents jaate-vaarte hon, team ka access control ho, SLA ya audit requirement ho. Demo chahiye toh ek script kaafi; production chahiye toh har stage ko service ki tarah sochna padega.

Kahan milta hai?

Support assistants, policy Q&A, document intelligence, compliance search — jahan documents regularly badalte hain aur jawab ka source dikhna zaroori hai.

Interviewer ka asli test ye hai: kya tum har arrow pe poochh sakte ho "ye down ho jaaye toh?" Pipeline mein har arrow ek contract hai — freshness, availability, quality, cost. Demo mein ye arrows invisible hain, production mein yahin se incidents aate hain.

Rule of thumb: Har stage ka teen cheezein likho: metric, fallback, owner. Jis stage ka metric nahi hai, wo production mein chupke se mar jaati hai — aur interview mein wahi stage sabse pehle poochhi jaati hai.

Ingestion — messy enterprise data ka asli kaam

raw docs (PDF/HTML/scans, roz badalta hai)
  -> parse + layout detect (tables, figures, columns, headers)
  -> clean + dedupe (versions, boilerplate, email signatures)
  -> chunk (structure-aware: heading path + table + row context)
  -> enrich (metadata: doc id, version, section, ACL, date)
  -> embed + index (batch, retries, idempotent re-runs)
  -> quality gate (parse coverage, chunk sample audit, index diff)

har doc: content_hash + version -> incremental re-ingest (sirf badle hue docs)

Messy PDFs: nested tables, multi-column layouts, scanned pages. Parser har page pe confidence de, low-confidence pages ek human review queue mein jaayein — andha OCR sab kuch theek nahi kar sakta.
Diagrams aur figures: image se seedha text kabhi nahi. Ya toh caption/alt-text uthao, ya multimodal model lagao, ya expectation set karo ki "ye sawaal figure-text pe answer hoga".
Continuously changing docs: content hash + versioned metadata rakho, re-ingest sirf badle hue docs ka — warna har night job poora corpus dobara embed karega aur budget udegaa.
Chunking mein structure carry karo: heading path har chunk ke saath metadata mein — warna ek table ka row akele chunk ban gaya aur jawab ka context chhoot gaya.

Dukh ka 80%: Production RAG ka zyada-tar dukh ingestion mein hai, retrieval mein nahi. Parser, chunker, ACL metadata aur incremental re-ingest — yehi cheezein interview mein "messy enterprise data" wale sawaal ka asli jawab hain.

Architecture ko services mein todo — stateless vs stateful

stateless services (scale independently, retry-safe):
  api_gateway:      auth, rate limit, request shaping
  orchestrator:     query plan: retrieve -> rerank -> generate -> verify
  retrieval_svc:    vector + keyword + filters (metadata, ACL)
  inference_svc:    LLM calls (provider ya self-hosted), batching, timeouts
  ingestion_svc:    parse -> chunk -> embed -> index (queue consumer)

stateful stores (services se alag, replaceable):
  object_store:     raw docs + parsed artifacts
  vector_index:     embeddings + metadata (+ embedding version)
  search_index:     keyword/hybrid
  relational_graph: ACL, doc registry, doc relations
  cache:            hot queries, embeddings (TTL ke saath)
  queue:            ingestion jobs, async answers

Stateless services ka faida: scale out, rolling deploy, retry-safe — kisi pod ki memory mein kabhi bhi state nahi, isliye mar diya toh agla uth kar chal jaata hai.
State stores mein jaata hai: embeddings vector index mein, files object store mein, ACL relational/graph mein. Service disposable hai, data precious hai.
ACL kabhi embedding mein mat chhupao: retrieval ke baad (ya pre-) filter lage, aur jawab ke source document pe bhi access check — delete hua doc ka chunk index mein reh jaata hai warna.
Embedding service alag kyun: embedding model badalne pe poora re-embed chahiye. Isliye har index entry mein embedding version likho — naapne ka zariya bhi versioned hona chahiye.

Stateless ka matlab retry-safe hai: Agar ek request dobara chal sakti hai bina kisi side-effect ke, toh tum retries, load balancing aur autoscaling free mein le sakte ho. State agar service mein hi ho toh ye teeno band — interview mein yahi connection banana hai.

Production infrastructure — queues se CI/CD tak

queues        -> ingestion + async answers; backpressure absorb, burst ko level
fallbacks     -> LLM down -> alternate provider -> smaller model -> cached answer -> honest error
observability -> per-stage latency, retrieval hit-rate, token spend, error rate; SLA per stage
networking    -> private endpoints, egress rules (raw docs bahar nahi jaate)
IAM           -> service identities, least privilege, per-doc ACL enforcement
secrets       -> API keys vault mein, rotation bina deploy ke
encryption    -> at rest + in transit; PII ka retention policy bhi likho
CI/CD         -> eval gate: retrieval regression toh deploy block, quality theatre nahi

Queues: ingestion jobs kabhi request path pe nahi. Burst aaye toh queue level kare — user ko latency na dikhe.
Fallbacks: har external dependency ka plan B. Provider down -> alternate -> chhota model -> cached/empty answer — charo stages pe alag-alag degradation.
Observability aur SLA: per-stage SLO rakho (retrieval ka alag, generation ka alag) — aggregate latency se stage kharab chhup jaati hai.
Networking, IAM, secrets, encryption: docs kahan ja rahe hain, kaun kya kar sakta hai, keys kahan padi hain — enterprise interviewer yahin se security round shuru karta hai.
CI/CD: har deploy se pehle golden query set ka eval gate — retrieval regression detect ho toh deploy ruk jaaye, prod mein quality drop hone se pehle.

Infrastructure bhi answer hai: "Bas RAG laga denge" wala jawab sirf demo ke liye hai. Queues, fallbacks, SLOs, IAM aur eval gates — jis candidate ne sirf pipeline banayi hai aur jisne wo chalayi hai, farq yahin dikhta hai.

"Quality ek hafte baad gir gayi" — debugging ka frame

frame: stage by stage isolate karo, guess mat karo
1. timeline  -> kab se gira? us hafte kya deploy hua, kaunse docs aaye
2. data      -> naye docs, parser/chunker version, ACL changes, purge jobs
3. retrieval -> hit-rate, chunk freshness, index diff, embedding version drift
4. generation-> prompt/model/provider update? temperature/config drift?
5. infra     -> timeouts, queue backlog, cache serving stale, rate limits (429s)

tool: fixed golden query set chalao (pehle se stored expected answers)
  retrieval hi kharab -> dosh data/index mein, model ko nahi
  retrieval theek, phir bhi jawab kharab -> generation/config change

Pehla kaam timeline nikalna hai: incident ka "git blame" — kab se gira, us window mein kya change hua. Do changes ek saath ho toh ek kar ke chalao.
Retrieval ko alag naapo: fixed query set se hit-rate check karo. Retrieval kharab hai toh generation ko dosh dena bewakoofi hai.
Generation ko alag naapo: same retrieved context, purana known-good config chala kar dekho — farq dikhe toh model/prompt/config culprit.
"Quality" ko metric banao: bina golden set ke debugging guess hai. Rolling eval score, retrieval hit-rate aur empty-retrieval rate — teeno ka graph inciden se pehle se chal raha hona chahiye.

Investigation ka order: Data (kya badla) → retrieval (context mil raha hai) → generation (context se jawab ban raha hai) → infra (dono slow/throttled toh nahi). Ye order isliye hai kyunki upar ka stage neeche ka poison kar deta hai — isolate karke hi pata chalega kaun asli culprit hai.

Exercise

Question: Har document ka duplicate check aur versioning kis identifier se hoti hai? (do shabd)

Question: Rolling deploy ko retry-safe banane ke liye services kaisi honi chahiye? (ek shabd)

Question: "Quality dropped after a week" — debugging mein sabse pehle kaunsa layer isolate karte hain?

Question: ACL ko embedding mein chhupane ke bajaye kaunsa layer enforce karte hain? (metadata filter / reranker)

Question: Embedding model badalne poore corpus ka kya karna padta hai?

Common mistakes

Demo wala RAG seedha scale kar lena: Ek script mein sab — embedding bhi request path pe, inference bhi. 100 users aate hi pipeline choke. Fix: ingestion async (queue), serving stateless, har stage apna service.
ACL embedding mein chhupana: Document delete hua par purana chunk index mein pada — data leak. Fix: ACL ko metadata filter mein rakho aur delete/purge ka runbook likho jo index tak pahunche.
Quality ka koi metric hi nahi: "Lagta hai theek hai" debugging nahi hai, opinion hai. Fix: golden query set + retrieval hit-rate + rolling answer eval — teeno ka dashboard pehle se chalte hon.
Retrieval aur generation ek saath badalna: Do changes ek saath ship kiye, ab farq kis ka hai pata hi nahi. Fix: ek waqt mein ek change, har change pe eval gate, aur version pinning har layer pe.

Desi example: Courier ka hub — parse se delivery tak

Poora RAG ek courier hub jaisa hai: parcel aaya (parse), box mein daala (chunk), address database (index + ACL), dhaakaa laga (queue/backpressure), aur delivery boy jawab pahunchta hai (generation). MIS report (observability) ke bina ek din gumshuda parcel kisi ko dikhta hi nahi:

parcel aaya       -> parse (label padho, low-confidence -> human check)
box bana          -> chunk (structure ke saath)
address database  -> index (metadata + ACL + version)
dhaaka (queue)    -> backpressure jab volume badhe
delivery boy      -> generation (customer tak jawab, fallback ke saath)
MIS report        -> observability (kitne late, kitne gumshuda, kitne damaged)

Interview angle — yahan se sawaal aate hain

Q: Messy PDF pipeline mein kya karte ho? — Parse with layout detect + per-page confidence, structure-aware chunking, metadata enrichment, quality gate; low-confidence pages human queue mein.
Q: Services stateless kyun, stores alag? — Stateless = scale out, rolling deploy, retry-safe. State stores mein (vector, object, graph, cache) taaki service disposable rahe, data zinda.
Q: "Quality ek hafte baad giri" — kya karoge? — Timeline nikalo, stage-by-stage isolate karo (data → retrieval → generation → infra), golden query set se retrieval aur generation alag naapo.
Q: ACL kahan enforce hoga? — Metadata filter (pre/post retrieval) + source pe bhi check; embedding mein kabhi nahi — delete ke baad bhi chunk index mein reh jaata hai.
Q: Corpus roz badalta hai — re-ingest strategy? — Content hash + versioned metadata; sirf badle hue docs ka incremental re-ingest, purge ka runbook, embedding version har entry mein.

Lesson complete?

Production RAG ka frame ready — ab tum demo aur production ke farq ko whiteboard pe defend kar sakte ho. Agla lesson — Agentic AI Interview Scenarios — mein usi production discipline ko agent architectures pe lagayenge.