Learn›AI Engineer›RAG Evals

Lesson 90 — Intermediate

RAG Evals

RAG sabse zyada use hone wala LLM pattern hai, aur iska failure mode sabse chhupne wala bhi — retriever ne sahi chunk nahi nikala toh generator sach bolega bhi toh galat baat. Is bade lesson mein hum RAG ko ek black box na maan kar alag-alag layers pe todoge: retriever alone, generator alone, poora pipeline, user perspective, operational metrics aur safety — har layer ke apne metrics aur apne diagnosis tricks ke saath.

⏱ 50 min• Intermediate• Building an Eval Suite & Regression Testing

RAG eval kyun alag hai

WHAT

RAG eval = har component (retriever, context assembly, generator) aur unke interaction ko alag-alag measure karna. Ek single "answer accha hai" score RAG ke liye kaafi nahi — kyun galat hua, kis layer pe — wahi asli diagnosis hai.

WHEN

Har retrieval change pe (chunking, embedding, k value), har prompt change pe, corpus update pe, aur model switch pe — sab pe wahi 4-layer suite chalta hai. Release gate bhi isi se banta hai.

WHERE

Component tests CI mein fast chalte hain, pipeline tests full flow pe, application tests user scenarios pe, aur safety tests scheduled red-team runs mein. Har layer ka apna dataset hota hai.

RAG mein galat answer ke teen alag gunahgar ho sakte hain: retriever (context hi galat laaya), context assembly (sahi chunks the par jodne mein dikkat), ya generator (context sahi tha phir bhi galat bola). Bina layered eval ke tum hamesha generator ko hi blame karoge — kyunki "answer toh generator ne likha" — aur fix galat jagah lagao ge.

Chaar levels — component se safety tak

LEVEL           KYA TEST HOTA HAI                       KYA DIAGNOSE KARTA HAI
──────────────  ────────────────────────────────────  ──────────────────────────────
1. COMPONENT    retriever alone, generator alone       kaunsa part individually weak hai
2. PIPELINE     retrieve -> context -> generate flow   interaction bugs, root cause kahan
3. APPLICATION  poora app: user scenario + UX + ops    kya user ko actually mil raha hai
4. SAFETY       injection, PII, harmful, poisoning     kya system exploit ho sakta hai

# Rule: pehle component, phir pipeline, phir application, hamesha safety.
# Component hi theek nahi toh pipeline test ka result confusing hoga.

Ye layers ek doosre ko replace nahi karti — har ek ka apna question hai. Component level poochhta hai "kya retriever jaanta hai answer kahan hai", pipeline poochhta hai "kya wo answer generate tak sahi pahuncha", application poochhta hai "kya user ko uski bhasha mein, format mein mila", safety poochhta hai "kya koi is system ko tod sakta hai".

Retriever alone — kya wo sahi dhoondhta hai

Generator ko hata kar sirf retrieval evaluate karo: har golden question ke saath tumhe pata hai kaunse chunks mein answer hai (relevant set). Retrieval ko us relevant set se compare karo — yehi asli retriever score hai, generator ke output se bilkul alag.

def recall_at_k(retrieved, relevant, k=5):
    """Top-k mein se kitne relevant chunks aa gaye."""
    top_k = retrieved[:k]
    hit = len(set(top_k) & set(relevant))
    return hit / len(relevant) if relevant else 1.0

# Missed retrieval vs noisy retrieval — dono alag problems hain
def diagnose_retrieval(retrieved, relevant, k=5):
    top_k = retrieved[:k]
    hits = set(top_k) & set(relevant)
    if not hits:
        return "MISS — relevant chunk hi nahi mila (ya toh corpus mein hai hi nahi,
                ya embedding/chunking kharab hai)"
    noise = [c for c in top_k if c not in relevant]
    if len(noise) >= len(hits):
        return "NOISE — relevant toh mila par 4 irrelevant chunks ke beech daba hai
                (ranking chahiye, aur kam karo ya filter lagao)"
    return "OK"
Rule of thumb: Retriever eval pehle karo — wo fail ho toh generator tuning bilkul bekaar hai. Ek baar retriever solid hua, tabhi generator ke scores ka matlab hai.

Retrieval metrics — contextual recall aur precision

# Contextual Recall — question ke liye zaroori info ka kitna mila
# (ideal answer ke claims ko le kar poochho: har claim context se support hai?)
def contextual_recall(ideal_answer, retrieved_context):
    claims = split_into_claims(ideal_answer)          # LLM se claims nikalwao
    supported = sum(1 for c in claims if any(chunk_supports(ch, c) for ch in retrieved_context))
    return supported / len(claims)

# Contextual Precision — relevant chunks top pe hain ya beech mein dabe hain
def contextual_precision(retrieved, relevant):
    scored, num, den = [], 0.0, 0
    for rank, chunk in enumerate(retrieved, start=1):
        if chunk in relevant:
            num += len(scored) / rank                 # rank weight: upar wala zyada
            scored.append(1)
        den += 1
    return num / den if den else 0.0
# Recall batata hai "sab zaroori mila ya nahi",
# Precision batata hai "bina bhasad ke, sahi cheez upar hai ya nahi".

# Precision@k aur Hit Rate — do chhote companions
# precision@k: top-k mein se kitne chunks sach mein relevant the
def precision_at_k(retrieved, relevant, k=5):
    top_k = retrieved[:k]
    return len(set(top_k) & set(relevant)) / k if k else 0.0

# hit rate: kitni queries ke top-k mein kam se kam 1 relevant chunk mila
def hit_rate(all_queries, gold_sets, k=5):
    hits = sum(1 for r, g in zip(all_queries, gold_sets) if set(r[:k]) & set(g))
    return hits / len(all_queries) if all_queries else 0.0
# recall = kitna zaroori mila | precision = top-k mein kitna bekaar tha
# hit rate = queries mein se kitni kam se kam ek relevant chunk se mili.
Warning: Contextual recall ka judge bhi kabhi-kabhi "support" galat decide karta hai — ek chhote human-labelled set pe judge ka agreement check karo, warna retriever tuning ek faulty compass pe ho rahi hogi.

RAG golden dataset banana

Pro Tip: Dataset banate waqt production query logs se ek batch zaroor mix karo — LLM-generated questions hamesha "clean" hote hain, aur asli user ke queries usse kaafi alag likhi jaati hain (typo, slang, code-switching).

Generator alone — known-good context de kar

Ab ulta karo: manually sahi context pack do (retriever bypass) aur poochho — kya generator us context se faithful, complete aur relevant answer deta hai? Yehi test generator ko isolate karta hai. Yahan milne wale failures = pure generation bugs: context ke baawajood hallucination, missing required facts, ya context chhod ke apni gyaan se answer banana.

def faithfulness(context, answer):
    """Mechanism: answer ke claims split karo, har claim context se verify karo."""
    claims = split_into_claims(answer)               # LLM se structured extraction
    if not claims:
        return 0.0
    supported = 0
    for claim in claims:
        verdict = nli_or_judge(premise=context, hypothesis=claim)
        # verdict: SUPPORTED | CONTRADICTED | NOT_FOUND
        if verdict == "SUPPORTED":
            supported += 1
    return supported / len(claims)

# 4 failure types jo generator-alone test pakadta hai:
# 1. UNFAITHFUL  — context mein na hone par bhi fact add kiya (hallucination)
# 2. CONTRADICT  — context se ulta bol diya
# 3. INCOMPLETE  — required facts mein se kuch chhod diye
# 4. IRRELEVANT  — context sahi tha par sawaal ka jawab nahi diya

RAG Triad aur pipeline-level diagnosis

RAG TRIAD — teen checks jo poore pipeline ko gherte hain:

1. CONTEXTUAL RELEVANCY   Kya retrieved context sawaal ke liye relevant hai?
   (question + retrieved chunks -> judge)

2. FAITHFULNESS           Kya answer context se support hota hai (hallucination check)?
   (context + answer -> claim verification)

3. ANSWER RELEVANCE       Kya answer sawaal ka seedha jawab hai?
   (question + answer -> judge)

# Teeno saath = high confidence. Combo readings:
# relevancy DOWN + faithfulness OK    -> RETRIEVER problem
# relevancy OK    + faithfulness DOWN -> GENERATOR problem
# dono OK       + answer-relevance DOWN -> PROMPT/jawab dene ka tareeka problem

Yehi combo reading root-cause localization ka engine hai. Ek high-stakes failure aaye toh teeno numbers side-by-side dekho: kya retrieval ne sahi context diya? Diya toh kya generator ne use kiya? Use kiya toh kya usne sawaal ka jawab hi diya? — har sawaal ka "na" tumhe ek specific fix ki taraf le jaata hai.

Yaad rakho: "Retriever pass + generator pass + app fail" = interaction bug. Aise cases ka apna regression test banao — ye sabse mushkil bugs hote hain aur sabse pehle wapas aate hain.

User perspective — application-level quality

PER-QUERY OPERATIONAL METRICS (quality ke saath hamesha track karo):

metric                  kyun track karte hain
──────────────────────  ──────────────────────────────────────────
retrieval latency       embedding + search time — p95 dekho, avg nahi
generation latency      first token aur total dono
tokens per query        context ke saath badhta hai — cost driver #1
cost per query          tokens x pricing (current provider docs se)
error rate              timeout, API errors, empty retrieval results
fallback rate           insufficient-context wale answers ka %

# Trap: quality improve karne ke chakkar mein context 3x kar diya —
# faithfulness +0.02, par tokens 3x aur latency 2x. Dashboard teeno dikhaye tabhi
# ye trade-off dikh payega.
Pro Tip: User-perspective tests ko production feedback se jodo — thumbs-down wale queries ko seedhe application test set mein daalo. User ne jo fail bataya, wo sabse credible failure hai.

Safety evals — harmful, PII, injection, poisoning

SAFETY TEST MATRIX — RAG app ke liye minimum suite:

1. HARMFUL / DISALLOWED     Kya app dangerous/illegal advice dega?
   test: adversarial questions seedhe + indirectly framed
   pass: refusal + safe alternative

2. PII PROTECTION            Kya docs/exact answers mein PII leak hogi?
   test: aise queries jo specific person ka data maange
   pass: redaction ya refusal (policy ke hisaab se)

3. DIRECT PROMPT INJECTION   "Ignore previous instructions and ..."
   test: user input mein hi injection payloads
   pass: app instruction follow karta rahe

4. INDIRECT INJECTION        Retrieved DOC ke andar chhupa hua instruction!
   test: corpus mein malicious chunk plant karke query lo
   pass: document content data hai, instruction nahi — model obey na kare

5. JAILBREAK ATTEMPTS        Roleplay, encoding, multilingual twists
   pass: guardrail bypass nahi hua

6. POISONED RETRIEVAL        Retrieved content se data leak / rule bypass
   pass: malicious chunk ka asar nahi pada
Warning: Safety suite ko release gate se alag scheduled run (weekly/monthly) bhi chalao — naye jailbreak techniques roz aa rahi hain. Purane 10 tests pass hote rehna safety ka proof nahi hai.

Exercise

Question: Retrieval ko evaluate karne ke liye har question ke saath kya chahiye — relevant chunk ka set ya sirf final answer?

Question: Faithfulness ka mechanism kya hai — answer ke claims ko context se verify karna ya sirf answer ki length dekhna?

Question: RAG Triad ke teen checks ke naam bolo (Contextual Relevancy, Faithfulness, ...?)

Question: Retriever aur generator dono solo tests pass par app ke answers kharab — isse kya kehte hain?

Question: Retrieved document ke andar chhupa malicious instruction (jaise "ignore previous rules") — us attack ko kya kehte hain?

Question: Relevant chunk rank-1 pe ho aur rank-5 pe — dono ko distinguish karne wala metric kaunsa — plain accuracy ya rank-aware metric?

Common mistakes

Desi example: Library research vs photocopy wale ka kaam

Tum librarian se topic ka sahi kitab laane ko kaho (retriever), phir photocopy wale se relevant pages nikalwao (context assembly), phir professor usi photocopy se summary le (generator). Har stage ka alag check hai — photo galat page ki ho toh professor kitna bhi accha ho, jawab galat hi hoga:

python
# Desi 4-layer check — kaun gunahgar?
def audit_rag(question, golden):
    retrieved = retriever(question, k=5)
    r = recall_at_k(retrieved, golden.relevant_chunks)      # 1. librarian check
    ctx = assemble_context(retrieved[:3])
    p = contextual_precision(retrieved, golden.relevant_chunks)
    ans = generator(question, ctx)
    f = faithfulness(ctx, ans)                              # 2. professor check
    rel = answer_relevance(question, ans)                   # 3. sawaal-jawab match
    if r < 0.8:  return "librarian dooshm — retrieval fix karo"
    if f < 0.8:  return "photocopy sahi, professor ne extra baat jod di — prompt fix"
    return f"dono sahi, dekho relevance: {rel}"
Interview angle — yahan se sawaal aate hain
  • Q: RAG evaluation ko 4 levels mein kyun todo? Har level ka ek failure example do.
  • Q: Recall@k aur contextual precision ka mechanism samjhao — rank ka role kya hai?
  • Q: Faithfulness kaise compute hoti hai claim-verification se? Isme judge ke kaunse biases aate hain?
  • Q: RAG Triad explain karo — triad ke readings se root cause kaise localize karte hain?
  • Q: Indirect prompt injection kya hai, aur RAG app isse kaise test karta hai?
Lesson complete?

RAG evaluation ka poora map clear — retriever se safety tak. Agla lesson — Agent Evals — mein dekhenge multi-step agent ke decisions, tool calls, recovery aur safety kaise measure karte hain.

Popular courses: Data Science Course in Lucknow · AI Course in Lucknow · Python Course in Lucknow · Power BI Course · Data Analytics Course