Lesson 114 — Advanced

Evaluation as a Release Gate

Eval set tab tak paperwork hai jab tak uska result kuch roke na sake. Is lesson mein hum evaluation suite ko automated tests mein badalna, thresholds set karna, non-determinism se kaise deal karna, aur fast subset versus full suite ka budget dekhenge — jisse ek kharab change CI mein hi ruk jaaye.

Release gate kyun chahiye

Socho ek neeche-wala bandaa prompt edit karta hai, local pe 3 sawaal dekhta hai, "sahi lag raha hai", merge kar deta hai — aur faithfulness gir jaati hai. Gate ka kaam hai wo teen sawaal nahi, poora eval set chalana aur decide karna. Insan thakta hai, gate nahi.

Eval suite = automated tests (DeepEval)

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

def build_cases(dataset):        # tumhara Level 7 ka eval set
    cases = []
    for row in dataset:
        out = rag_app(row["question"])
        cases.append(LLMTestCase(
            input=row["question"],
            actual_output=out.answer,
            retrieval_context=out.contexts,
            expected_output=row.get("gold"),
        ))
    return cases

def test_faithfulness():
    cases = build_cases(FAST_SUBSET)
    evaluate(cases, [FaithfulnessMetric(threshold=BASELINE.faithfulness - TOL)])

def test_answer_relevancy():
    cases = build_cases(FAST_SUBSET)
    evaluate(cases, [AnswerRelevancyMetric(threshold=BASELINE.relevancy - TOL)])

• RAG ke liye metrics: faithfulness (context ke against jawab), answer relevance (sawaal ke against jawab), contextual precision (retrieval ranking).
• Agent ke liye metrics: tool selection sahi ya nahi, task completion, step efficiency (kitne steps mein kaam hua).
• Test cases eval set se aate hain — dataset JSONL hai, test code sirf usse case objects mein badalta hai.
• Ye tests ordinary test runner mein chalte hain — pytest ke saath baaki unit tests ke hi order mein.

Thresholds — baseline se tie karo

baseline.json (recorded):
  faithfulness   = 0.81
  relevancy      = 0.83
  p95_latency_s  = 3.0

policy.yaml (gate rules):
  faithfulness   : >= baseline - 0.02   # chhoti girawati chalegi
  relevancy      : >= baseline
  p95_latency_s  : <= baseline + 0.5
  tokens_avg     : <= baseline * 1.2

# gate = run ke numbers vs policy — fail toh build red
# threshold baseline se hai, "0.8 achha hai" ke belief se nahi

• Absolute thresholds ("faithfulness >= 0.9") jaldi stale ho jaate hain — baseline-relative zyada durable hai.
• Har metric ka apna threshold — latency aur quality ek hi number se naape gaye toh koi bhi ek doosre ko daba dega.
• Threshold mein thoda tolerance band do — non-determinism hai, warna gate hamesha flip hoga.
• Threshold ko git mein versioned policy file rakho — gate kis rule pe ruka, wo bhi history.

Non-determinism se deal karna

problem: same change do baar run -> do alag scores -> gate kabhi pass kabhi fail

fix ke tareeke (priority order):
1. temperature 0 rakho jahan task allow kare (classification, extraction)
2. ek case ko N baar chalao -> median lo, single sample nahi
3. tolerance band: baseline - margin (gate ko thoda elasticity do)
4. flaky metric ko quarantine karo, ignore mat karo — usse fix karo

gate rule: ek baar fail = signal, teen baar fail = pattern

• Temperature 0 bhi provider pe kabhi-kabhi exact repeat nahi deta — isi liye sirf temperature par bharosa mat rakho.
• N repetitions = N guna cost — chhote N (do-teen) se bhi stability kaafi sudhar jaati hai.
• Median ya mean use karo, best-of-n nahi — best-of-n gate ko dhoka deta hai.
• Flaky gate bhi khatarnak hai: log "try again" karne lagte hain aur gate ka respect khatam.

Fast subset vs full suite

Har PR pe poora eval set = ghanton ka CI aur kharcha. Isliye do gears: fast subset (chhota, har change pe) aur full suite (poora, release se pehle). Subset ke cases representative hone chahiye — hamesha wahi 10 easy sawaal nahi.

PR (har push pe):
  unit tests -> FAST subset (representative, minutes) -> build?

release (tag pe):
  FULL suite (poora eval set, multiple runs) -> gate -> deploy?

# gated run ka result MLflow mein log:
# metric history mein dikhega ki kaunsa release kis number ko le gaya

• Fast subset mein wo cases zaroor rakho jo pichhli baar fail hue the — regression cases pehli line of defense hain.
• Full suite ka schedule fixed rakho (release se pehle, aur raat ko scheduled bhi) — "kabhi kabhi chalate hain" ka matlab kabhi nahi.
• Gate results (pass/fail + numbers) hamesha MLflow ya tracker mein log — metric history isi se banti hai ki kaunsa release kis score pe gaya.
• Prompt change, model change aur retrieval change — teeno exact same gate se guzarte hain, koi exception nahi.

Exercise

Common mistakes

• Gate sirf khush-feeling pe chalna: Developer ne teen sawaal dekhe aur merge kar diya. Fix: prompt/model/retrieval change = gate se guzrega, bina exception.
• Absolute magic thresholds: Faithfulness >= 0.9 likh diya bina apna baseline dekhe. Fix: threshold = baseline-relative policy file, git mein versioned.
• Flaky scores pe ignore karna: Gate kabhi pass kabhi fail, log retry maar rahe hain — gate ki credibility khatam. Fix: temperature 0, N runs + median, tolerance band; flaky metric ko fix karo.
• Synthetic cases bina review ke suite mein daalna: Judge model ki bias seedhe gate mein aa gayi. Fix: har synthetic case ek insaan ke review ke baad hi suite mein jaata hai.

gaadi (change) -> checkpoint (CI gate)
papers        -> fast subset report
rule book     -> policy.yaml (thresholds)

pass -> aage (build + deploy)
fail -> wapas (PR red, reason report mein)
# aur: koiVIP entry nahi — prompt change bhi same checkpoint