Lesson 89 — Intermediate

Building an Eval Suite & Regression Testing

Ab tak tumne alag-alag eval techniques seekhi hain — ab unhe ek system banaenge jo har change pe automatically chalta hai aur galat release ko rokta hai. Is lesson mein hum dekhenge suite ko kin levels pe todna hai, tests/configs/results ko kaise version karte hain, CI mein quality gate kaise lagte hain, aur production failures se suite kaise khud improve karta rehta hai.

⏱ 20 min• Intermediate• Custom Model Evaluations

Suite ka structure — teen levels

WHAT

Eval suite = versioned test cases + scoring functions + runner + baselines, jo har change pe automatically chalta hai aur pass/fail verdict deta hai. Ye tumhare app ka safety net hai — code regression tests jaise, par LLM behaviour ke liye.

WHEN

Har prompt change, model update, retrieval tweak aur config change pe — CI mein automatic. Release gate ke roop mein: suite fail = ship nahi. Weekly full run + daily smoke run ka rhythm aksar kaam karta hai.

WHERE

Repo mein eval/ directory (tests, configs, scorers) + ek results store (experiment tracker ya simple DB) + CI pipeline mein ek job. Dashboard optional par useful.

EVAL LEVELS — kya kahan test hota hai
──────────────────────────────────────────────────────────
COMPONENT level   ek ek part: retriever, prompt template, single LLM call
                  (fast, cheap, precise diagnosis — kis part ne fail kiya)

PIPELINE level    components jod kar: retrieve -> assemble context -> generate
                  (interaction bugs pakadta hai jo component pe nahi dikhte)

APPLICATION level poora app jaisa user dekhta hai: streaming, tools,
                  formatting, safety — end-to-end behaviour + SLOs

Har level ka apna dataset, apna scorer aur apna failure signature.

Funda: jitna neeche fail hoga, diagnosis utna aasaan. Application test fail hua toh pata nahi retrieval bigda ya prompt. Component test fail hua toh seedha culprit. Isliye suite design karte waqt har bug ko uske sabse neeche wale level pe test banane ki koshish karo.

Versioning — tests, configs, prompts, results

# configs/exp-042.yaml — har run ka poora fingerprint
run_id: exp-042
date: 2026-06-10
model:
  name: gpt-class-model        # exact version pin karo, "latest" kabhi nahi
  temperature: 0.0
  max_tokens: 512
prompt:
  template_id: support-v7      # prompt file ka git commit hash bhi log karo
retriever:
  index: docs-2026-06-01       # corpus snapshot date
dataset:
  name: golden-v3
  cases: 84
baseline:
  run_id: exp-039
# Result store mein har row: run_id + case_id + scores + latency + tokens.
# Isliye 6 mahine baad bhi bata sakte ho — kaunse change ne score chalaya.

Tests bhi code hain: golden dataset + scorer git mein versioned. Har case ka stable ID rakho — wahi case naye run mein dobara dikhe, tabhi trend meaningful hai.
Config fingerprint zaroori: model name, temperature, prompt hash, index date — bina ye log ke 2 runs compare hi nahi kar sakte. "Score badla kyun?" ka jawab tabhi milta hai.
Results immutable rakho: har run ka result overwrite mat karo — append karo. Experiment tracking ka funda hi yahi hai: history chahiye tabhi baselines ka matlab hai.
Model versions pin karo: "latest" alias kabhi use mat karo — provider ne alias silently update kar diya, tumhara "stable" suite bina code change ke badal gaya.

Rule of thumb: Koi bhi result dekhte waqt 3 cheezein poochho — kis dataset pe, kis config pe, kis baseline ke against. Teeno pata ho tabhi number ka matlab hai.

Baselines aur CI/CD quality gates

# Quality gate — CI mein ye script release ko rokti hai
BASELINE = load_baseline("exp-039")     # last known-good run
GATES = {
    "faithfulness":     {"min_abs": 0.80, "max_drop": 0.03},
    "format_compliance": {"min_abs": 0.95, "max_drop": 0.01},
    "p95_latency_ms":   {"max_abs": 2500, "max_rise": 0.10},
    "cost_per_1k":      {"max_rise": 0.15},
}

def check_gate(current):
    failures = []
    for metric, rule in GATES.items():
        base, now = BASELINE[metric], current[metric]
        if "min_abs" in rule and now < rule["min_abs"]:
            failures.append(f"{metric}: {now} < absolute floor {rule['min_abs']}")
        if "max_drop" in rule and now < base * (1 - rule["max_drop"]):
            failures.append(f"{metric}: dropped >{rule['max_drop']*100:.0f}% vs baseline {base}")
        # max_abs / max_rise similarly
    return failures   # khaali list = release jaari, warna BLOCK

# Run: python evals/gate.py --current results/exp-043.json -> exit 1 = CI fail

Do tarah ke threshold: absolute floor (kitna bhi ho, faithfulness 0.80 se neeche nahi) aur relative drop (baseline se zyada girna nahi). Dono chahiye — kabhi kabhi baseline hi gira hua ho sakta hai.
Non-critical metrics = warn, critical = block: har metric pe gate lagana friction banata hai. 2-3 truly critical metrics block karein, baaki warning mein rahen.
Flakiness se bacho: gate deterministic hona chahiye — fixed model pin, sufficient dataset size, ya multiple samples. Warna engineers "retry maar dete hain" seekh jaate hain aur gate ki izzat mar jaati hai.
Gate ka output actionable ho: sirf "FAIL" nahi — kaunse cases, kaunse metrics, kis direction mein. Debug karna tabhi possible hai.

Warning: Quality gate tabhi kaam karta hai jab team uski respect kare. Ek hafte mein 2 baar false alarm aaya toh log gate ko bypass karna shuru kar denge — pehle flaky tests saaf karo, phir gate lagao.

Experiment tracking aur comparison

Har change (prompt v7, naya retriever, model switch) ek experiment hai — usko ek run_id do, results store karo, aur baseline se side-by-side compare karo. Yehi discipline batati hai ki "improvement" sach mein hai ya ek metric sudhar gayi par doosri chupke se bigad gayi (jaise faithfulness upar, completeness neeche).

# Comparison view — kya badla, kis direction mein, kaunse cases pe
def diff_runs(baseline, current, cases):
    rows = []
    for c in cases:
        b, n = baseline[c.id], current[c.id]
        if passes(b) and not passes(n):        # naya failure = REGRESSION
            rows.append((c.id, "REGRESSION", b.scores, n.scores))
        elif not passes(b) and passes(n):      # fix
            rows.append((c.id, "FIXED", b.scores, n.scores))
    return rows
# Rule: aggregate average se pehle wahi 5-10 REGRESSION rows dekho.
# Average upar gaya bhi ho, agar 3 critical cases toote hain toh release nahi.

Pro Tip: Regression rows ko ticket ya PR comment mein auto-post karo. "Kya badla" ki conversation tabhi hoti hai jab failing cases saamne hon — numbers se discussion nahi chalta.

Self-improving golden dataset

Source 1 — production failures: har hafte online eval se aaye verified failures, unke corrected answers ke saath suite mein. Ye sabse valuable additions hain.
Source 2 — near-misses: jo cases gate pass ho gaye par barely — unhe bhi rakho, warna agla change unhe paar kar jaayega.
Source 3 — user complaints: support tickets, thumbs-down feedback — seedhe user pain points, hand-verifiye karke cases banao.
Hygiene: duplicate aur trivial cases hatao, category balance maintain karo, aur har addition pe human review rakho — bina verification ke dataset polluted ho jaayega.

Ye loop isi liye "self-improving" hai: app use hota hai → failures milte hain → wo cases test bante hain → app aur strong hota hai. Ek saal chalne wale LLM teams ka farq yahi hota hai — jinke paas ye loop hai, unka suite har mahine behtar hota hai; baaki ke suite launch wale din ke 50 cases par atak jaate hain.

Quality ke saath latency aur cost track karna

# Har run ka result row — quality ALONE nahi, teeno saath
result = {
    "run_id": "exp-042",
    "quality": {"faithfulness": 0.91, "completeness": 0.84, "format_ok": 0.99},
    "latency": {"p50_ms": 800, "p95_ms": 2300},
    "cost": {"tokens_per_case": 2100, "cost_per_1k_usd": 4.20},  # pricing docs se
}
# Classic trap: naye prompt se faithfulness 0.86 -> 0.91, par tokens double
# (2x cost) aur p95 latency 2x. Kya ye improvement hai? Decision team ko chahiye.
# Isliye dashboard hamesha teeno panels saath mein dikhaye.

Ek metric ko optimise karte-karte doosre chupke se bigam jaate hain — ye LLM systems mein bahut common hai. Lamba context = better faithfulness par 3x cost. Judge lagana = better coverage par 2x latency. Suite ka kaam sirf score dena nahi, teeno axis ek saath dikhana bhi hai — tabhi trade-off conscious decision banta hai.

Yaad rakho: Dashboard ka layout hi culture banata hai — agar quality ka panel sabse upar hai aur cost/latency neeche chhupi hai, toh team cost bhool jaayegi. Teen equal panels = teeno importance mein.

Exercise

Question: Eval ko component, pipeline aur application — teen levels mein todte hain. Sabse precise diagnosis kis level pe milta hai?

Question: CI gate ko kya karna chahiye — har metric pe block ya critical metrics pe block aur baaki warn?

Question: Config mein model ka version kaise pin karo — "latest" alias ya exact version?

Question: Self-improving dataset ka sabse valuable source kya hai — production failures ya hand-written cases?

Question: Faithfulness behtar par 3x cost — isse kya kehte hain quality aur cost ke beech?

Common mistakes

Baselines ke bina absolute scores dekhna: — "faithfulness 0.85 hai" ka matlab tab tak nahi jab tak pichhle known-good run ka number na ho. Har metric ko baseline se diff karo — wahi regression hai ya nahi batata hai.
"latest" model alias pe production chalana: — provider ne alias silently update kar diya toh bina code change ke behaviour badal jaayega. Hamesha exact version pin karo, upgrade ek deliberate experiment ke roop mein karo.
Gate mein har metric pe hard block: — gate itna strict ho toh team use bypass karne lagti hai. 2-3 critical metrics block karein, baaki warnings — aur flaky tests pehle saaf karo.
Quality improvement ko bina cost/latency ke celebrate karna: — 5-point faithfulness gain jo 3x cost laaye, wo decision hai improvement nahi. Har result teeno axes pe padho — warna ek jagah ka fayda doosri jagah ka nuksan ban jaayega.

Desi example: Assembly line ka har-inspection

Factory mein har part ka inspection (component test), full assembly check (pipeline test) aur final road test (application test) hota hai — aur koi bhi check fail toh gaadi showroom nahi jaati. Tumhara LLM app bhi wahi factory hai:

# Har commit pe teeno levels ka smoke run
levels = {
    "component":  test_retriever_and_prompt_unit_cases,
    "pipeline":   test_retrieve_generate_flow,
    "application": test_end_to_end_user_scenarios,
}
for name, suite in levels.items():
    run = suite.run(config=exp_config)
    if run.has_regressions_vs(baseline):
        block_release(f"{name} level regression: {run.regressions}")
# Factory rule: inspection fail = gate banda. Koi exception nahi, koi jaan-pehchaan nahi.

Interview angle — yahan se sawaal aate hain

Q: Component, pipeline aur application-level evals mein farq? Ek ek failure example do.
Q: Quality gate design karte waqt absolute floor aur relative drop threshold ka farq kya hai?
Q: Experiment tracking mein config fingerprint kyun zaroori hai — kya-kya log karte ho?
Q: Self-improving golden dataset ka loop explain karo — production se suite tak kaise aata hai?
Q: Ek change se faithfulness badhi par latency doubled — tum kaise decide karoge ki ship karna hai ya nahi?

Lesson complete?

Eval suite ka system design clear. Agla lesson — RAG Evals — mein ek RAG pipeline ko component se lekar safety tak kaise evaluate karte hain, wo detail mein dekhenge.