Lesson 85 — Intermediate

Offline vs Online Evaluations

Offline evaluation tumhe release se pehle batata hai ki "kya model/prompt toota toh nahi" — par production mein wahi cheezein hoti hain jo test set mein kabhi sochi hi nahi gayi thi. Is lesson mein hum dono layers ka design samjhenge: offline suites jo har change pe chalte hain, aur online evaluation jo deployed system ki actual quality roj measure karti hai.

Dual-layer strategy — offline aur online

Funda simple hai: offline rokta hai bura release, online pakadta hai chupi hui problems. Sirf offline = "test toh pass ho gaya" bol kar ship kar diya, par users ke actual inputs ne phir bhi fail kiya. Sirf online = pata tab chalta hai jab damage ho chuka ho. Dono layers zaroori hain.

Offline suites — pre-release gate

• Version comparison: model A vs model B, prompt v1 vs v2 — same dataset, same scorers, alag-alag runs. Bina offline suite ke comparison sirf 5 hand-picked examples pe hota hai.
• Regression prevention: pehle wale failing cases bhi suite mein rehte hain (regression tests jaise). Koi fix wapas na toote — iska proof sirf suite deta hai.
• Experimentation: naya retrieval strategy, naya system prompt — sab pehle offline dustane pe, phir canary, phir full rollout.
• Cost control: offline run mein hi decide ho jaata hai ki model upgrade "worth it" hai ya nahi — production mein sabko dikhane se pehle.

# Ek minimal offline suite — pytest style
def test_refund_answer_faithful():
    case = load_case("refund-03")
    out = app(case["input"])
    assert fact_check(out, case["required_facts"]) >= 0.8, f"missing facts: {out}"
    assert not contains_claims(out, case["forbidden_claims"]), "hallucinated claim"

def test_json_mode_schema():
    out = app_json("List 3 refund rules")
    assert json.loads(out)["rules"].__len__() == 3

# Har naye production failure ka regression case:
def test_regress_2024_user_said_X():
    out = app("tumhara sabse bura input yahan")
    assert meets_criteria(out), "wahi purana bug wapas aa gaya"

# Run: pytest evals/ --tb=short  -> CI mein gate ke roop mein

Post-deployment risks — kyun online chahiye

• Unexpected inputs: tumne Hindi, code-switched, slang wale inputs test hi nahi kiye — par unhe users roz bhejte hain.
• Emergent systematic failures: har example pe chhota hai, par ek specific user segment ke liye pattern ban jaata hai. Aggregate offline score mein wo chhup jaata hai.
• Changing user behaviour: launch ke 3 mahine baad log app se naye tarike se baat karne lagte hain — jo vocabulary tumhare test set mein nahi thi.
• Data drift: RAG ka corpus update hota hai, prices badalte hain, policy badalti hai — retrieval aur answers dono shift hote hain bina tumhare code change kiye.
• Silent provider changes: model provider version update kar sakta hai. Tumhara code same, behaviour naya — sirf production monitoring hi batayega.

System metrics vs quality metrics

SYSTEM METRICS (auto-capture, har request pe)     QUALITY METRICS (LLM/human se score)
─────────────────────────────────────────────     ──────────────────────────────────────
latency (p50/p95), timeouts                        faithfulness / groundedness
error rate, HTTP 5xx, tool failures                answer relevance, completeness
tokens per request, cost per request               refusal correctness (refuse karna tha?)
retry count, fallback usage                        tone / format / instruction-following
cache hit rate                                     safety flags (harmful, PII, injection)

# System metrics billing & SLO ke liye, quality metrics trust ke liye.
# Dono ek dashboard pe side-by-side chahiye — ek behtar doosre ke bina adhura.

System metrics tumhe pata bhi nahi chale kab capture ho jaati hain — request ke saath latency, tokens, error code apne aap log ho jaate hain. Quality metrics ka effort zyada hai: ya toh LLM judge lagao ya human sampling. Teams aksar sirf system metrics dekhti hain — p95 latency theek hai toh sab "green" — par users ko toh galat jawab ho sakta hai latency bilkul sahi hone ke baad bhi.

Continuous online evaluation pipeline

# Production pipeline ka sketch: log -> sample -> score -> dashboard -> alert

def log_request(req, resp, meta):
    db.insert({"ts": now(), "input": req, "output": resp.text,
               "latency_ms": meta.latency, "tokens": meta.tokens,
               "prompt_hash": meta.prompt_hash, "model": meta.model})

def sample_for_scoring(n_per_day=200):
    # uniform sample + stratified: har category/user-segment se minimum
    return db.sample_uniform(n_per_day // 2) + db.sample_per_category(n_per_day // 2)

def score_and_alert(samples):
    for s in samples:
        s["faithfulness"] = judge_faithfulness(s["context"], s["output"])
        s["toxic"] = safety_classifier(s["output"])
    daily = aggregate(samples)          # per-metric avg + failure rate
    dashboard.publish(daily)
    if daily["faithfulness"] < BASELINE["faithfulness"] - 0.05:
        alert("faithfulness regressed vs 7-day baseline")

• Sampling zaroori hai: har request judge karwana mehenga hai. Stratified sampling lo — har language, category aur user type se thoda-thoda, taaki chhote segment chhup na jaayein.
• Baseline se compare karo: absolute number se zyada matlab hai trend se. 7-day rolling baseline ke against kal ka score — isi se "aaj kuch bigda hai" dikhta hai.
• Dashboard + alerting: score sirf tab kaam ka hai jab koi dekhe. Daily dashboard + threshold alerts (Slack/email) minimum chahiye.
• PII pehle redact karo: logs mein user data jaata hai — sampling se pehle PII redaction pipeline mandatory hai.

Failures ko offline mein wapas bhejna

Online pipeline ka sabse valuable output dashboard nahi, failure queue hai — wo concrete examples jinme app galat tha. Har verified failure ko ek candidate regression case banao, human verify karo, aur offline golden dataset mein add karo. Isi se tumhara test set live rehta hai — warna wo launch wale din mar jaata hai.

# Failure -> golden dataset ka loop
def promote_failure(sample, reviewer):
    if sample["faithfulness"] < 0.5 and reviewer.confirm("wrong answer"):
        golden.add({
            "id": f"prod-{sample["id"]}",
            "input": redact_pii(sample["input"]),
            "reference_answer": reviewer.writes_correct_answer(),
            "category": sample["category"],
            "source": "production-2026-06",
        })
        offline_suite.add_regression_test(golden[f"prod-{sample["id"]}"])
# Har hafte: failure queue review meeting — yehi tumhari quality ka asli engine hai.

Exercise

Common mistakes

• Sirf offline suite par bharosa rakhna: — test set launch din pe representative tha, ab nahi. Production inputs, user behaviour aur data sab drift karte hain — online sampling ke bina tum andhere mein chal rahe ho.
• Quality ko latency se judge lena: — p95 latency theek hai iska matlab nahi ki answers sahi hain. System metrics operational health batate hain, quality metrics trust — dono dashboard pe chahiye.
• Production failures ko "ek baar ka issue" bol kar ignore karna: — har verified failure ek regression test ban sakta hai. Ignore kiya toh wo dobara aayega, aur tab pata bhi nahi chalega ki purana bug hi hai.
• Bina baseline ke online scores dekhna: — "faithfulness 0.81 hai" ka koi matlab nahi jab tak pata na ho pichhle hafte kya thi. Rolling baseline ke against trend dekho, alerts wahi set karo.

# Offline: release se pehle
for case in golden_dataset:
    score = evaluate(app(case.input), case.criteria)
    if score < THRESHOLD:
        block_release(f"{case.id} failing")

# Online: release ke baad, roz ka sample
for s in db.daily_sample(200):
    q = judge(s.input, s.output, s.context)
    weekly_trend.log(q)
# Chef ka rule: customer ki plate pe 3 complaints -> menu review (failure -> dataset)