Lesson 88 — Intermediate

Custom Model Evaluations

Public benchmarks ne shortlist bana di, par final decision yahan hoga — tumhare task pe. Ek model leaderboard pe top hai par tumhari JSON format chhod de, aur doosra mid-ranked hai par perfectly follow karta hai. Is lesson mein hum seekhenge apne real task ka custom eval dataset kaise banate hain, models ko like-for-like kaise compare karte hain, aur quality-latency-cost ka trade-off kaise resolve karte hain.

Custom eval kyun — leaderboard ke baad bhi

Decision rule: leaderboard se shortlist karo (3-4 models), custom eval se winner chuno. Leaderboard pe top-1 model directly select karna aksar galat nikalta hai — uski strength tumhare task pe matter nahi karti, aur weakness (format-following, hallucination, cost) matter karti hai.

Candidate shortlist — 4 axes pe filter

• Capability: tumhare task type pe relevant benchmarks + provider ke model cards dekho. Reasoning-heavy task vs extraction-heavy task ka shortlist alag hoga.
• Context window: tumhara worst-case input (sabse lamba document + prompt + expected output) window mein fit hona chahiye — margin rakh kar.
• Latency class: chat UX ke liye pehla token jaldi chahiye, batch processing mein 10x latency bhi chalega. Shortlist hi latency class ke hisaab se filter karo.
• Cost bracket: per-token pricing providers ke docs mein public hai — monthly request volume se rough cost nikaal kar ek bracket chuno. (Pricing badalta rehta hai — hamesha current docs se verify karo.)

Custom eval dataset banana — Text-to-SQL example

# Ek chhota Text-to-SQL eval dataset — apne schema se banao
cases = [
    {
        "id": "sql-01",
        "schema": "orders(id, customer_id, total, created_at, status)",
        "question": "March mein kitne cancelled orders aaye?",
        "reference_sql": "SELECT COUNT(*) FROM orders WHERE status='cancelled' AND created_at >= '2026-03-01' AND created_at < '2026-04-01';",
        "difficulty": "easy",
    },
    {
        "id": "sql-02",
        "schema": "orders(id, customer_id, total, created_at, status)\ncustomers(id, name, city, segment)",
        "question": "Har city ke top customer ka naam aur unka total spend — sirf enterprise segment.",
        "reference_sql": "...",  # hand-written, phir DB pe run karke verify kiya hua
        "difficulty": "hard",
    },
]
# Scoring: reference se exact match nahi — RUN karke result set compare karo.
# Do alag SQL queries same result de sakte hain = dono sahi.

• Dataset kahan se aaye: production logs ke real inputs (sabse valuable) + hand-written edge cases (ambiguity, missing columns) + LLM se variations (par human verify mandatory).
• Har case mein kya rakhein: input, reference output ya rubric, category/difficulty tag, aur scoring hints (required facts, forbidden claims).
• Execution-based scoring: Text-to-SQL mein SQL pe judge mat lagao — query chala kar result set compare karo. Execution-based scoring mechanism-level truth hai.
• Size: 30-100 well-chosen cases > 1000 scraped. Har category ka representation hona chahiye, warna average ek side ko favour karega.

Like-for-like comparison runner

def compare_models(models, cases, prompt_fn):
    """Same input, same prompt, same params — sirf model badalta hai."""
    report = {}
    for name, call_fn in models:
        scores, latencies, tokens = [], [], []
        for case in cases:
            t0 = time.time()
            out = call_fn(prompt_fn(case), temperature=0.0, max_tokens=512)
            latencies.append(time.time() - t0)
            tokens.append(out.usage.total_tokens)
            scores.append(score_case(case, out.text))      # same scorer for all
        report[name] = {
            "quality": mean(scores),
            "p95_latency": percentile(latencies, 95),
            "avg_tokens": mean(tokens),
        }
    return report
# Rules: temperature 0 (ya same seed), same prompt template, same max_tokens,
# aur SAME scorer. Ek bhi variable alag hua toh comparison bekaar.

Do comparisons mein sabse common galti hai — ek model ko tuned prompt dena aur doosre ko default. Ya ek ka temperature 0 aur doosre ka 0.7. Tab score difference model ka nahi, configuration ka hota hai. Runner mein configuration ek jagah define karo, sab models wahi lein.

Token cost estimate aur quality-latency-cost trade-off

# Cost ka formula — pricing current provider docs se lo (ye placeholder hai)
def cost_per_1k_requests(avg_input_tokens, avg_output_tokens, price_in, price_out):
    # price_* = per million tokens (USD jaisa unit — docs se verify karo)
    in_cost  = avg_input_tokens  * 1000 / 1_000_000 * price_in
    out_cost = avg_output_tokens * 1000 / 1_000_000 * price_out
    return in_cost + out_cost

# Prompt caching ka effect — system prompt 2000 tokens ka hai, roz 100k requests:
# bina cache: 2000 * 100_000 = 200M input tokens (har request pe charge)
# cache se:   cached read rate bahut kam hota hai (provider docs mein ratio dekho)
# Rule: same prefix wale bulk traffic pe caching pehla optimization hai.

# Trade-off resolution order:
# 1. Quality floor se neeche wale models OUT
# 2. Latency requirement se neeche wale OUT
# 3. Bacho mein sabse sasta chuno

• Quality floor pehle: "sabse sasta jo bhi ho" se pehle minimum quality criteria set karo — hallucination rate, format compliance. Fail wale models turant bahar, phir cost compare.
• Latency bhi requirement hai: customer-facing chat mein 3 second ka difference experience bigad deta hai — batch jobs mein farak nahi padta. Requirement context se aati hai.
• Caching levers: prompt caching (stable prefix), aur smaller model ka use — simple steps ke liye chhota model, complex reasoning ke liye bada (router pattern).
• Re-evaluate karte raho: pricing aur model versions badalte rehte hain — jo decision aaj sahi tha, saal baad stale ho sakta hai. Calendar reminder rakho.

Exercise

Common mistakes

• Leaderboard #1 ko seedha select kar lena: — general benchmark ki strength tumhare task pe matter nahi karti. 3-4 models ka shortlist banao aur apne data pe head-to-head chalao — wahi asli decision hai.
• Models ko alag-alag configuration pe test karna: — ek ko tuned prompt, doosre ko default, alag temperature — score fark model ka nahi configuration ka hota hai. Runner mein config ek hi jagah define karo.
• Cost ko per-request soch kar ignore karna: — chhota per-request fark bhi monthly volume pe bada ban jaata hai. 1000 requests ka cost table hamesha quality score ke saath rakho.
• Custom dataset mein PII chhod dena: — production logs se bana dataset valuable hai, par redaction ke bina wo customer data export ban jaata hai. Eval files bhi code repo mein jaati hain — pehle redact, phir commit.

# Billboard (leaderboard) -> shortlist
shortlist = ["model-A", "model-B", "model-C"]

# Trial stitch (custom eval) -> apne 50 cases pe sab chalao
for m in shortlist:
    r = compare_models([m], my_50_cases, my_prompt_template)
    print(m, r["quality"], r["p95_latency"], r["cost_per_1k"])

# Final fitting rule: quality floor -> latency budget -> sabse sasta winner