Lesson 87 — Intermediate

The Evolution of Knowledge Benchmarks

Ek time tha jab MMLU ka score hi model ki knowledge ki poori kahani thi. Phir top models ne us benchmark ko saturate kar diya, aur research community ko harder tests banane pade. Is lesson mein hum knowledge benchmarks ka evolutionary chain dekhenge — kis benchmark ne kya naya measure kiya, aur kyun har generation ka benchmark pehle se zyada mushkil hota gaya.

Knowledge benchmark kya measure karta hai

Knowledge aur reasoning alag-alag cheezein hain, par benchmarks aksar mila kar test karte hain — kyunki ek hard question solve karne ke liye dono chahiye. Isliye knowledge benchmarks ka evolution dekhte waqt difficulty badhane ka matlab sirf "zyada facts" nahi, "zyada layered understanding" hai.

Evolution — MMLU se harder tests tak

BENCHMARK              KYA TEST KARTA HAI
─────────────────────  ──────────────────────────────────────────
MMLU                   57 subjects, exam-style MCQ — broad knowledge ka baseline
TruthfulQA             Kya model sacchi hai ya common misconceptions dohrata hai?
AGIEval                Human competitive exams (SAT, LSAT, bar exams) — human vs model
MMLU-Pro               MMLU ka harder version — zyada distractors, reasoning zyada
GPQA                   PhD-level science questions (domain experts ke liye bhi mushkil)
SimpleQA               Short factual questions — "seedha fact hai ya nahi" — hallucination probe
Humanity's Last Exam   Frontier researchers dwara diye gaye sabse mushkil sawaal

• MMLU: pehla bada "general knowledge" standard — aaj bhi reference point hai, par top models ise saturate kar chuke hain (isliye "MMLU ka score ab informative nahi").
• TruthfulQA: sawaal "kya model sacchi bolega" — common myths aur misconceptions pe test. Knowledge ka quantity nahi, honesty measure karta hai.
• AGIEval: human exams ko benchmark banaya — "model kis human percentile pe hai" wala framing pehli baar mainstream hua.
• MMLU-Pro: MMLU ke 4 options ki jagah 10 distractors — ab guessing se score nahi aata, real understanding chahiye.
• GPQA: biology/physics/chemistry ke aise questions jinme domain experts bhi galat ho jaate hain — PhD-level knowledge + reasoning.
• SimpleQA: chhote seedhe factual sawaal — ismein "model ko sach bolna nahi aata" wali problem nangi dikhti hai. Yeh hallucination tendency ka clean probe hai.
• Humanity's Last Exam: frontier researchers ke contributed questions — sabse mushkil current benchmark, designed to stay hard for the next model generation.

Kyun benchmarks lagataar mushkil hote jaate hain

Yeh ek co-evolution hai: models better → old tests saturate → researchers harder tests → models phir better. Iske peeche teen forces hain — contamination (purane public tests training data mein ghus jaate hain), saturation (top scores ke beech koi fark nahi rehta), aur genuine capability growth (models sach mein aage badhe hain, unhe naya challenge chahiye).

# Saturated benchmark ka effect — do models ka score almost same:
scores_v1 = {"old_model": 88, "new_model": 89}   # 1 point ka fark — informative?
scores_hard = {"old_model": 41, "new_model": 63}  # 22 point ka fark — ab farq dikhta hai

# Isliye leaderboard padhte waqt poochho:
# 1. Ye benchmark kitna saturated hai? (top scores kahan hain)
# 2. Dataset kab bana, public hai ya private?
# 3. Kya ye meri capability ke liye relevant hai?

Model ke level ka sahi test chunna

• Weak/mid models shortlist kar rahe ho? Basic knowledge + instruction-following tests kaafi hain — wahan GPQA jaisa extreme test confusing signals deta hai.
• Frontier models compare kar rahe ho? Purane saturated tests se kuch nahi milega — harder, fresher benchmarks chahiye (GPQA-style, contamination-checked sets).
• Honesty/hallucination dekhna hai? SimpleQA-style factual probes ya TruthfulQA-style misconception tests use karo — general knowledge benchmark se ye alag cheez hai.
• Apne domain ki knowledge? Public benchmark kabhi bhi tumhara jargon nahi cover karega — apne docs se exam-style questions banao (agla lesson isi pe hai).

Exercise

Common mistakes

• Purane saturated benchmark ko current capability maanna: — MMLU jaisa purana test top models ke beech fark nahi dikhata. Naye frontier models compare karne ke liye harder aur fresher tests chahiye.
• Har benchmark ka score ek hi "intelligence" number samajhna: — knowledge, honesty, reasoning alag-alag hain. Ek model knowledge mein strong ho sakta hai par hallucinate zyada kare — category-wise padho.
• Human-exam benchmark ka matlab literally "human se better" samajhna: — AGIEval-style comparison exam conditions pe hota hai. Real practice, tools aur context mein human-model farq alag ho sakta hai — framing samjho.
• Tough benchmark ko apne app ka proxy banana: — GPQA score se tumhara customer-support task thodi better hoga. Tough benchmark general capability ka signal hai, task-specific decision ke liye apna eval chahiye.

# Difficulty ladder — kab kaunsa test meaningful hai:
ladder = {
    "basic":    "MMLU-style — kya foundation hai",
    "harder":   "MMLU-Pro — guessing se kaam nahi chalega",
    "expert":   "GPQA — experts ke liye bhi mushkil",
    "frontier": "HLE — agla generation bhi struggle kare",
}
# Rule: jis level pe student (model) hai, wahi test do.
# Olympiad de kar board student ko judge karna — unfair aur bekaar.