Lesson 26 — Beginner

Classical Metrics: BLEU, ROUGE & Perplexity

Generation ka number dekhna aasaan hai, uska matlab samajhna mushkil. Is lesson mein tum dekhoge ki BLEU, ROUGE aur perplexity kaise calculate hote hain — reference vs candidate overlap — aur wo kyun hamesha wahi nahi naapte jo tumhe chahiye. Mental model saath rakho: ye numbers surface match naapte hain, truth ya usefulness nahi; isi liye kab human/LLM-judge eval ko sambhalna padta hai, ye bhi yahin se decide hoga.

Overlap metrics — kya naapti hain

Chain ye hai: reference + candidate → n-gram match → score. Score batata hai kitne shabd/matching patterns mile — bas. Iska matlab wo nahi batata ki jawab sach hai, useful hai, ya instructions follow kiye. Ek hi meaning ke sau tarike hote hain, aur single reference unmein sirf ek ko "sahi" maanta hai.

BLEU — precision + brevity penalty (haath se dekho)

from collections import Counter

ref = "the cat sat on the mat".split()              # 6 tokens
c1  = ref.copy()                                   # copy wala candidate
c2  = "a cat was sitting on a mat".split()         # 7 tokens (behtar paraphrase)

def uni_prec(c, r):
    return sum((Counter(c) & Counter(r)).values()) / len(c)

print(round(uni_prec(c1, ref), 2))   # 6/6 = 1.00  -> copy sabse achha?
print(round(uni_prec(c2, ref), 2))   # 3/7 = 0.43  -> cat, on, mat
# bigrams of c2: (a,cat)(cat,was)(was,sitting)(sitting,on)(on,a)(a,mat)
# ref ke bigrams se koi match nahi -> p2 = 0 -> BLEU-2 = 0 (bina smoothing)

• BLEU = modified n-gram precision × brevity penalty: unigram, bigram, trigram... ke match rates ka geometric mean — matlab "candidate ke shabd mein se kitne reference ke the" (precision side).
• Brevity penalty: chhote answer ko rokna — BP = 1 agar candidate lamba hai, warna exp(1 − r/c). Example: r=6, c=4 → exp(1 − 1.5) ≈ 0.61, score aadha ho gaya.
• Machine-translation origin: ek sentence ka translation naapne ke liye bana tha — jahan reference wahi matlab ki doosri line hoti hai.
• Chat answers kyun fail karte hain: lambai alag, phrasing alag, ek hi meaning ke sau tarike — bigram precision zero aane wali baat hai (upar wala c2 example).
• Smoothing zaroori hai: chhote outputs mein koi n-gram mila nahi toh poora score zero — isliye evaluation code mein smoothing ka concept hota hai.

ROUGE — recall-oriented overlap

ref = "the cat sat on the mat".split()      # 6 tokens (reference)
c2  = "a cat was sitting on a mat".split()  # 7 tokens (candidate)
matched = 3                                  # cat, on, mat

print(round(matched / len(c2), 2))   # ROUGE-1 precision = 0.43
print(round(matched / len(ref), 2))  # ROUGE-1 recall    = 0.50

# ROUGE-L: longest common subsequence = cat..on..mat = 3
print(round(3 / 6, 2), round(3 / 7, 2))   # R-L recall 0.50, precision 0.43

# copy trap: candidate = reference -> ROUGE 1.00, meaning khaali

• ROUGE-1: unigram overlap. Recall side (matched / reference tokens) summarization mein chalti hai — "source ka kitna cover hua".
• ROUGE-2: bigram overlap — thoda zyada context (sirf "cat" nahi, "cat sat" chahiye). Chhoti reference pe ye number jaldi girta hai.
• ROUGE-L: longest common subsequence — order ke saath match, par beech ke extra shabd chalta hai (paraphrase thoda bardasht).
• Copy-the-reference trap: candidate ne reference chipka diya toh ROUGE 1.0 — par wo toh cheating/paraphrase hai. Score high, kaam khaali.
• Precision aur recall dono dekho: candidate lamba hai toh precision girega, chhota hai toh recall. Sirf ek side dekhna aadhi kahani hai.

Perplexity — next-token ki uncertainty

from math import exp, log

# model har next token ko p = 0.25 deta hai (4 equal options ka jhund)
p = 0.25
print(1 / p)                      # 4.0 -> perplexity: itna confused hai

# do tokens ki predicted probability se (geometric mean of 1/p)
probs = [0.5, 0.25]
nll = -sum(log(x) for x in probs) / len(probs)   # (0.693 + 1.386)/2 = 1.040
print(round(exp(nll), 2))         # 2.83  -> sqrt(2 * 4) = 2.83

# rule: perplexity sirf SAME tokenizer wale models mein comparable

• Mechanics: model har position pe apni next-token probability deta hai. Us probability ka negative log (surprise) le kar average karo, phir exp — perplexity effectively "model kitna confused hai" ka number hai.
• Kyun ye metric hai: training objective (next-token prediction) ka seedha nateeja — LM ko compare karne ka sasta automatic number, par bina task ke.
• Same-tokenizer rule: alag vocab = alag difficulty. Do models alag tokenizers pe toh numbers hi alag scale ke — compare karna jaise feet aur metre.
• Low perplexity ≠ achha answer: generic, predictable text bhi low-surprise hota hai. Model confidently bhi galat ho sakta hai — ppl uski confidence ki property hai, correctness ki nahi.
• Contamination ka dhoka: agar test text model ki training mein thi toh perplexity trivially low — number achha, eval bekaar.

METEOR aur string overlap ki seema

SAME WORDS, ALAG MEANING              ALAG WORDS, EK MEANING
------------------------------------  ------------------------------------
"delivery in 2 days" vs               "arrives day after tomorrow" vs
"delivery in 2 days but delayed"      "ships within 48 hours"
-> overlap zyada, meaning FARQ         -> overlap KAM, meaning SAME

yahi string-overlap metrics ki core limit hai:
surface similarity != correctness

• METEOR ka idea: exact match se thoda flexible — stem/synonym-aware matching, aur recall ko zyada wazan. Paraphrase thoda pakadta hai, phir bhi string matching hi hai.
• Common failure modes: paraphrase (match kam), opposite meaning ke same words (match zyada), reference copy (match poora), fluent fabrication (match aadha, sach nahi).
• Metric ki jagah: overlap metrics regression pakadte hain (kal se score gira = kuch bigda). Correctness aur usefulness ka sawaal wahan se shuru hota hai jahan metric khatam ho jaata hai.
• Purane metrics aur LLM text: LLM outputs lambi, structured, paraphrase-heavy — isi liye purane numbers unpe behave nahi karte. Metric bhi era ke saath update hota hai.

Beyond overlap — human eval aur LLM-as-judge

rubric (judge ya insaan ko dena — vibes nahi):
  1. claim/source check: jawab ki har factual line source mein hai?   yes/no
  2. coverage: instruction ke saare points cover hue?                 count
  3. format: expected shape mein hai?                                 yes/no
  4. harmful/false content:                                          yes/no

human loop:  har release pe ek chhota fixed sample -> insaan same rubric se ->
             judge ke scores ke saath disagreement review

judge rules: reference ki jagah rubric do; order randomize karo;
             judge ki length/style bias sample pe check karo
stop point:  score stable aur decision clear ho toh BLEU band kar do

• Human eval: thoda sample, clear rubric, do logon se independent scoring. Ye anchor hai — judge ko isi se calibrate karte hain.
• LLM-as-judge: rubric ke sawaal mein deta hai (yes/no, count, 1-5) — vague "rate this" nahi. Anchors do: "ye 1 jaisa hai, ye 5 jaisa".
• Judge ke biases: lambe/format-wale output ko zyada dena, order (kaunsa pehle) ka asar — isliye order randomize aur spot checks zaroori.
• Kab overlap chhod do: paraphrase-heavy ya chat tasks, single reference, aur jab decision ke liye "usefulness" hi asli sawaal ho.
• Kab overlap rakho: regression gate ke roop mein (kal se BLEU gira = kuch bigda) — lekin kabhi single decision-metric nahi.

Exercise

Common mistakes

• BLEU/ROUGE ko truth maanna: Score high = jawab sahi? Nahi — wo sirf shabd match naapta hai, aur copy sabse zyada match karta hai. Fix: overlap ko trend/regression gate rakho, correctness ke liye rubric + sample reading.
• Alag tokenizer wale models ka perplexity compare karna: Vocab alag, difficulty alag — number hi incomparable. Fix: same tokenizer, same text distribution; warna per-token trend apne hi setup mein dekho.
• Bina rubric ke LLM-as-judge: Judge vibes se deta hai — score badal jaata hai, meaning nahi. Fix: explicit rubric (source check, coverage, format), anchors, randomize order, disagreement review.
• Single reference pe chat-style eval: Paraphrase sahi hai par score girega — metric hi wrong tool hai. Fix: multi-reference jahan possible, warna rubric/judge; BLEU ko sirf regression gate tak simit rakho.

keyword check (BLEU/ROUGE):  shabd match -> number   (sasta, tez, andhadhundh)
apne shabdon mein sahi:      match kam -> number bhi kam (yahi dhoka hai)
rubric wala TA (LLM judge):  sawaal-by-sawaal score   (structured, par calibrate karo)
examiner khud (human eval): meaning check -> asli anchor