Lesson 152 — Advanced

Evaluating Fine-Tunes

Tune ka value tabhi asli hai jab wo numberon mein prove ho — ya phir number uske khilaf jaaye aur tum honestly accept kar lo. Is lesson mein hum training se pehle baselines banana, sabhi candidates ko same held-out set aur same judge pe chalana, single score ke bahar dekhna, individual failures aur regressions pakadna, aur faisla likhna dekhenge.

Fine-tune eval kyun alag hai

Fine-tune ka eval isliye khaas hai ki tune hamesha apne target task pe behtar dikhta hai — usi data ke pattern pe train hua hai. Asli sawaal ye hai: same held-out set pe kitna behtar, kitne naye failures aaye, aur off-task cheezein bachi ya nahi. Sirf "target metric upar gaya" aadha answer hai.

Baselines — training se pehle wali table

BASELINES (training shuru karne se pehle bana lo)
1. base model, bare         -> current reality
2. base + best prompt       -> ASLI competitor — yahi beat karna hai
3. stronger model + prompt  -> kabhi kabhi pura tune hi bekaar

# har number isi table mein aayega
# baseline na ho toh "behtar laga" ke alawa kuch nahi bachega

• Baseline 2 hi asli dushman hai: tune ko ek lazily likhe hue prompt se nahi, uske best prompted version se harana hai.
• Prompt effort genuine rakho: ek ghanta achhe se prompt + few-shot examples do — warna strawman baseline ko hara kar kya prove kiya?
• Baseline 3 zaroor chalao: kabhi kabhi stronger model ka seedha prompt hi tune ka saara kaam kar deta hai — pehle jaan lo.
• Numbers training se pehle note karo — tune ke baad dobara purane numbers bhool jaana sabse aasaan galti hai.

Same held-out set, same judge

def compare(candidates, test_set, judge):
    rows = []
    for name, model in candidates.items():
        scores = [judge(model.ask(q), gold) for q, gold in test_set]
        rows.append({"model": name, "mean": mean(scores),
                     "fails": [q for q, s in zip(test_set, scores) if s < PASS]})
    return rows

# same prompts, same judge, same rubric — warna ye comparison
# nahi, do alag exams ke marks jodne ki koshish hai

• Ek hi held-out set: har candidate usi test set pe — alag-alag set ka matlab alag difficulty, comparison meaningless.
• Ek hi judge aur rubric: judge prompt badal diya ya model badal diya toh scores hi alag scale pe aa gaye. Judge ko version banao.
• Blind judge karo: judge ko candidate ka naam na dikhao — warna brand bias scores mein ghul jaata hai.
• Judge noisy ho (LLM judge aksar hota hai) toh har candidate ko multiple runs pe chala lo aur spread bhi record karo.

Single score ke bahar dekho

COMPARISON MATRIX (fill before deploy)
                    tuned   prompted-baseline
quality (task)      <n>     <n>
p95 latency         <n>     <n>
cost per 1k         <n>     <n>
consistency         <n>     <n>       # same prompt, same shape?
new failure modes   ?       none      # ye row sabse zaroori

# ek hi average number poora faisla nahi deta

• Latency aur cost: tune ka apna kharcha hai (serving) — baseline ke token cost se compare karo, per request basis pe.
• Consistency: format task mein average ke saath saath variance bhi matter karta hai — 9 baar sahi 1 baar toota wala model production mein dukh deta hai.
• Specific failures: kaunse examples bigde, kaunse sudhare — aggregate ke peeche chhupa case hi asli story hai.
• Decision matrix mein har row ka apna weight ho — quality upar gayi par latency budget toda toh bhi verdict "no" ho sakta hai.

Regressions, individual failures aur decision

• Off-task prompts chalao: general instruction-following, safety behaviour, aur aise formats jo tune ne kabhi nahi dekhe — yahin se naye regressions nikalte hain.
• Individual failures padho: average improvement ek nayi galti ko chhupa sakta hai jo base model mein kabhi thi hi nahi — failure list khud scan karo.
• Simpler alternatives ek baar phir: behtar prompt ya stronger model jeet jaaye toh result accept karo — tune zid nahi hai.
• Decision likho: kya improve hua, kitna, kya regress hua, aur serving+maintaining ka kharcha gain justify karta hai ya nahi.

DECISION (deploy se pehle bhar do)
task:              <ek line ka task>
metric improved:   <before> -> <after>   (same held-out set)
what regressed:    <off-task / format / safety — list>
latency / cost:    <vs prompted baseline>
verdict:           ship | keep prompting | retrain
sign-off:          <naam, date>

# ye line hi agli team ke liye documentation hai

Exercise

Common mistakes

• Baselines bina training shuru karna: Baad mein "pehle se behtar laga" — par pehle ka number hi nahi tha. Fix: base + best prompt + stronger model ka table training se pehle bharo.
• Har candidate ko alag judge ya alag set pe chalana: Alag exams ke marks jod rahe ho — comparison hi nahi. Fix: ek held-out set, ek judge, ek rubric, sabke liye same.
• Sirf aggregate score dekhna: Average upar, par ek nayi failure category chhup gayi. Fix: failure list scan karo, off-task sample bhi chalao, individual cases padho.
• Negative result chhupana: Tune kaam nahi kiya par report mein sirf positives — aage wali team galat raaste pe jaayegi. Fix: decision document mein sach likho: improved, regressed, verdict teeno.

same pitch + same bowlers = same held-out set + same judge
batting + fielding        = quality + latency + cost + consistency
injury check              = off-task regressions
written report            = decision document

bina protocol ke "iska form achha lag raha hai" = andaza