Lesson 154 — Advanced

Capstone: a LoRA that beats a prompted baseline

Is capstone mein tum poore L13 ko ek project mein jodoge: ek task chunna, uska honest prompted baseline banana, saaf dataset curate karna, LoRA train karna, aur eval ke through prove karna ki tumhara tune us baseline ko beat karta hai — ya honestly accept karna ki nahi karta.

⏱ 10 min• Advanced• Serving the Adapter

Capstone kya banega

Kya hai?

Ek end-to-end fine-tuning project: task + failure → baseline → dataset → LoRA run → eval → decision. Success ka matlab leaderboard nahi, apne hi best prompted baseline ke against measurable gain hai.

Kab kaam aata hai?

Jab L13 ke saare lessons ek dusre se jud jaayein — yehi wo project hai jo interview mein aur portfolio dono mein dikhega, kyunki ismein decision se lekar serving tak sab kuch hai.

Kahan milta hai?

Tumhare repo mein: data/ (versioned), runs/ (configs + logs), eval/ (comparison reports) aur ek decision README jo poori kahani likhi rakhe.

Asli objective do hi hain: (1) ek defensible gain dikhana, (2) poora process traceable rakhna. Tune thoda bhi behtar ho par process likha nahi toh koi verify nahi kar payega; process perfect ho par gain nahi toh finding yahi hai ki prompt hi kaafi tha — dono outcomes valid hain.

Pehle success criteria likho: Training shuru karne se pehle ek line mein likh do: "X metric pe baseline B ke against target T chahiye, off-task regressions zero." Criteria baad mein likhe toh koi bhi result "kaafi achha" lagne lagega.

Plan — paanch steps, ek order

CAPSTONE PIPELINE
1. task + failure  -> ek named task, metric, current prompted score
2. baseline        -> best prompt likho (1 ghanta), number note karo
3. data            -> spec -> examples -> clean -> leak-free split
4. train           -> LoRA run + config + dataset hash save
5. eval + decide   -> same held-out set, regressions, decision doc

har step ka artifact repo mein — ek bhi step skip = proof nahi

Step 1: task aisa chho jiska failure abhi measured ho — "tone bigadta hai" nahi, "10 mein se 4 replies format se bahar" haan.
Step 2: baseline ko utni hi mehnat do jitni tune pe — few-shot examples tak bhar do, warna strawman jeet ka kya maza.
Step 3: kuch saaf sau examples aksar kaafi hain — spec, cleaning aur split L13.2 wale yaad rakho.
Step 4-5: LoRA default (L13.4), eval protocol wahi jo L13.8 mein bana — naya mat socho, jo seekha wo lagao.

Timebox rakho: Pehla version chhota rakho: chhota task, chhota dataset, ek LoRA config. Complete hona zyada zaroori hai perfect hona se — iterate karna baad mein aasaan hota hai, khatam project se shuru karna mushkil.

Train karo aur honestly compare karo

RESULT TABLE (fill honestly — dono taraf same held-out set)
                      prompted-baseline   lora-tune
task metric            <number>           <number>
p95 latency            <number>           <number>
cost per 1k            <number>           <number>
off-task sample        pass/fail          pass/fail
failure notes          <2-3 cases>        <2-3 cases>

verdict: tune wins | tie (baseline hi rakho) | tune loses

Har candidate ko same held-out set, same judge, same rubric pe chalao — comparison ka protocol hi faisla credible banata hai.
Tuned model ke 10 individual outputs padho — average ke saath saath per-case behaviour, yahin nayi galtiyan milti hain.
Off-task sample zaroor chalao: general instructions, safety, unseen formats — capability given up ka proof yahin hai.
Adapter, base, dataset aur config ka release manifest banao — jo bhi verdict nikle, wo reproduce ho sake.

Tie bhi finding hai: Agar tune baseline se barabar ya neeche nikla toh "prompt kaafi tha" likhna sukoon hai — aur ek valid result. Har project har baar jeet nahita, par honest table har baar kuch sikhati hai.

Ship, scrap ya retrain — faisla likho

Tune wins: adapter versioned karo, gateway pe canary se launch karo, latency-cost table production mein bhi monitor karo.
Tie / lose: baseline ko hi rakho, decision README mein likho kyun — data iteration chahiye, ya task hi prompt se hal ho raha tha.
Failure analysis: jo cases tune ne miss kiye unhe label karo (format/tone/fact) — wo list hi agli dataset ka backlog hai.
Future triggers: naya base model aaye, input drift dikhe ya naye failure aayein — agli tuning tabhi, wahi eval gate ke saath.

DECISION README (repo mein visible)
what:     <task> ka LoRA vs best prompted baseline
result:   <before> -> <after> (same held-out set, same judge)
costs:    data + train + serve ka ownership hisaab
regress:  <kya bigda off-task>
verdict:  ship | keep prompting | retrain next
artifacts: data/v3, run-004, eval/report.pdf

# interviewer se yahi document tum dikhao ge

Portfolio wali baat: Interview mein ye project ise tarah sunao: measured failure → decision → data → train → honest eval → verdict. Numbers, artifacts aur verdict — teeno dikhein, tabhi fine-tuning samajh aaya hai ye proof hai.

Exercise

Question: Capstone ka success criterion kya hai — tune kis cheez ko beat karna chahiye?

Question: Training se pehle jo number note hota hai jise tune beat karna hai — wo kya kehlata hai?

Question: Final number hamesha kis set se aane chahiye jo training ke dauran kabhi khula nahi?

Question: Tune jeet na paaye toh faisla kya karna chahiye — chhupana ya repo mein likhna?

Common mistakes

Strawman baseline se jeetna: Kamzor prompt ko hara kar tune "safal" dikhana — production mein prompt wala hi chal raha hoga. Fix: baseline ko few-shot examples tak bhar kar best dena, uske baad hi compare.
Success criteria training ke baad decide karna: Jo number mila usi ko target bol diya — kabhi haar nahi lagti, par kuch sika nahi. Fix: target metric aur threshold likho pehle, training se pehle.
Data bina spec aur split ke utha lena: Ganda, leak-prone dataset se tune ka number hi jhooth bolega. Fix: behaviour spec, cleaning, leak-free split — L13.2 ka order hi follow karo, shortcut nahi.
Sirf target metric dekhna: Format aa gaya par off-task abilities bigad gayi — table mein wahi row missing thi. Fix: regressions, latency aur cost ka column table mein compulsory rakho.

Desi example: Mess ka naya menu trial

Mess committee naya menu laati hai — pehle purane menu ka feedback note hota hai (baseline), chhote batch pe naya trial chalta hai (train), phir usi rate pe usi taste-test se compare hota hai (eval), aur finally likha jaata hai ki menu badla ya nahi (decision):

purana menu + feedback = prompted baseline
chhota batch trial      = LoRA run (config save)
same taste test        = same held-out set + judge
verdict: badlo | wahi rakho = decision README

bina pehle ke feedback ke "naya better hai" = andaza

Interview angle — yahan se sawaal aate hain

Q: Fine-tuning project kaise start karoge? — Measured failure se, phir best prompted baseline, tabhi data aur training.
Q: Tune ka success kaise prove karoge? — Same held-out set, same judge pe baseline vs tuned ka table: quality, latency, cost, failures.
Q: Tune jeeta nahi toh kya karoge? — Decision document mein honestly likh ke deliver karunga: keep prompting ya data iteration — result chhupana nahi.
Q: Ek fine-tune release ko kaise describe karoge? — Adapter + base + dataset + config + template ka versioned bundle, eval report ke saath.
Q: Project ke baad retraining kab? — Triggers pehle se likhe: naya base model, input drift, naye failure cases — har naye version ko wahi eval suite gate karta hai.

Lesson complete?

Capstone complete — ab tum ek justified fine-tune ka poora lifecycle chala sakte ho: decision se lekar data, training, eval, serving aur retraining tak. L13 Fine-Tuning yahan poora hua — ab jo bhi tune karo, uske peeche number, artifact aur reasoning hogi.