Lesson 149 — Advanced

Preference Tuning & DPO

Kuch behaviours ek "sahi jawab" se nahi, balki "ye usse behtar hai" se define hote hain — tone, helpfulness, refusal, concision. Is lesson mein hum demonstration aur preference ka farq, pairs kaise bante hain, rater agreement aur length bias jaise traps, DPO ka intuition bina RL ke maths, aur SFT ke baad DPO kyun chalta hai dekhenge.

⏱ 20 min• Advanced• LoRA & QLoRA

Demonstration vs preference

Kya hai?

Preference data mein har example ke teen hisse hote hain: ek prompt, ek preferred response aur ek rejected response. Training ka target ek hi answer dikhana nahi, balki preferred wale ko rejected se zyada likely banana hai.

Kab kaam aata hai?

Jab desired behaviour ek single correct answer mein likhna mushkil ho — tone, helpfulness ka level, refusal ka dhang, jawab ki chawta. "Iska ek hi sahi answer" nahi hota, par "ye usse behtar" bilkul clear hota hai.

Kahan milta hai?

Human raters, apne eval judges, production feedback (thumbs up/down, edits) se bane pairs mein — phir DPO jaisi training libraries ko diye jaate hain.

Sawaal simple hai: do jawab hain, tumhe sirf bolna hai kaunsa behtar — perfect jawab likhne ki zarurat nahi. Isliye preference data annotate karna aksar aasaan hota hai, aur "quality" jaise vague maamlo mein bhi rater aaram se decide kar leta hai.

Order yaad rakho: Pehle SFT (model ko basic demonstrations se baithao), phir DPO (us checkpoint ko preferences se polish karo). Preference tuning aksar SFT ko replace nahi karti, uske baad chalti hai.

Preference pairs kaise bante hain

ek preference pair (JSONL):
{
  "prompt": "Refund kab tak milega?",
  "chosen":  "Order check karke batata hoon — aam taur par 5-7 working days lagte hain.",
  "rejected": "Refund ke baare mein koi guarantee nahi de sakta, aage ka support se baat karo."
}

dono jawab SAME prompt pe, aur farq SIRF us quality mein hona chahiye
jise tum train kar rahe ho (tone, concision, helpfulness) — baaki sab same

Sources: human raters (slow par reliable), tumhare apne eval judges (fast par unki bias inherit), production feedback (real par rare aur noisy).
Rater agreement check karo: same pair pe do raters kitni baar agree karte hain? Agreement low toh guideline hi clear nahi — pehle rubric theek karo, phir data badhao.
Length bias: agar chosen hamesha lamba aur rejected hamesha chhota hota hai, model "lamba = behtar" seekh jaata hai. Lamba-achha aur chhota-achha dono rakh kar balance karo.
Irrelevant differences mat rehne do: formatting, extra examples, politeness filler — jo quality tumhe chahiye uske alawa dono responses mein same hona chahiye.
Rejected ko zaroor rakho: ye negative signal hi DPO ki taakat hai — sirf chosen bhej dena SFT ho jaayega.

Pehle 50 pairs khud banao: Chhota batch banao, do logon se blind rating karwao, agreement dekho. Rubric baith gayi toh scale karo — bina rubric ke scaled data sirf noise hai jo model ko galat lesson sikhayega.

DPO intuition — bina RL ke maths

# DPO ka dil (bina reinforcement learning ke formalism ke):
# model se chosen aur dono ke log-likelihood lo

def dpo_margin(logp_chosen, logp_rejected):
    return logp_chosen - logp_rejected

# margin > 0  -> chosen already zyada likely, pair theek hai
# margin < 0  -> model galat ko zyada pasand kar raha, loss zyada
# training: loss ko chhota karke margin ko positive zone mein push karo
# ek hi objective: preferred ko relative mein upar, rejected ko neeche

SFT aur DPO ka farq: SFT kehta hai "ye jawab likho"; DPO kehta hai "do jawabon mein se ye zyada likely ho". Do alag signals, do alag capacities.
Reference model: DPO ek reference (usually tumhara SFT checkpoint) ke against relative mein shift karta hai — isliye base behaviour thoda sa reh jaata hai, wipe nahi hota.
Hyperparameter ka ek knob: strength ka wala parameter decide karta hai ki preference signal ko kitna follow karna hai — zyada strong = pairs par zyada overfit ho sakte ho.
Data size ka khayal: DPO ko sau-double-sau pairs se achha chalne wala milta hai — hazar nahi chahiye, saaf pairs chahiye.

DPO ka failure mode: Ganda pair = galat direction. Agar chosen mein hi galti hai ya farq hi irrelevant hai, model usi direction mein shift hoga aur tumhe pata bhi nahi chalega. Pair quality hi DPO ki training quality hai.

SFT ke baad DPO — pipeline kaise dikhti hai

stage 1: SFT        examples -> checkpoint_sft
stage 2: DPO        pairs (chosen/rejected) + checkpoint_sft -> checkpoint_dpo
stage 3: EVAL       base vs sft vs dpo, same held-out set

# kyun SFT pehle:
# - DPO ko ek achha starting point chahiye, warna pairs
#   bas model ko thoda-thoda khinchte hain, naya skill nahi sikhate
# - SFT bina model abhi bhi format galat likhta hai, DPO usse nahi sikhata

Preference tuning kis liye: tone, helpfulness, refusal behaviour, concision — wo cheezein jinka koi ek "gold answer" nahi hota.
SFT kab kaafi hai: jab tumhare paas ek hi sahi jawab hai aur format strict hai — wahan pairs banaane ka kharcha bekaar hai.
Dono compare karo: final decision ke liye base, SFT-only aur SFT+DPO teeno ko same eval set pe chalao — DPO ka value isi se prove hota hai.
Feedback loop: production mein thumbs down aaye unhe rejected candidate banao — par review zaroor, warna user preference ki noise seedhi training mein jaayegi.

Blind judge se check karo: DPO ke baad outputs ka A/B karo jise pata na ho kaunsa version hai. Judge ko naam dikhoge toh bias aa jaayega — blind comparison hi batata hai ki preference tuning ne asli farq kiya ya nahi.

Exercise

Question: Ek preference example mein kitne response hote hain — chosen aur rejected? Sirf number daalo.

Question: Preference pairs mein "ye behtar hai" ke saath "ye worst hai" bhi chahiye — us doosre response ko kya kehte hain?

Question: DPO aksar kis training stage ke baad chala jaata hai, usko replace karke nahi?

Question: Pairs jahan chosen hamesha lamba aur rejected hamesha chhota ho — model kya galat lesson seekhta hai? (ek shabd)

Question: Do raters same pair par kitni baar agree karte hain — us measure ko kya kehte hain?

Common mistakes

Pairs mein irrelevant farq rehna: Dono jawab alag-alag length/format mein hain toh model wahi seekhega, desired quality nahi. Fix: sirf target quality chhodo, baaki sab same rakho.
Bina rater agreement ke large data annotate karna: Rubric hi clear nahi toh data noise hai — model confuse hoga. Fix: pehle chhota batch, do raters, agreement dekho, phir scale.
DPO ko SFT ka substitute samajhna: Bina SFT base abhi format bhi galat likhta hai — preference signal usse nahi sudhaarta. Fix: SFT checkpoint = DPO ka starting point, hamesha.
Chosen mein hi chhupi galti: Reviewer ne jo approved usi ko gold maan liya — training direction galat. Fix: chosen ko bhi spec check se guzaro, judge se blind review karwao.

Desi example: Do rasoiyon ki thali — andha taste test

Khane ki competition mein do thaliyan aati hain, judge ko pata nahi kaun kisne banayi — sirf bolna hai kaunsi behtar. Preference tuning bhi yahi hai: perfect recipe likhne ki zarurat nahi, do ke beech faisla karna hai — aur judge ki poori thali dekhni hai, sirf mirch nahi:

prompt:   "menu ke liye 2 lines"
thali A:  chosen   -> sahi tone, chhota, kaam ka
thali B:  rejected -> lamba bhara hua, tone alag

DPO: A ko B se zyada likely banao
check: andha judge -> kaunsi behtar? (blind A/B)

Interview angle — yahan se sawaal aate hain

Q: Demonstration data aur preference data mein kya farq hai? — Demonstration ek sahi jawab dikhata hai; preference ek prompt ke do jawab deta hai jinmein ek behtar hai.
Q: DPO kya achieve karta hai (bina RL ke)? — Preferred ko rejected ke relative mein zyada likely karta hai, reference model se shift karke.
Q: Length bias kaise banti hai aur kaise bachein? — Chosen aksar lamba ho toh model lamba = behtar seekhta hai; pairs balance karke aur irrelevant farq hata ke.
Q: Kab SFT hi kaafi hai? — Jab ek hi gold answer wala narrow format task ho; tone/refusal/concision jaise vague qualities pe hi preference chahiye.
Q: DPO ke baad kaise prove karoge ki value aayi? — Base vs SFT vs SFT+DPO ko same held-out set pe, blind judge ke saath compare karke.

Lesson complete?

Preference tuning clear — ab tum tone aur quality jaise vague behaviours pairs se train kar sakte ho. Agla lesson — Reinforcement Fine-Tuning (RFT) — mein dekhenge reward/grader wale approach ki duniya.