Lesson 150 — Advanced

Reinforcement Fine-Tuning (RFT)

RFT mein model ko sahi jawab dikhane ke bajaye score ke through reward milta hai — aur wo score tumhara khud ka grader deta hai. Is lesson mein hum reward-based tuning ka mechanism, RFT vs SFT vs DPO ka farq, kaunse tasks RFT ke layak hain, grader design aur reward hacking, aur practical entry points dekhenge.

⏱ 10 min• Advanced• Preference Tuning & DPO

Reward-based tuning kya hai

Kya hai?

RFT mein har model output ek grader (automatic scorer) se score hota hai, aur training us direction mein weights push karti hai jahan score badhe. Gold jawab ka example nahi chahiye — scoring function hi target ban jaata hai.

Kab kaam aata hai?

Jab output apne aap check ho sake — code jo tests pass kare, math jiska answer verify ho, structured output jiske fields validate hon. Jab tak score reliable aur automatic na ho, RFT ka sawaal hi nahi uthta.

Kahan milta hai?

Aksar managed fine-tuning services ya specialized RL frameworks mein, API ke through — kyunki rollouts, reward bookkeeping aur model stability ki kaafi machinery chahiye hoti hai.

Mechanism simple hai: model ne output banaya → grader ne score diya → us score ke against weights thode badle. Model sirf ek attempt pe nahi rukta — explore karke behtar score wale behavior ki taraf khinchta hai. Isliye jin tasks mein "try karke sahi kar sakta hai", wahan RFT ki asli taakat hai.

Rule of thumb: Grader likhna training shuru karne se pehle ka sabse important kaam hai. Grader jo maanega, wahi model sikhayega — chahe wo tumhara asli goal ho ya sirf uska proxy. Proxy aur goal alag hue toh training bhi alag disha mein chalegi.

RFT vs SFT vs DPO — kya seekhata hai kya

SFT   -> examples (gold jawab dikhao)     "ye likho"
DPO   -> pairs (chosen vs rejected)       "ye usse behtar"
RFT   -> reward (grader ka score)         "score badhao"

RFT ka edge: model ek attempt pe nahi rukta — explore
karke behtar score wale behavior ki taraf seekhta hai,
khas kar tab jab sahi raasta pehle se saaf nahi hai

par: grader hi poora curriculum hai — ganda grader = ganda model

SFT: sabse stable aur sasta — direct demonstrations. Format aur narrow behaviour isi se baithta hai, foundation bhi yahi hai.
DPO: comparison ka signal — chosen vs rejected. Rollout ki zarurat nahi, chhote pairs pe achha chalta hai.
RFT: scoring ka signal — best-of-many exploration milta hai, par compute zyada aur grader ka risk sabse bada.
Order aksar yahi hai: SFT pehle (base banao), phir DPO (polish), RFT tabhi jab task gradable ho aur baaki dono kam pade.

Interview line: SFT demonstrations se seekhta hai, DPO comparisons se, RFT rewards se. Teenon ka data type alag hai — isliye pehle decide karo tumhare paas examples hain, pairs hain, ya ek scoring function hai.

Kaunse tasks RFT ke layak hain

def grade(output: str, item: dict) -> float:
    if not parses_json(output):
        return 0.0                          # format hi nahi = zero
    ans = extract_number(output)
    if item.get("unit") and "unit" not in output.lower():
        return 0.3                          # partial credit
    return 1.0 if close(ans, item["answer"]) else 0.0

# reliable: har jagah same verdict
# automatic: koi human click nahi chahiye
# gaming-proof: chhota sa loophole = poora reward hack

Acche candidates: code jiske paas tests hain, math/logic jiska answer check hota hai, structured outputs jiske fields validate hon, extraction jismein gold fields known hon.
Kharab candidates: open-ended writing, tone, "kya ye fact sahi hai" jahan knowledge chahiye — ye SFT/DPO ke zone hain, RFT ke nahi.
Partial credit do: sirf 0/1 grader se learning dheere chalti hai — thoda-thoda score gradient jaisa kaam karta hai.
Determinism rakho: grader ka output same input pe same hona chahiye, warna reward ki direction hi badalti rehti hai.

Grader ko pehle fool karo: Training se pehle grader ko haath se chhup chhup kar do — aise outputs banao jo score achha lein par galat hon. Mil gaya loophole? Grader theek karo, tabhi training start. Ye 30 minute ka adversarial test aadhe reward-hacking disasters rok deta hai.

Reward design aur reward hacking

GANDA GRADER (aise mat banao)
score = len(output)                     # lamba = zyada score
-> model seekh gaya: repeat-repeat-repeat -> bhara hua output

THEEK GRADER
score = AND(json_valid, fields_present, answer_match, tests_pass)
# partial credit bhi cap karo, hard-fail pe zero

yaad rakho: grader ka version bhi release hai —
grader badla = objective badla = dobara eval

Reward hacking matlab: model grader ko optimize karta hai, asli objective ko nahi — score upar, kaam wahi ka wahi.
Proxy vs goal: "lamba = detailed", "keywords = relevant" jaise shortcuts grader mein likh do toh model wahi shortcut seekhega.
Score distribution monitor karo: training ke dauran reward achanak perfect hua toh grader mil gaya, kaam nahi.
Grader ko versioned rakho aur har reward change ke baad evaluation dobara chalao — objective badal gaya toh purana comparison invalid hai.

Grader = syllabus: School mein jo exam pattern hota hai wahi poora class wahi padhta hai — RFT mein bhi grader hi syllabus hai. Grader ka har check tumhara ek implicit instruction hai, isliye likhte waqt socho ki model is line se kya lesson lega.

RFT practical mein kaise chalta hai

Managed service ya specialized framework: rollouts, reward tracking, stability tricks aur compute scheduling sab built-in — isliye aksar yahin se entry hoti hai.
DIY from scratch = research project: policy updates, exploration, variance control — khud likhna semester-long kaam hai, production shortcut nahi.
Compute SFT se zyada: har prompt pe multiple rollouts ban-te hain — ek example ek baar nahi, baar baar sample hote hain.
Data ka shape badal jaata hai: ab tumhe gold jawab nahi, prompts + grader chahiye — dono pehle se tested hone chahiye.

Entry point: Pehla RFT project chhota rakho: ek gradable task, ek chhota prompt set, ek grader jise tum haath se fool kar chuke ho. Managed API se shuru karo — infra ke bajaye objective design pe waqt kharch karo, wahin asli mushkil hai.

Exercise

Question: RFT mein training signal kis cheez se aata hai — examples, comparisons ya reward?

Question: Model output ko automatic score karne wala function kya kehlata hai?

Question: Model grader ke loophole ko exploit karke asli goal bhool jaata hai — uska naam? (ek shabd)

Question: Math jiska answer automatically check ho sakta hai — aise outputs ko kya kehte hain? (gradable / open)

Common mistakes

Grader bina test ke trust karna: Chhota loophole = model poora reward games karega aur score dekh kar sab khush. Fix: training se pehle grader ko haath se fool karo, loopholes patch karo, grader ko version banao.
Ungradable task pe RFT lagana: Tone ya open-ended writing pe reliable automatic score likhna hi mushkil hai — training noise par chalegi. Fix: gradable task chuno, warna SFT/DPO se jao.
Proxy score ko asli objective samajhna: Lamba output, keyword match — grader ka shortcut hi model ki aadat ban jaata hai. Fix: har scoring rule ka soch samajh ke justification likho, distribution monitor karo.
RFT ko SFT ka replacement samajhna: Bina SFT base abhi format galat likhta hai — reward usse nahi sudhaarta. Fix: order rakho — SFT (foundation) → DPO (polish) → RFT (gradable deep tasks).

Desi example: Length wale checker ka exam

Agar teacher ke marking mein sirf lamba answer zyada marks de, toh poori class lamba-chauda likhne lagti hai — content peeche reh jaata hai. RFT mein bhi yahi hota hai: jo grader measure karta hai, wahi model sikhata hai — isliye checker ka rubric hi asli curriculum hai:

checker = lambai?   -> class: repeat karke bhara do (hack)
checker = sahi+saaf? -> class: content pe kaam karo (goal)

RFT: grader ka rubric hi model ka syllabus hai

Interview angle — yahan se sawaal aate hain

Q: RFT aur SFT ka core farq? — SFT gold examples dikhata hai; RFT outputs ko score karke reward ke direction mein push karta hai.
Q: RFT aur DPO ka farq? — DPO do jawabon ka comparison leta hai; RFT ek grader ka score, jismein exploration aur rollouts hote hain.
Q: Reward hacking kyun hoti hai aur kaise rokein? — Grader proxy optimize hota hai; haath se fool-test, capped partial credit, grader versioning se.
Q: Kaunse task RFT ke liye sahi hain? — Automatically, reliably grade hone wale: code with tests, checkable math, verifiable structured outputs.
Q: RFT aksar khud nahi banate — kyun? — Rollouts, reward bookkeeping aur stability ki machinery bhaari hai; managed service ya specialized framework se entry sasti hai.

Lesson complete?

RFT ka frame clear — ab tum grader-first soch sakte ho: objective likho, grader test karo, tabhi train karo. Agla lesson — Distillation — mein dekhenge ek bade model ki capability ko chhote model mein kaise transfer kiya jaata hai.