Lesson 158 — Intermediate

Reinforcement Learning Foundations for LLMs

Is lesson ka mental model ek line hai: Act → observe reward → update policy — aur RLHF usi loop ka avtar hai jismein reward insaan ke bajaye ek learned reward model deta hai. Bandit se shuru honge (ek decision, immediate reward), phir MDP (sequence aur credit assignment), phir wahi frame language pe, aur aakhir mein RLHF/PPO ka intuition jahan har cheez ka jagah tay hoti hai — policy, reference model, KL penalty aur DPO ka bridge.

RL ka frame — reward se policy tak

Sabse pehle soch lo ki problem ka shape kya hai: ek decision hai ya sequence? Ek decision aur immediate reward = bandit. Sequence jismein aakhri reward ko peeche tak assign karna hai = MDP. Language ke liye jawab ek sequence hai — isliye RLHF mein MDP frame ghusta hai, aur bandit isi liye seekha jaata hai kyunki wo uska chhota version hai.

Bandits — explore vs exploit ka sabse chhota khel

import random
# do arms (jaise do prompt variants) — asli means A=2, B=5, agent ko nahi pata
Q, N = {"A": 2.0, "B": 3.0}, {"A": 10, "B": 2}
eps = 0.2

if random.random() < eps:
    arm = random.choice(list(Q))     # explore — kam jaane wala arm try karo
else:
    arm = max(Q, key=Q.get)          # exploit -> B (3.0 bada hai)

# regret: best se kitna chhoot gaya
# 10 rounds tak A chunta raha (2 milte) toh regret = (5 - 2) * 10 = 30

• Bandit kyun: ek hi decision baar baar lena hai (kaunsa banner, kaunsa reply, kaunsa model route) aur har choice ka immediate reward milta hai — state ka track nahi rakhna padta.
• Epsilon-greedy: eps probability se random (explore), baaki purana best (exploit). Sabse simple version — paper pe 10 round chalao, samajh aayega ki balance kahan hai.
• UCB ka idea: uncertainty ko score mein jod do — kam explore kiya hua arm pehle aata hai. Andha exploration nahi, soch samajh kar.
• Regret intuition: best-known se har round kitna chhoot gaya. Explore band karna tab hota hai jab regret curve flat ho jaati hai.
• Products mein kahan: traffic allocation (kis variant ko), copy testing, model routing (sasta model pehle, mushkil case bade ko) — sab bandit-style decisions hain.

MDP basics — state, action, reward aur credit assignment

# MDP ka chhota episode: support chat resolve karna
gamma = 0.9
steps = [
    ("state: user angry",   "action: acknowledge",  "reward: 1"),
    ("state: clarified",    "action: ask order id", "reward: 0"),
    ("state: resolved",     "action: refund",       "reward: 1"),
]
# return (discounted): G = 1 + 0.9 * 0 + 0.9^2 * 1
G = 1 + gamma * 0 + gamma ** 2 * 1
print(G)          # 1.81
# policy = state -> action ka rule ; value = us state se hone wala expected return

• Paanch cheezein: state (abhi kya hai), action (kya kar sakte ho), reward (us step ka feedback), policy (action chunne ka rule), value (is state/action se aage expected return).
• Episodic vs continuing: episode ka end ho (chat resolve, game khatam) ya system chalta rehe (ads serving) — dono ka credit assignment alag mushkil hai.
• Credit assignment hi asli sawaal: reward aakhri step pe mila — kaunsa action deserve karta hai? Value functions isi ko hal karte hain, aage ka expected return hisaab mein laate hue.
• Discount gamma: kal ka reward aaj se kam. 0.9^10 ≈ 0.35 — long horizon mein abhi ke action ka credit dheere dheere ghista hai, aur exploration aur mushkil hoti hai.

Language ke liye RL kyun — cross-entropy ka limit

Next-token cross-entropy sikhata hai ki training data mein agla token kya tha — "is sawaal ka accha jawab" ka judgement nahi. Ek jawab par poore sequence ka ek hi grade aata hai (helpful tha / nahi), aur us grade ko har token tak pahunchana wahi credit-assignment problem hai jo MDP mein dekhi. Isliye RL tab aata hai jab quality ka label pairwise ya sequence-level ho.

• Answer = sequence of actions: har token ek action, poora answer ek episode, uska grade reward — yahi framing RLHF ko SFT se alag banati hai.
• Imitation vs judgement: SFT copy karta hai ("data jaisa"); RL grade ke against policy adjust karta hai ("is sawaal ke liye behtar").
• Preference data sasta pada: sahi/galat label likhne se aasan hai do jawab mein behtar chunna — judging easier than writing, isi se reward models bane.
• Order matter karta hai: SFT pehle (basic competence), phir preference/RL (quality tuning). Seedha RL pe jaana policy ko collapse karwata hai.

Reward model — preference pairs, scorer aur hacking

preference pair (reward model ka training data):
{
  "prompt":   "Summarise this ticket in 2 lines",
  "chosen":   "Refund pending since 3 days; escalated to payments.",
  "rejected": "Sure! Here is a detailed summary: this ticket is about
               a refund request ... (9 lines of filler)"
}
# scorer seekhta hai: chosen wale ko zyada reward, rejected ko kam
# phir RL isi score ko maximise karta hai -> yahin hacking shuru hoti hai

• Reward model kya hai: preference pairs se train ek scorer jo prompt ke saath output ko number deta hai. Wo number hi aage ka target ban jaata hai.
• Hacking 1 — verbosity: judge ne lambi jawab ko behtar maana tha toh model lamba likhne lagega — score up, asli helpfulness flat.
• Hacking 2 — flattery/sycophancy: "bilkul sahi aapne!" wali tone judgement mein aksar jeeti hai, accuracy nahi — model user ki haan mein haan milata hai.
• Hacking 3 — format tricks: markdown headers aur bullet list jo rubric ke keywords hit karein, content khaali ho.
• Pakadne ka tareeka: length penalty, diverse rubric, aur sabse zaroori — reward se ALAG holdout task metric. Reward up par task metric flat = hacking chal raha hai.

RLHF loop & PPO intuition, phir DPO ka bridge

# RLHF step ka ek line ka economics
rm_score = 3.2   # reward model ne is jawab ko diya
beta     = 0.1   # base behavior se door jaane ka daam
kl       = 1.5   # policy vs reference model ka difference
total    = rm_score - beta * kl
print(round(total, 2))        # 3.2 - 0.15 = 3.05

# PPO-style loop (spirit):
#   sample answer -> RM score -> advantage (is answer ne average se kitna behtar)
#   -> policy update (clipped: ek step mein itni hi door jaao)
#   -> reference model wahi frozen SFT model hai, KL usi se measure hota hai

• Reference model: base (SFT) model ki frozen copy — policy ko uske paas rakhne ka anchor. Door nikal gaya toh tone, format aur quality base jaisi rehna band.
• KL penalty (beta): zyada = safe le boring, kam = creative le drift. Ye hi "mat bhool jaao tum kya the" wala line hai — training ka sabse important safety dial.
• Advantage bina math ke: A = is action ka result minus us state ka average expectation. Matlab "ye jawab average se kitna behtar tha" — credit proportional milta hai, poora nahi.
• PPO ka clip karta kya hai: ek update mein policy kitni door ja sakti hai uska cap — overshoot aur collapse se bachata hai. Formula yaad mat rakho, spirit rakho.
• DPO ka bridge: preference pairs se seedha loss — alag reward model nahi, alag online sampling nahi, PPO ka machinery nahi. Wahi idea (behtar upar, kharab neeche), kam moving parts. Detail L13 — Preference Tuning & DPO wale lesson mein.

Exercise

Common mistakes

• Exploration ko paisa barbaad samajhna: Bina explore ke pata hi nahi chalega ki behtar option tha bhi ya nahi — regret chhup jaata hai. Fix: chhota fixed fraction explore rakho, aur bandit ko fixed A/B split se compare karo.
• Reward model score ko ground truth maanna: Wo proxy hai — verbosity, flattery aur format tricks se usko fool kiya ja sakta hai. Fix: independent holdout task metric + human spot checks, reward ke saath saath.
• KL penalty ko waste samajh kar zero kar dena: Policy base se door nikal jaati hai — tone drift, degenerate output. Fix: beta ek safety dial hai; usko sirf tab hataao jab base behaviour hi chahiye nahi ho.
• SFT ke bina seedha RL pe chala jaana: Policy ke paas shuruaati competence nahi, toh random reward signals use collapse karwa dete hain. Fix: order — SFT pehle, preference/RL baad mein.

eps = 0.2
order = random_menu() if random.random() < eps else best_seller

# stall ka feedback seedha hai (sales); RLHF mein:
reward   = waiter_score(answer)          # learned scorer
adjusted = reward - 0.1 * recipe_change  # beta * KL — purani recipe ke paas