Lesson 12 — Beginner

How LLMs Learn

Ek model jo internet jaisa text padh kar aata hai, phir bhi aksar bekaar assistant ki tarah behave karta hai — aur usi ko banane wale banate hain chat model. Is lesson mein hum dekhenge pretraining mein model kya sikhta hai, self-supervision ka funda kya hai, aur SFT, RLHF aur preference tuning ke baad wo assistant ban jaata hai jo tum use karte ho.

⏱ 20 min• Beginner• Inside an LLM: Embeddings, Attention & Transformers

Pretraining — self-supervision ka khel

WHAT

Pretraining mein model ko internet-scale text par next-token prediction ke kaam se train kiya jaata hai. Har sentence mein ek hissa chhupaya jaata hai, aur model se uska guess maanga jaata hai — guess galat toh model ke andar ke numbers update hote hain. Yahi self-supervision hai: label bahar se nahi, text khud deta hai.

WHEN

Model banate waqt, ek hi baar — pretraining sabse pehle aur sabse mehenga step hai. Iske baad ke steps (SFT, preference tuning) usi model par chalte hain, alag se naya model nahi bante.

WHERE

Base model — jo model abhi sirf agla shabd deta hai, instruction follow nahi karta. Providers se milne wale models mein kai base versions aise hi hote hain, jinke upar post-training karke chat version banaya jaata hai.

Pretraining ka ek chhota sa glimpse:

Text   : "Mumbai ki raiti sabse famous"
Chhupa : "…"
Guess  : [famous: 0.4, achhi: 0.3, mehngi: 0.2 …]
Galat  → loss → model ke andar ke numbers (weights) update

Yahi loop internet-scale text par crore baar = PRETRAINING
Model ne sirf grammar nahi — facts, code syntax aur reasoning ke
patterns bhi isi se sikhe.

Yahi hai self-supervised learning ka matlab — koi insaan alag se label nahi deta. Text ka apna agla shabd hi uska label hai, isliye duniya ka likha hua text hi training data ban jaata hai. Isi vajah se pretraining kabhi rukta nahi — naya text aaya, naya signal mil gaya.

Pretraining se kya nahi aata

Instruction-following nahi: model ne sirf text ka continuation dekha hai — "summarize karo" likho toh wo usse bhi ek aur line samajh kar uska continuation likh sakta hai, summary nahi.
Format aur tone ka control nahi: bina post-training ke model kabhi list, kabhi essay, kabhi bakwaas likh deta hai — koi fixed style nahi.
Boundaries nahi: harmful request par rok nahi hai — model toh bas agla probable token bana raha hai, uska koi irada nahi.

Yahi wajah hai ki pretrained model ke paas knowledge bohot hoti hai, phir bhi wo ek bekaar assistant hota hai. Usko chat banane ke liye agla step chahiye — post-training. Interview mein yahi sabse common sawaal hai: pretrained model itna jaanta hai, phir bekaar kyun? — jawab hai post-training.

Post-training pipeline — base se assistant tak

POST-TRAINING PIPELINE (simplified)

1. PRETRAINING          → internet-scale text, next-token prediction
2. SFT                  → curated instruction → answer examples
3. PREFERENCE / RLHF    → achhe aur bure jawaab rank karke reward seekhana
4. SAFETY TUNING        → harmful requests par refuse, boundaries
5. DEPLOYED ASSISTANT   → wahi model jo chat app mein dikhta hai

Har step usi base model par chalta hai — naya model nahi banta.

SFT (Supervised Fine-Tuning) mein model ko logon ne likhe hue achhe instruction-jawaab pairs dikhaye jaate hain — "aise jawaab dene hain". Phir RLHF (Reinforcement Learning from Human Feedback) mein do jawaab ban kar insaan se rank karwaya jaata hai, usse ek reward bana kar model ko us direction mein tune kiya jaata hai. RLAIF mein yahi feedback AI se liya jaata hai. Preference optimization in sab ka thoda alag tareeka hai — sab ka maqsad ek hi hai: model ko dikhana ki achha jawaab kaisa hota hai.

Yaad rakho: RLHF se model ka knowledge nahi badhta — uska behaviour badhta hai. Facts pretraining se aate hain, tone aur obedience post-training se. Isliye purane facts wale model ko RLHF se naya fact nahi milega.

SFT aur preference tuning — details

SFT: "yeh prompt, yeh jawaab" wale pairs — chhote par bohot saaf dataset se. Isi se model sikhta hai ki instruction ko jawab dena hai, continuation nahi.
RLHF / preference tuning: do jawaab bana kar rank karwao, phir model ko bhi wahi pasand sikhao. Yahi step model ko polite, structured aur helpful dhang mein laata hai.
Safety tuning: kuch request par refuse karna, kuch par bolna ki jankari kam hai — yeh bhi isi post-training ka hissa hai, baad mein model-guardrails bhi judte hain (L8).

Pro Tip: Jab bhi koi naya model mile, model-card mein padho ki uske liye kya post-training hua — instruction-tuned hai, RLHF hai, safety tuning hai ya nahi. Yehi bataata hai ki wo base model hai ya chat assistant.

Chat model aisa kyun behave karta hai

Instruction follow karna, clarification poochhna, structured output dena — sab post-training ki den hain, model ka apna gun nahi.
Hedge karna ("shayad", "lagta hai") aur refuse karna — bhi trained behaviour hai. Model ka koi irada nahi, bas usi dhang ke examples par train hua hai.
Phir bhi boundaries probabilistic hain: chat model safe nahi ho jaata — jailbreak aur prompt injection jaise tricks se ched jaata hai (L8 Security level mein detail).

Interview line: Ek pretrained model bahut jaanta hai, par assistant nahi — SFT, preference tuning aur safety tuning usse usable banate hain. Yahi farq hai base model aur chat model mein, aur yahi jawab interview mein sabse zyada jaana jaata hai.

Exercise

Question: Pretraining ka asli kaam kya hai? (do-teen shabd)

Question: Self-supervised learning mein training signal (label) kahan se aata hai?

Question: RLHF mein insaan sabse pehle kya karta hai? (ek shabd)

Common mistakes

Pretraining ko ratta samajhna: Model facts ratta nahi — patterns seekhta hai, isliye wo naye, pehle na dekhe sawaal bhi solve kar leta hai. Aur hallucination bhi isi wajah se sambhav hai — pattern continuation truth verification nahi hai.
Base model ko assistant samajhna: Base model continuation deta hai, instruction follow nahi. Use karne se pehle check karo ki model chat/instruction version hai ya base — kai providers dono bechte hain, naam milta-julta hai.
RLHF se knowledge improve samajhna: Preference tuning tone aur obedience badhati hai, facts nahi. Purane facts wale model ko naya fact chahiye toh context, retrieval ya tools lagao — fine-tuning se nahi.

Desi example: Kaccha sabzi se tayyar dabba — training ka desi pipeline

Ghar mein khana banne ka process hi poora training pipeline hai — sabzi mandi se lekar feedback tak. Har step usi kacche maal (base model) ko aage badhata hai:

Sabzi mandi (pretraining)       : duniya bhar ki sabzi + pakane ka andaza
Recipe book (SFT)              : "aise banao" ke examples dikhana
Ghar walon ka feedback (RLHF)  : "ye achha tha, wo thoda namkeen"
Tayyar dabba (chat model)      : sab kuch mil kar — assistant ban gaya

Interview angle — yahan se sawaal aate hain

Q: Pretraining aur fine-tuning ka farq kya hai? — Pretraining next-token prediction par bada text corpus; fine-tuning specific behaviour (instruction following) ke curated examples par.
Q: Self-supervised learning kya hai? — Label bahar se nahi milta, text ka apna hissa hi label ban jaata hai — chhupa hua shabd guess karwao, guess galat toh weights update.
Q: Pretrained model itna jaanta hai phir bekaar assistant kyun hai? — Usne sirf continuation seekha hai — instruction following, format aur boundaries post-training (SFT + RLHF) se aate hain.
Q: RLHF ka matlab kya hai? — Log generated jawaab rank karte hain, usse reward bana kar model ko preference seekhata hai; RLAIF mein yahi feedback AI se liya jaata hai.
Q: Chat model polite kyun hota hai? — Post-training mein use isi dhang ke examples dikhaye gaye hain; model ka koi irada nahi, behaviour training ka result hai.

Lesson complete?

Training ka poora rasta clear — pretraining se SFT aur RLHF tak. Agla lesson — How LLM Inference Works — mein dekhenge tumhara prompt bhejne ke baad server ke andar prefill, decode aur KV cache kaise chalte hain.