Lesson 11 — Beginner

Inside an LLM: Embeddings, Attention & Transformers

Tumne ab tak tokens dekhe. Ab andar jaate hain — token ke ID se lekar embeddings, transformer layers, attention aur aakhirkar next-token probabilities tak. Bina matrix-math ke, sirf intuition level par. Saath mein jaanenge ki parameters ka size kya batata hai aur dense aur MoE model mein kya farq hota hai.

⏱ 30 min• Beginner• Tokenization

Pipeline ka overview — tokens se probabilities tak

WHAT

Transformer model ka poora andaruni structure hai — sequence of layers jismein har token ki representation uske context ke hisaab se har layer mein update hoti hai, aur end mein next-token ki probabilities nikalti hain. Yehi aaj ke laakho-cron ke LLM ka dhaancha hai.

WHEN

Har LLM call ke andar — chat ho, code generation ho, summarization ho — yehi pipeline chalta hai. Tum jab bhi API mein prompt bhejte ho, yehi andar chal raha hota hai, chahe tumhe dikhe na.

WHERE

Model ke andar. Bahar se tum sirf prompt dete ho aur jawab lete ho; andar ka poora khel — embeddings, attention, layers — chhupa rehta hai. Is lesson mein hum us darwaze ke peeche jhakenge.

Ek line ka mental model: Text → Tokens → Embeddings → Transformer layers (attention + feed-forward) → Next-token probabilities. Har stage apna kaam karti hai aur agle ko behtar representation deti hai. Isi sequence ko yaad rakhna, baaki sab usi ke details hain.

Yaad rakho: Transformer ek fixed-size ki machine hai — input jitna bhi ho, output hamesha ek distribution hota hai (agla token ki probabilities). Isi liye generation loop mein hum bar-bar is machine ko chala rahe hote hain (M1.5).

Token IDs se embeddings — arth ka numeric roop

Har token ka ID ek sirf number hai — usmein koi arth nahi chhupa. Embedding har token ko ek chhoti vector (sankhyaon ki list) mein badalta hai, jahan aas-paas wale arth wale shabdon ke vectors bhi paas-paas hote hain. Yahi wajah hai ki model ko shabd ka arth numerical form mein milta hai — arth bhi, relationships bhi, sab numbers mein.

Token: "delhi"  → ID: 84213  → embedding: [0.12, -0.45, 0.78, ...]  (hundreds of numbers)

Funda:  "rajdhani" ke vector ka "delhi" ke vector ke paas hona
 matlab — dono ka arth paas hai.

Yeh vector static nahi rehta: har transformer layer usse badalti hai,
kyunki ek hi shabd ka arth uske aas-paas ke shabdon se badalta hai.

Pro Tip: RAG mein documents ko vectors mein badal kar search kiya jaata hai (L4) — wahan bhi yahi idea chalega. L1 mein sirf itna pakad lo: embedding = arth ka numeric roop, aur ye context ke saath badalta rehta hai.

Attention — token doosre tokens ko kyun dekhta hai

Attention ka intuition ekdum simple hai: jab ek token apni representation bana raha hai, toh wo doosre relevant tokens ki taraf dekhta hai — dekh kar decide karta hai ki unka kitna asar mere arth par hai. "it" jaisa shabd tab hi samajh mein aata hai jab wo dekhe ki "it" kis cheez ko refer kar raha hai.

Sentence: "The animal didnt cross the street because it was tired"

Jab "it" apni representation bana raha hai (attention view):
  it → animal  : strong  (yahi referent hai — "tired" animal hai)
  it → street  : weak
  it → cross   : weak

Causal attention: har token sirf APNI LEFT ki tokens ki taraf dekh sakta hai —
right side ka future abhi bana hi nahi hai, isliye generation mein
har token sirf pehle wala context dekh kar aage badhta hai.

Queries, keys aur values: har token teen cheezein rakhta hai — ek query (main kya dhoondh raha hoon), ek key (main kya hoon) aur ek value (meri information kya hai). Query-key ka match attention score deta hai, aur us score ke hisaab se value milti hai.
Layers ka kaam: har transformer layer attention + feed-forward ka ek round hai. Pehli layers chhote patterns (grammar, punctuation) pakadti hain, aakhiri layers bade concepts (arth, intent).
Causal attention: generation mein har token sirf usse pehle wale context par depend kar sakta hai — baad wala text padh kar nahi, sirf pehle wala dekh kar.

Interview line: Ek token apni representation banane ke liye doosre relevant tokens ko dekhta hai (attention) — bina matrix-math ke, yahi asli intuition hai. Aur causal attention ka matlab hai sirf left-side context.

Parameters — model ke andar ke knobs

Parameters model ke andar ke laakho-crore chhote knobs hain — har ek ek sankhya. Inhe aise samjho ki model ne laakho sawaal-jawaab par training karke apne patterns ko in numbers mein likha hai. 8B, 70B, 400B+ — yehi sankhya parameters ki ginti hai.

Zyada parameters = zyada capacity: model utne hi complex patterns ko store aur combine kar sakta hai — lekin training aur inference ka kharcha bhi badhta hai.
Bigger is not automatically better: purane bade model se chhota naya model (behtar data aur post-training ke saath) aksar behtar performance de sakta hai. Sirf size dekh kar decision mat lo.
Parameter count ek capacity indicator hai, quality ka guarantee nahi: quality ke liye benchmark aur apne task pe test chahiye (M1.11, M1.12).

Dense vs Mixture-of-Experts

Dense model mein har parameter har token par kaam karta hai. MoE (Mixture-of-Experts) mein model ke feed-forward blocks ko bohot se experts mein tod diya jaata hai, aur ek router har token ke liye sirf chune hue experts ko chalta hai. Baaki experts us token ke liye sote rehte hain.

DENSE (70B):
  Har token → poore 70B parameters activate → zyada compute per token

MoE (8x7B jaisa):
  Har token → router chunta hai → sirf kuch experts on → active parameters chhote

Model card mein dikhta hai:
  "total parameters: (badi sankhya)"    → memory ismein
  "active parameters: (chhoti sankhya)"  → compute ismein

MoE ka faayda: total parameters zyada (zyada knowledge), par per-token compute chhota — isliye dense ke muqable sasta aur tez chal sakta hai.
MoE ka kharcha: sab experts ko memory mein rakhna padta hai — GPU/RAM toh badi chahiye, chahe compute chhota ho.

MoE ko practice mein kaise padhein

Model card padhne ka tareeka: "8x7B" ka matlab — 8 experts, har expert 7B parameters. Host karne wale ko total size dekhna chahiye, speed aur cost ka estimate active parameters se.
Total vs active: total parameters memory batate hain, active parameters per-token compute. Isi farq se samajh aata hai ki MoE itna bada hone ke baad bhi sasta kyun chal jaata hai.
MoE ko free upgrade mat samjho: chalane ke liye memory badi chahiye, aur quality dense ke muqable alag-hoti hai — apne task pe test karo, size par trust mat karo.

Decision rule: MoE tab chuniye jab latency aur cost ka pressure ho aur memory budget ho — warna dense model ka behaviour zyada predictable hota hai. Model card par hamesha total aur active parameters dono padho.

Exercise

Question: MoE mein har token ke liye sirf kuch experts ko chunne wale component ka naam kya hai?

Question: Causal attention mein ek token apni left ya right side dekh sakta hai? (ek shabd)

Question: Dense aur MoE mein se kaunsa model har token par poore parameters ko active karta hai?

Question: Total aur active parameters mein se MoE ka kaunsa chhota rehta hai?

Common mistakes

Bigger parameters = better model samajhna: Size sirf capacity hai, quality nahi. Chhota naya model behtar data aur training se bade purane model ko hara sakta hai — comparison karo, size pe blind trust mat karo.
Embedding ko fixed dictionary entry samajhna: Vector static nahi — har transformer layer context ke saath usse badalti hai, isi liye ek hi shabd ka arth badal sakta hai. Embedding dynamic representation hai, ratta hua meaning nahi.
MoE ko chalne mein sasta samajhna: Compute active parameters se hota hai, par memory total se — MoE host karne ke liye GPU badi chahiye. Faayda sirf per-token speed aur cost mein hai, RAM mein nahi.

Desi example: Chhat wala teacher — attention ka desi analogy

Class mein teacher poochhti hai "usne kyun roka?" — aur bachche peeche mur kar dekhte hain ki "us" kis cheez ka hai. Token bhi yahi karta hai: apni representation banate waqt doosre relevant tokens ki taraf dekhta hai (attention), aur causal rule hai ki peeche hi dekh sakta hai, aage nahi:

Teacher: "The animal didnt cross the street because it was tired"
Bachcha "it" par ruk gaya → peeche dekha → "animal" mile → samajh gaya

Token "it" → attention → animal (strong), street (weak)
Aur haan — future words abhi board pe likhe hi nahi hain (causal attention).

Interview angle — yahan se sawaal aate hain

Q: Embedding kya hoti hai? — Token ka numeric roop — sankhyaon ki vector — jahan arth wale shabdon ke vectors paas hote hain; har layer mein context se update hota hai.
Q: Attention ka intuition kya hai? — Ek token doosre relevant tokens ko dekh kar apni representation banata hai; Q/K/V scores se decide hota hai kitna asar lena hai.
Q: Dense aur MoE model ka farq kya hai? — Dense mein har token par poore parameters chalte hain; MoE mein router sirf chune hue experts ko chalata hai — active parameters kam, memory phir bhi zyada.
Q: 8x7B ka matlab kya hai? — 8 experts, har expert 7B parameters; total parameters memory, active parameters per-token compute batate hain.
Q: Kya bada model hamesha behtar hai? — Nahi — capacity zyada hoti hai, par training data, post-training aur task fit bhi utna hi maayne rakhta hai.

Lesson complete?

Andar ka khel clear — embeddings, attention, layers aur MoE. Agla lesson — How LLMs Learn — mein dekhenge pretraining se SFT aur RLHF tak, aur samjhenge ki pretrained model chat assistant kaise banta hai.