Lesson 13 — Beginner

How LLM Inference Works

Prompt bhejte hi tumhe jawab dikhne lagta hai — par us chhote se gap ke andar do alag phases chalte hain. Is lesson mein hum dekhenge prompt bhejne ke baad andar kya hota hai, prefill aur decode ka farq, TTFT aur tokens-per-second kaise pade jaate hain, aur KV cache kya hai jo inference ko sasta banata hai.

⏱ 10 min• Beginner• How LLMs Learn

Prompt bhejne ke baad — kya hota hai

WHAT

Inference ka matlab hai trained model ko chalana — prompt dena aur token generate karwana. Ek request ke andar do phase hote hain: pehle poora input process karna (prefill), phir naye token ek-ek bana-na (decode).

WHEN

Har API call, har chat message, har batch job — jahan bhi tum model se jawab mangte ho, inference hi chal raha hota hai. Streaming ho ya non-streaming, phases wahi dono hain.

WHERE

Provider ke servers par (ya apne GPU par). Tum sirf request bhejte ho aur jawab lete ho — scheduling, batching aur caching ka poora khel server-side chalta hai.

Flow ekdum seedha hai: Prompt → input process → predict token → generate token → repeat. Naya token bante hi wo wapas input ka hissa ban jaata hai, aur agla prediction isi badhe hue context par hota hai. Generation isi tarah chalta hai jab tak stop na aa jaaye — aur purani mehnat bachane ka naam hai KV cache.

Prefill vs decode — do alag phases

REQUEST = [SYSTEM + HISTORY + USER PROMPT]        (input tokens)

PHASE 1  PREFILL
  Poora input ek saath process hota hai (parallel, tez)
  Yahan se time-to-first-token (TTFT) banta hai

PHASE 2  DECODE
  Naya token ek-ek banta hai, har token poore
  context par predict hota hai (sequential, dheere)
  Streaming isi phase se dikhne lagta hai

Prefill compute-heavy hai par parallel ho sakta hai — isliye lamba input TTFT badhata hai.
Decode sequential hai — har step ka output agle step ka input. Isi liye output length ka asar latency par seedha padta hai.
Batches mein scaling: providers ek GPU par laakho requests ke decode steps bhi mix karke chalate hain — throughput isi se banti hai (frameworks ka detail L2 mein).

TTFT, tokens per second, total latency

Teen numbers har inference discussion mein aate hain — aur teeno ek doosre se jude hue hain. Inhe samajh lo, koi bhi inference report padh loge.

TTFT (time to first token): pehla token aane mein kitna waqt — chat experience ka sabse zaroori number, prefill ki speed se banta hai.
Tokens per second: decode phase ki speed — jawab kitni tezi se stream hota hai.
Total latency: poora jawab ban-ne mein kul waqt — TTFT plus baaki decode ka time. Input tokens ki wajah se input cost, output tokens ki wajah se time aur cost dono badhte hain.

Debugging tip: Jab app dheere lage toh pehle poochho — deri prefill mein hai (lamba prompt, lamba TTFT) ya decode mein (lambi output, kam tokens-per-second)? Alag bottlenecks hain, alag ilaaj bhi.

KV cache — purani mehnat dohrana nahi

Har naye token ke liye poora context dobara process karna bekaar mehnat hogi — jo tokens ek baar ban chuke hain, unki attention keys aur values fix rehti hain. Inhe save karke rakhna hi KV cache hai. Isliye decode mein har step sirf naya token process karta hai, purana nahi.

BINA KV CACHE:
  Token 100 ke liye tokens 1–99 DOBARA process — bar-bar ki bekaar mehnat

KV CACHE (default har serious inference server par):
  Purane tokens ki key/value vectors yaad rakhi jaati hain
  Naya step = sirf NAYA token process + cache lookup

Isi se decode fast hota hai — aur lambi conversation ki
memory (cache) bhi server ki cost banti hai (L6 mein caching).

Yaad rakho: KV cache ka matlab hai tumhari conversation server par memory gher rahi hai. Lambi history = bada cache = zyada memory — yahi ek wajah hai ki providers purani history trim ya cache-policy lagate hain.

Inference mechanics seedha cost aur latency se jude hain

Lambi output = zyada decode steps = zyada waqt aur zyada cost. "Chhota jawab do" sirf style nahi, performance ka command hai.
Lamba prompt = lamba prefill = TTFT badhta hai, aur har request ki input cost bhi badhti hai — dono taraf kharcha.
Reasoning models andar bhi tokens banate hain (thinking) — unka TTFT aur cost jawab dikhne se pehle hi badh jaata hai (M1.7).

Decision rule: User-facing chat mein TTFT priority hai; batch pipeline mein cost. Dono jagah output length cap karo — default limit se chhota rakho jab tak kaam chal jaaye.

Exercise

Question: Input ko ek saath process karne wala inference phase kya kehlaata hai?

Question: Naye token ek-ek bana-ne wala phase kya kehlaata hai?

Question: Pehle token tak ka lag-time metric kya hai? (acronym bhi chalega)

Question: Purane tokens ki key/value vectors ko yaad rakhne wale cache ka naam?

Common mistakes

Lambi output ko sirf cost ka issue samajhna: Latency bhi badhti hai — decode sequential hai, har token ka waqt seedha output length se judta hai. User-facing chat mein yahi chhupi hui sabse badi frustration banti hai.
Prefill aur decode ka farq na samajhna: Lamba prompt TTFT badhata hai, lamba output total time. Dono alag bottlenecks hain — debugging mein pehle dekho deri kis phase mein hai, phir wahi thik karo.
KV cache ko free samajhna: Cache memory ghera hai — lambi conversation par server ko memory pay karni padti hai. Purani history trim karna ek real optimization hai, sirf politeness nahi.

Desi example: Chai wale ka order — prefill aur decode ka desi analogy

Prefill = order samajhna (ek baar, sab ek saath), decode = ek-ek biscuit nikalna (chhota chhota kaam, baar-baar). Bade order mein pehla step thoda waqt leta hai, phir har biscuit alag-se — aur jitne zyada biscuits, utna zyada total time:

Order aaya (prefill)      : "4 kulhad chai, adrak wali" → sab ek saath taiyaar
Biscuit nikalna (decode) : 1 … 2 … 3 … 200 …        (ek-ek karke)
Pehla sip (TTFT)         : order ke turant baad
Poora order (latency)    : 200 biscuit baad

Interview angle — yahan se sawaal aate hain

Q: Prefill aur decode ka farq kya hai? — Prefill input ko ek saath process karta hai (parallel, TTFT yahin se); decode naye token ek-ek banata hai (sequential).
Q: KV cache kyun hota hai? — Purane tokens ki key/value fix rehti hain; dobara compute ki zarurat nahi, isliye decode fast aur sasta — aur lambi conversation par memory cost banti hai.
Q: 2000 output tokens 50 ke muqable kyun zyada time lete hain? — Decode sequential hai — har token ek step — isliye time seedha output length ke saath chalta hai.
Q: TTFT kis phase se aata hai? — Prefill se; input jitna lamba, prefill utna lamba, TTFT utna zyada. Isliye bada context hamesha tez nahi hota.

Lesson complete?

Inference ka mechanical hissa clear — prefill, decode, KV cache. Agla lesson — Sampling & Generation Parameters — mein dekhenge probability distribution se token kaise chunta hai aur temperature, top-p jaise knobs kya karte hain.