Lesson 14 — Beginner

Sampling & Generation Parameters

Model ke paas agla token ki poori probability list hai — par list se akhir mein token choose kya hota hai? Yahan tumhara control aata hai: temperature, top-p, top-k, penalties aur stop sequences. Is lesson mein hum dekhenge har knob kya karta hai, aur task ke hisaab se kaunsi setting chalni chahiye.

⏱ 10 min• Beginner• How LLM Inference Works

Probability distribution se token tak

WHAT

Sampling ka matlab hai model ki next-token probability distribution se ek token choose karna. Greedy decoding = hamesha sabse zyada probable token; sampling = distribution ke hisaab se chance bana kar. Temperature aur top-p isi khel ke controls hain.

WHEN

Har generation mein — default settings hote hain. Par structured output, extraction aur creative writing jaise tasks par tum inhe chhedna chahoge, warna defaults tumhara kaam kharab bhi kar sakte hain.

WHERE

API call ke parameters mein — temperature, top_p, max_tokens, stop, penalties. Providers inhe alag-alag naam se bechte hain, kaam ka principle ek hi hai.

Sabse pehle greedy decoding samjho: model hamesha sabse zyada probable token chunta hai — koi chance nahi. Yehi temperature zero ke kareeb ka khel hai. Sampling tab hai jab probability ke hisaab se chance bana kar token chuna jaaye — kabhi yeh shabd, kabhi woh. Creativity isi se aati hai, galti bhi isi se.

Parameters ka toolbox

from openai import OpenAI

client = OpenAI()

resp = client.chat.completions.create(
    model="your-model-name",
    temperature=0.2,        # low = focused, high = creative
    top_p=1.0,              # nucleus sampling ka head
    max_tokens=200,         # output cap — cost aur latency control
    stop=["END"],           # yeh marker aate hi ruk jaao
    messages=[{"role": "user", "content": "Ticket ka category: BILLING ya TECHNICAL?"}],
)
print(resp.choices[0].message.content)

Temperature: low (0 ke kareeb) = top choices par focus; high = distribution flatten, surprise badhta hai. 1 ke upar aksar bakwas shuru hoti hai.
Top-p (nucleus): sirf top tokens rakho jinki cumulative probability p tak hai — 0.9 matlab us 90% wale head se hi choose karo.
Top-k: sirf top k candidates se choose karo — chhota fixed set. Kuch providers isko expose karte hain, kuch nahi.
Penalties (repetition/frequency/presence): same token baar-bar dohrane se rokta hai — long generation mein loop se bachaat.
Max tokens aur stop: output ki lambai cap karo ya fixed marker par roko — structured pipelines ke liye zaroori.

Task ke hisaab se randomness

Kam randomness chahiye (0 ke kareeb): extraction, classification, JSON output, factual QA — jahan ek hi sahi jawab hai.
Zyada randomness chahiye: brainstorming, creative writing, diverse alternatives — jahan variety hi maqsad hai.
Default se shuru karo, phir apne task pe test karke adjust karo — andhere mein temperature ghumana sabse common beginner mistake hai.

Decision rule: Tum code likh kar output check kar rahe ho (exact JSON, exact label) — randomness low. Output logon ko dikh raha hai aur variety chahiye — thoda upar. Jab tak zarurat na ho, default se mat hato.

Har API ek jaisa nahi hota

Ek baat yaad rakho: har API sab knobs nahi deta. Kuch providers top-k bechte hain, kuch nahi; kuch ke penalties alag naam se hote hain. Aur reasoning models mein aksar temperature direct control nahi hota — unka andar ka thinking budget alag parameter se set hota hai (M1.7).

Trap: Temperature zero = output hamesha identical — guaranteed nahi. Server-side optimizations ke chalte chhoti variation aa sakti hai. Testing mein exact-string match se bacho, structure aur rubric se check karo.

Exercise

Question: Sabse zyada probable token hamesha chunne wale decoding ka naam kya hai?

Question: Factual QA aur structured extraction ke liye temperature zyada chahiye ya kam?

Question: Top-p sampling candidate set ko kis cheez se limit karta hai? (ek shabd)

Question: Output ki lambai cap karne wala parameter kya kehlata hai?

Common mistakes

Temperature bina samjhe badalna: Default se shuru karo. Andhere mein zyada temperature = variety nahi, bakwas; bahut kam = focus nahi, repetition. Har setting ka maqsad task se juda hona chahiye.
Temperature zero par bhi hamesha same output ki guarantee samajhna: Bilkul identical output ki guarantee sampling se nahi — infrastructure ki wajah se chhoti variation aa sakti hai. Tests mein rubric ya structure check karo, exact string nahi.
Stop sequences aur max_tokens chhod dena: Bina cap ke model lamba khinch sakta hai — cost aur latency dono phati hai. Structured pipelines mein stop marker ekdum zaroori hai, warna parsing hi nahi hoga.

Desi example: Pani puri wale ka swaad — sampling ka desi analogy

Probability list pani puri wale ki demand list hai — har flavour ki popularity. Temperature kam = hamesha sabse popular flavour do; temperature zyada = kabhi naya combo try karo. Top-p = sirf top flavours rakho, baaki menu se hata do:

Demand: [papdi: 0.5, teekha: 0.3, meetha: 0.2]
Temperature LOW   → hamesha papdi (greedy ke kareeb)
Temperature HIGH  → kabhi teekha, kabhi meetha (surprise)
Top-p = 0.6       → sirf papdi tak ka menu — meetha bahar

Interview angle — yahan se sawaal aate hain

Q: Greedy decoding kya hai? — Har step par sabse zyada probable token chunna, bina kisi randomness ke; temperature zero ke kareeb ka hi khel.
Q: Temperature kya karta hai? — Probability distribution ko shape deta hai — low par top tokens par focus, high par surprise aur creativity.
Q: Top-p aur top-k ka farq? — Top-p cumulative probability ke head se candidates limit karta hai; top-k fixed k count se. Dono ka maqsad chhota candidate set banana hai.
Q: Extraction task ke liye kya settings chahiye? — Low temperature, max-tokens cap aur stop sequence — predictable structured output ke liye teeno zaroori.

Lesson complete?

Sampling ke knobs clear. Agla lesson — Reasoning Models & Test-Time Compute — mein dekhenge kuch models sochne mein extra compute kyun kharch karte hain aur kab tumhara paisa wahin sahi jagah lagega.