Lesson 10 — Beginner

Tokenization

Model kabhi text nahi padhta — wo numbers ki list hi padhta hai. Text ko un numbers mein badalne ka kaam tokenizer karta hai. Is lesson mein hum dekhenge tokens hote kya hain, tokenizer alag-alag bhashaon ko alag tarah kyun todta hai, aur tumhari bill (cost) seedha token count se kyun jud jaati hai.

⏱ 10 min• Beginner• What an LLM Actually Is

Characters, words aur tokens

WHAT

Token text ka chhota tukda hai — aksar ek poora chhota shabd, ya bade shabd ke hisse. Har token ke paas ek token ID hoti hai, aur model sirf inhi IDs par chalta hai. Tokenizer woh translator hai jo text ki lines ko ID numbers mein badalta hai.

WHEN

Jab tum prompt bhejte ho tab input tokenize hota hai; jab model jawab deta hai tab har naya token bhi usi vocabulary se hi chunta hai. Har request, har generation — tokenizer kaam kar raha hota hai.

WHERE

API ke input aur output dono ke billing mein, context-window ke hisaab mein, aur streaming ke output pieces mein — token hi currency hai. Tum har jagah token ka count dekhoge.

Teen cheezein alag-alag hain: characters (ek-ek akshar), words (space se alag shabd), aur tokens (model ke liye unit). Ek token ek word nahi hota — aam English chhote shabd aksar ek hi token hote hain, jabki lambi ya durlabh shabd multiple tokens mein toot sakti hain. Model ke liye asli unit token hai, word nahi.

Yaad rakho: Model ke andar vocabulary fixed hoti hai — har possible token ke liye ek ID. Naya shabd aaye toh tokenizer use us vocabulary ke known tukdon mein tod dega. Isliye token count aur word count kabhi same nahi aate.

Tokenizer kaise todta hai — bhasha, code, numbers

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

samples = {
    "english":  "The quick brown fox jumps over the lazy dog",
    "hindi":    "Namaste duniya, aaj mausam bahut sundar hai",
    "devanagari": "namaste duniya — aaj mausam bahut sundar hai",
    "code":     "def calculate_total(items): return sum(i.price for i in items)",
    "numbers":  "48217362519",
}

for name, text in samples.items():
    ids = enc.encode(text)
    print(name, "chars=", len(text), "tokens=", len(ids))
    print(" first ids:", ids[:12])

Common English: chhote aam shabd aksar ek token mein fit ho jaate hain — isliye English ke tokens words ke kareeb hote hain.
Durlabh shabd: technical ya rare shabd, ya typo wale shabd — multiple tokens mein tootte hain, aur spelling bhi model ke liye mushkil ho jaati hai.
Numbers: ek lamba number aksar alag-alag digit-pieces mein tootta hai — isliye model ko maths mein dikkat hoti hai (agle lessons mein iska asar dekhoge).
Code: code ke tokens bhi uske patterns se hi bante hain — indentation, brackets, function names sab tokens hote hain.
Emoji & symbols: har emoji bhi vocabulary ka ek token hota hai — kabhi akela, kabhi groups mein.
Hindi (Devanagari): training corpus mein Hindi text kam tha, isliye aam Hindi shabd bhi chhote-chhote pieces mein tootte hain — English sentence ke muqable Hindi mein tokens zyada lagte hain.

Experiment karo: Upar wala Python snippet apne machine pe chalao (pip install tiktoken) — ek English paragraph, uska Hindi translation, thoda source code aur ek number list daalo. Token counts khud dekh kar samajh aa jaayega ki bhasha aur content se farq kitna padta hai.

Token seedha cost aur latency se judta hai

API providers tumhe har request ke input tokens (jo tumne bheje) aur output tokens (jo model ne likhe) ke hisaab se bill karte hain. Pricing pages per-token rate batate hain — exact aaj ka rate wahan check karo, yahan purani sankhya yaad karke mat chaliyo.

# Cost ka khel — API call se pehle estimate nikalo
input_text = open("contract.txt").read()          # bada document
prompt = "Is contract ki 5 key terms nikaalo:\n" + input_text

# Pehle apne tokenizer se input size naapo
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
in_tokens = len(enc.encode(prompt))
out_tokens_estimate = 150                          # chhota structured jawab

# Phir provider ki pricing page se rate utha kar:
# cost = (in_tokens / 1000000) * input_rate
#      + (out_tokens_estimate / 1000000) * output_rate
print("input tokens:", in_tokens, " | est output:", out_tokens_estimate)

Input vs output: dono bill hote hain, aur output wale aksar zyada rate ke — isliye "chhota rakho jawab" ek cost-control hai, sirf style nahi.
Context consumption: har token context window bhi khaata hai (agle lesson). Lamba document = tokens zyada = cost zyada, aur kabhi-kabhi limit bhi cross.
Latency: output tokens bante-bante time lagta hai — lambi output hamesha deri se aati hai (M1.5 mein decode ka khel).
Hindi ka hidden bill: wahi baat Hindi mein likhi ho toh tokens zyada — translation karke English mein bhejna kabhi-kabhi sasta bhi pad sakta hai.

Tokenization ke common traps

Character count = token count samajhna: rate-limiters ya field validators agar characters par bane hain toh text ke hisaab se galat baithte hain — token hi asli unit hai.
Spelling mistakes: typo wale shabd zyada tokens mein tootte hain, aur model ki spelling bhi bigadti hai — isliye codes aur IDs ke saath model ko careful rakhna padta hai.
Maths ke liye numbers: "48217362519" ko digit-pieces mein tod kar model ko add karana mushkil hai — exact calculation LLM ki kamzori hai, calculator/tool lagao.
Token budget ke bina prompt likhna: poora database prompt mein thoons doge toh cost aur latency dono phat jaayenge — chhote, targeted context se kaam chalao.

Decision rule: Jab bhi tumhara product token-heavy ho — lamba document upload, chhota frequent classification call, ya batch pipeline — tokenizer se pehle count naap lo. Token count hi tumhara panchami (cost, latency, context) ka seedha sootra hai.

Exercise

Question: Ek normal English word aksar kitne tokens mein aata hai — ek ya bahut zyada?

Question: API billing ka basic unit kya hai? (ek shabd)

Question: Model ko input dene se pehle text ko numbers mein badalne wale component ka naam kya hai?

Question: Code likhne ke liye language ka naam jo hum tokenizer example mein use karte hain?

Common mistakes

Characters ko tokens samajhna: Ek character aur ek token kabhi same nahi — rate limits, cost estimate aur field validation sab token par karo, character par nahi. Count nikalne ka tareeka: tokenizer API se encode karke len.
Hindi ko sasta samajhna: Hindi text aksar zyada tokens leta hai kyunki training corpus mein Hindi kam thi — cost aur context dono badi. Zaroorat na ho toh English prompt se bhi kaam chal jaata hai.
Exact maths ko LLM pe chhod dena: Numbers digit-pieces mein tootte hain, isliye calculations mein gadbad aam hai. Paise, totals aur dates ke liye tool/calculator call karo — LLM sirf decide karega ki kya calculate karna hai.

Desi example: Doodh wale ke glass — tokenization ka desi analogy

Doodh wale ke paas ek tanki hai aur woh doodh glass mein baant kar deta hai — tokenizer bhi yahi karta hai: text ki tanki ko fixed-size ke tokens (glasses) mein baant-ta hai. Common shabd ke glass chhote hain, Hindi ke bade tukde:

Tanki: "quick brown fox"        → glass: [quick] [brown] [fox]
Tanki: "Namaste duniya"        → glass: [Nam] [aste] [duni] [ya]  (zyada glasses)
Tanki: "48217362519"           → glass: [482] [173] [625] [19]   (tukdon mein)

Bill glasses ke hisaab se — tokens ke hisaab se. Hindi ke zyada
 glasses = zyada bill aur zyada jagah (context window).

Interview angle — yahan se sawaal aate hain

Q: Character, word aur token mein farq kya hai? — Character ek akshar, word space se alag shabd, token model ki unit; aksar chhota shabd ek token, par badi ya rare shabd multiple tokens.
Q: Do sentences jiske words barabar hain, unke tokens kyun alag aate hain? — Tokenizer vocabulary ke patterns se todta hai; shabd ki frequency aur language ke hisaab se tukde badal jaate hain.
Q: Hindi aur English sentence ka token count alag kyun hota hai? — Hindi (Devanagari) training corpus mein kam tha, isliye aam Hindi shabd bhi chhote tukdon mein tootte hain — tokens zyada.
Q: Token count seedha kis cheez se judta hai? — Cost (per-token billing), context window consumption aur latency (output tokens ban-ne mein time) — teeno se.

Lesson complete?

Tokenization clear — ab tum samajhte ho ki text numbers mein kaise badalta hai aur bill kaise banti hai. Agla lesson — Inside an LLM — mein dekhenge in tokens ke andar embeddings, attention aur transformer layers kaise kaam karte hain.