Lesson 133 — Advanced

Vision: How Images Become Tokens

Ek image model ke andar jaate hi tokens mein tootti hai, aur is tootne ka tareeka hi decide karta hai ki image kitni mehngi padegi aur kya-kya dikh paayega. Is lesson mein hum patches, tiles, detail settings, resolution-based token cost aur full-image-vs-crops ke trade-off ko ek chhote Python example ke saath samjhenge.

⏱ 10 min• Advanced• Beyond Text: What a Multimodal Model Is

Patches aur vision encoder

Kya hai?

Vision encoder image ko ek grid mein kaat-ta hai — chhote squares jinhe patches kehte hain. Har patch ko ek vector (embedding) mein badla jaata hai, aur ye vectors hi transformer ko milte hain, jaise words ke token embeddings milte hain.

Kab kaam aata hai?

Jab bhi tum image upload karo — screenshot, invoice, product photo. Tumhara har visual request pehle is patch-grid se guzarta hai, aur isi hisaab se token bill banta hai.

Kahan milta hai?

Multimodal API ke andar (vision encoder), ya open vision models mein jo locally chalte hain. Tumhe directly encoder chalane ki zarurat nahi — par uska hisaab samajhna zaroori hai.

Model image ko "pixels ke raste" se nahi dekhta — wo grid ke squares dekhta hai. Isliye jitni high resolution, utne zyada patches, utne zyada tokens. Ek hi photo ko zoom karke bhejna ya chhota karke bhejna, dono ka matlab alag hai.

Yaad rakho: Patch ek fixed physical area cover karta hai. Badi image mein same object chhote pixels mein baith jaata hai — isliye resolution badhane se detail milti hai, par bill bhi badhta hai. Dono hamesha saath chalte hain.

Ek image kitne tokens ki?

# Token cost resolution (pixels) se chalti hai, file size (MB) se NAHI
def image_token_cost(width, height, patch=14, tiles=1):
    """Approx model — apne provider ki docs se exact formula verify karo."""
    tokens_per_tile = (224 // patch) ** 2     # ek tile ke andar ke patches
    return tokens_per_tile * tiles            # high-detail = zyada tiles

print(image_token_cost(512, 512))            # chhota image = kam tokens
print(image_token_cost(2048, 2048))          # bada screenshot = bill triple

Resolution-driven: ek hi photo JPEG mein compress karke 2 MB kar doge toh tokens nahi ghate — pixels kam karne padenge.
Hundreds ya thousands: ek detailed screenshot aksar hundreds of tokens kha jaata hai — kuch screenshots daalte hi aadhi context gayi.
Same context window: image tokens wahi window use karte hain jahan instructions aur history hain. Chaar full screenshots = model ka poora working memory.
Batch uploads: "ye 10 photos dekh ke batao" poochhne se pehle socho — 10 × token cost context mein baithega, aur phir sawaal ke liye jagah bachegi kya.

Rule of thumb: Har image upload se pehle do sawaal: (1) kya is image ka har pixel chahiye, ya sirf ek region? (2) kya model ko poori resolution chahiye, ya thumbnail kaafi hai? Ye do sawaal aksar aadha bill bacha dete hain.

Tiling aur detail settings

# High-detail mode: badi image = tiles ka grid, har tile apne tokens
request_hi = {
    "type": "image_url",
    "image_url": {"url": "invoice.png", "detail": "high"}
}
request_lo = {
    "type": "image_url",
    "image_url": {"url": "invoice.png", "detail": "low"}   # ek fixed chhoti tile
}
# low: sirf "isme kya hai" wala gist  |  high: chhota text, tables, numbers bhi

High detail: image ko tiles mein kaat kar har tile ko alag tokens mein bhejta hai — zyada detail, zyada cost.
Low detail: poori image ek fixed chhoti tile mein squeeze hoti hai — sasta, par chhota text gaya.
Cropping: ek badi image ki jagah sirf relevant region crop karke high-detail mein bhejo — detail bhi milti hai, bill chhota rehta hai.
Aspect ratio: lambi files (poori website screenshot) mein bahut saare tiles sirf whitespace ke hain — pehle crop, phir upload.

Decision rule: Scene samajhna hai (kya photo mein hai?) → low detail. Chhota text, invoice number, chart label padhna hai → high detail, aur usse bhi behtar — crop karke bhejo.

Downscaling aur full image vs crops

Downscale kya uddaata hai: chhota text, thin lines, dense tables, chart ke labels — sab pehle jaate hain. Model ko blur-patches milte hain, aur wo phir bhi confident jawab de sakta hai (fabrication, agla lesson).
Full image: tab jab poore scene ka context chahiye — "ye warehouse hai, shelf pe kya hai".
Crops: tab jab specific value chahiye — invoice ka total, form ka field, screenshot ka error message.
Overview + crops: sabse achha combo — ek low-res full image se layout samjho, phir har region ka crop high-detail mein bhejo. Model ko bolo ki kaunsa crop kis page/region ka hai.

Practical workflow: pehle ek overview bhejo, model se pucho ki kaunsa region matter karta hai, phir usi region ka high-res crop bhejo. Ye do-step approach zyada tar use cases mein ek bada full-res image bhejne se sasti aur accurate banti hai.

Pitfall: Downscale karke chhota text "padhwana" sabse kharab pattern hai — model text ka shape dekh kar guess kar lega, aur guess hamesha plausible dikhta hai. Chhota text chahiye toh crop + high detail, kabhi bhi full-image-shrink nahi.

Exercise

Question: Image ko chhote grid squares mein todne wale pieces ka technical naam kya hai?

Question: Image ki token cost uske file size (MB) se chalti hai ya resolution (pixels) se?

Question: Badi image ko chhote tiles mein kaat kar bhejne wale detail mode ka naam?

Question: Image tokens aur text tokens kis cheez ko share karte hain model ke andar?

Common mistakes

Poori photo seedha upload kar dena: Camera ki full-resolution photo aksar zyada tokens kha jaati hai jahan sirf ek region matter karta tha. Fix: pehle crop karo, ya low-detail overview + high-detail crop ka combo bhejo.
Compress karke tokens bachane ki koshish: File size (MB) token bill ko nahi badalta — pixels badalte hain. Fix: resolution kam karo, format nahi; aur chhota text kabhi downscale karke mat padhwao.
Bahut saari images ek saath bhej dena: Har image apne tokens ke saath context window mein baithti hai — instructions aur history squeeze ho sakte hain. Fix: zaroorat ke hisaab se crop bhejo, ya images ko alag-turn mein process karo.

Desi example: Tile wala photographer

Ek bada hoarding shoot karna hai. Poori photo kheenchoge toh printer ko bhi utna hi area dena padega (full-res = zyada tokens). Aur agar hoarding ke ek corner ka rate list padhna hai toh usi corner ka close-up kheench lo — crop: zyada detail, kam kharcha:

full hoarding photo  -> sab dikh par chhota text gaya
rate-list ka close-up -> text clear, area chhota, bill chhota

# overview + targeted crops = best of both

Interview angle — yahan se sawaal aate hain

Q: Vision model image ko kaise dekhta hai? — Fixed grid mein patches, har patch ka apna embedding, phir wahi transformer.
Q: Token cost kis se scale karti hai? — Resolution (tiles/patches) se, file size se nahi. Detail setting isko control karti hai.
Q: High vs low detail kab? — Low: scene gist, sasta. High: chhota text, tables, numbers.
Q: Ek bada screenshot chhota karke bhejne se kya hota hai? — Chhota text, thin lines aur labels kho jaate hain; model phir bhi guess kar sakta hai.

Lesson complete?

Image-token ka economics clear — ab tum crop aur resolution ka trade-off consciously choose kar sakte ho. Agla lesson — Vision: Capabilities & Failure Modes — mein dekhenge model visual mein kya acche se karta hai aur kahan galat confident hota hai.