Lesson 132 — Advanced

Beyond Text: What a Multimodal Model Is

Text LLM ke baad agla kadam ye hai ki model sirf shabd nahi, image, audio aur video bhi le sake. Is lesson mein hum dekhenge multimodal model hota kya hai, native model aur OCR-transcription pipeline mein farq kya hai, aur jo tumne tokenization, context window aur sampling seekha tha wo sab yahan kaise apply hota hai.

Multimodal model kya hai

Funda simple hai: image, audio, video — sab ko token sequence mein badal do, phir transformer wahi karta hai jo text pe karta hai. Isliye tumhara purana saara gyaan — context window, sampling, failure modes — yahan bhi lagta hai. Sirf ek naya step judta hai: modality se tokens tak ka conversion.

Native multimodal vs bolted-on pipeline

# Bolted-on pipeline: teen alag models jode gaye
def pipeline(page_image):
    text = ocr_model.read(page_image)        # layout, tone, tables bigad sakte hain
    transcript = stt_model.transcribe(audio) # sirf shabd bachte hain
    answer = text_llm.ask(text + transcript)
    return answer

# Native multimodal: ek model, mixed training
def native(page_image, question):
    return model.ask([{"type": "image", "image": page_image},
                      {"type": "text", "text": question}])

• Native: model ne training ke dauran image-text-audio mix dekha hai, isliye uske paas vision ka apna understanding hai — layout, spatial relation, tone sab token sequence mein survive karte hain.
• Pipeline: ek text LLM ke aage OCR ya transcription laga diya. Tez aur sasta ban sakta hai, par beech ka converter sirf wahi bachata hai jo wo samajh paata hai.
• Product pehchano: agar upload ki image turant ek text-only model ko jaati hai, toh wo pipeline hai — iska matlab har visual detail pehle hi kho chuki hogi.
• Fallback pattern: kai teams pipeline se shuru karti hain aur confidence kam ho toh native model ko bhejti hain — dono ka eval alag karo.

Text as the interlingua

Chaahe input image ho ya audio, transformer ke andar sab kuch token sequence hi hai, aur uski reasoning waise bhi text-like tokens pe hoti hai. Isliye output aksar text hi hai: tum image dete ho, jawab shabdon mein aata hai. Ye "text is the interlingua" wala idea hai — duniya ka har modality ek common numeric language mein translate hoti hai.

image  -> vision encoder  -> image tokens  \
                                          >-> one sequence -> transformer -> text tokens
prompt -> text tokenizer  -> text tokens  /

# Ek hi context window, ek hi attention — sirf token sources alag

• Image tokens aur text tokens ek hi sequence mein baithe hain — isliye tum sawaal text mein daal ke image se exactly wahi pooch sakte ho jo text LLM se poochte the.
• Kuch models audio ya image output bhi dete hain, par unke andar bhi aksar text ya latent tokens hi reasoning ka medium hote hain.
• Ye samajhne se debugging aasaan hoti hai: galat jawab aaya toh socho — token conversion mein loss hua ya model ne sahi tokens dekar galat interpretation ki.

Text LLM se kya transfer hota hai

• Context window: image tokens bhi usi window mein adjust hote hain — teen screenshots daal ke instruction bhool sakte ho.
• Tokenization & sampling: output ab bhi token-by-token banta hai, temperature/top-p waise hi lagte hain, hallucination waise hi possible hai.
• Failure modes: model confident hota hai bina jaane — image mein number dhundhna ho toh wo guess bhi kar sakta hai bina bataye.
• Prompting discipline: system instructions, structured output, few-shot — sab waise hi kaam karte hain, bas image bhi usi message mein daalo.
• Jo nahi aata: image ka "token count" alag hisaab se chalta hai, aur vision-specific failures (counting, small text) text ke failures se alag nature ke hote hain — agle lessons mein.

"Any-to-any" marketing padhte waqt savdhan rahiye: aaj ke zyada tar "multimodal" models kai inputs lete hain, par output text hi karte hain. Any-to-any ka sach ye hai ki wo shabd ka matlab sirf input side ki flexibility se hota hai, full duplex audio-video generation abhi bhi alag class ke systems hain.

Exercise

Common mistakes

• Multimodal ko alag "dimag" samajhna: Native model bhi ek transformer hai — naya sirf input encoder hai. Jo tumne tokenization, context window aur sampling seekha wo yahan bhi poora lagta hai. Fix: purana mental model reuse karo, naya sirf conversion step add karo.
• Har photo-upload feature ko native samajh lena: Zyada tar products ka upload ek pipeline hai — image pehle caption/OCR mein convert hoti hai, phir text LLM chalta hai. Fix: product docs dekho, ya dono tarah se test karke answers compare karo.
• Any-to-any ko literally lena: Marketing mein "any-to-any" likha hota hai par aksar output text hi hai. Fix: pehle output modality confirm karo — kya model image/audio bhi generate karta hai ya sirf leta hai.
• Text wale safeguards bhool jaana: Image ke andar bhi prompt injection likha ja sakta hai, aur vision output bhi hallucinate karta hai. Fix: same structured-output, validation aur abstention patterns yahan bhi laagu karo.

pipeline:  photo -> (chachi/pandit: text banao) -> LLM -> jawab
native:    photo + sawaal -> ek hi model -> jawab

# jo detail secretary ne nahi samjhi, wo kabhi file tak pahunchi hi nahi