Lesson 134 — Advanced

Vision: Capabilities & Failure Modes

Vision model ek chart dekh kar uski kahani bata deta hai — par uski kahani hamesha sahi nahi hoti. Is lesson mein hum dekhenge kaunse visual tasks model bina help ke acche se karta hai, kaunse mein wo systematically fail hota hai, aur wo failure "pata nahi" bolne ke bajaye confident invented jawab kyun deta hai — aur phir safeguards kaise lagate hain.

Vision models kahan acche hain

• Text-in-image: signs, banners, screenshots, printed forms — badhiya reading, multiple languages mein.
• Scene & object description: "ye ek crowded market hai, stalls hain, logo ne masks pehne hain" — gist bilkul theek.
• Named field extraction: invoice ya form mein "invoice number kya hai" jaise sawaal — field dhundh ke value de deta hai.
• Coarse chart reading: "sales badh rahi hai", "Q3 sabse achha tha" — trend level pe achha; exact value nahi.

Failure modes jo baar baar dohraate hain

# Typical failure probe — har naye model pe apna test set chalao
tests = [
    ("shelf.jpg",   "kitne blue boxes hain?",        "counting"),
    ("geometry.png", "angle kitne degree hai?",        "spatial"),
    ("table.png",   "row 4 col 2 ki value kya hai?",  "dense alignment"),
    ("tiny.png",    "bottom-left chhota text kya hai?", "small text"),
]
for img, q, category in tests:
    print(category, "->", model.ask(img, q))   # yahin se reliability ka pata chalta hai

• Precise counting: "is shelf pe kitne bottles hain" — model aksar galat, aur guess confidently deta hai.
• Spatial & geometric reasoning: distance, angle, "kaun kaunse objects touch ho rahe hain" — patches se exact geometry nikalna mushkil hai.
• Dense small text: chhute hue cells, fine print, low-res scans — downscaling mein sabse pehle yahi jaata hai.
• Table alignment: row aur column ko sahi jodna, especially merged cells aur multi-page tables — alignment aksar bigadti hai.

Ye failures hote kyun hain

Root cause ye hai: vision model naap nahi karta, wo plausible description generate karta hai. Wo image ko patches mein dekhta hai aur wahi "description" banaata hai jo training mein aksar mili hai. Counting karna, perspective se angle nikalna, do rows ko align karna — ye measuring tasks hain jo generation ke DNA mein nahi hain.

• Patches ka grid spatial precision ki had rakhta hai — do objects ke beech ka exact gap patch boundaries ke through hi dikhta hai.
• Training data mein "invoice ka total" jaisi fields hamesha wahi hain jo dikhti hain — blank cell mein kabhi value nahi aati, phir bhi model value de sakta hai.
• Model ke paas "mujhe nahi dikhta" ka weak incentive hota hai — jawab banana default hai, abstain karna exception.

Safeguards jab reliability kam ho

# Reliability chain — vision task ke around safeguards lagao
def read_invoice(img):
    # 1. Image hi behtar bhejo
    crop = crop_to_region(img, "table")          # zoom, kam noise
    out = model.ask(crop, "total aur currency nikaalo", schema=INVOICE_SCHEMA)
    # 2. Structured output + validation
    if not validate(out):                          # format, range, sanity
        out = retry_with_prompt(out.errors)
    # 3. Uncertain ho toh insaan ke paas bhejo, DB mein mat daalo
    if out.confidence < 0.85:
        review_queue.append((img, out))
    return out

• Resolution & crop: pehla safeguard hi behtar input hai — failing task ko zoom karke do.
• Structured output: free text ki jagah schema maango — number, date, currency alag fields, phir type-check karo.
• Verification passes: doosra prompt ya doosra model se cross-check; counting ke liye vision ki jagah dedicated tool (detection/counting) use karo.
• Human review routing: confidence low ya validation fail — queue mein daalo, silent flow kabhi nahi.

Exercise

Common mistakes

• Model ke confident tone ko truth maanna: Vision model bhi hallucinate karta hai, aur uska galat jawab bilkul sahi jaisa dikhta hai. Fix: critical fields pe validation aur human review rakho, tone pe bharosa mat karo.
• Counting aur geometry vision pe chhod dena: Ginti, angle, distance — ye measuring tasks hain, generation nahi. Model aksar galat aur confident. Fix: cropping/zoom, dedicated counting tools, ya insaani review.
• Har task ka eval same rakhte jaana: Scene description accha chal raha hai iska matlab invoice extraction bhi chalega — zaroori nahi. Fix: har task ke liye alag, apne data ka test set banao aur wahi score karo.

gist wala kaam  -> model khud kar lega (scene, text reading)
naap wala kaam  -> crop + schema + validation + review
bina safeguard  -> confident galat jawab, chup-chaap DB mein