Lesson 140 — Advanced

Cross-modal Search & Retrieval

Ek text query se image dhoondhna, ya ek image se uska text dhoondhna — dono tab mumkin hain jab image aur text ek hi vector space mein baithte ho. Is lesson mein hum shared embedding space ka logic, joint encoders, caption-then-search baseline, kahan cross-modal fail hota hai, aur apne collection pe eval dekhenge.

Shared embedding space kya hai

Agar ek image aur uska caption wahi cheez describe karte hain, toh unke vectors paas hone chahiye — training isi goal se hoti hai. Iske baad jo machinery hai wahi hai jo L4 mein seekhi: vector store, cosine, top-k. Sirf encoder badal jaata hai — query ka vector text encoder se, corpus ka image encoder se.

CLIP-style joint embeddings

# Do encoders, ek shared space
def encode_pair(image, text):
    img_vec = image_encoder.encode(image)   # -> vector
    txt_vec = text_encoder.encode(text)     # -> vector (same dim, same axes)
    return l2_normalize(img_vec), l2_normalize(txt_vec)

img, txt = encode_pair("shelf.jpg", "blue water bottle on a shelf")
score = float(img @ txt)     # cosine similarity — bilkul text wali maths

# Index banate waqt: har image ka vector daalo
# Query karte waqt: text query ka vector, top-k image ids wapas

• Training ke dauran matching image-text pairs ko paas laaya jaata hai aur mismatched pairs ko door — isi se dono encoders ka coordinate system align hota hai.
• Dono encoders model ke saath fixed hain — unke outputs ek hi dimension aur ek hi axes mein hote hain, tabhi cross compare mumkin hai.
• L2-normalize + dot product = cosine, wahi L4 wali vector search yahan re-use hoti hai — naya kuch nahi, bas input modality badli.
• Direction dono taraf chalta hai: text se image query, image se text query, dono ek hi index pe. Machinery same, sirf encoder badalta hai.

Cross-modal vs caption-then-search

# Option A: caption-then-search (do stage)
captions = {img.id: caption_model.describe(img) for img in images}  # pehle text banao
hits = text_search(query, embed_all(captions.values()))             # phir text search

# Option B: cross-modal (ek stage)
index.build(image_encoder.encode(img) for img in images)            # seedha image vector
hits = index.search(text_encoder.encode(query))

# Trade:
# caption    -> explainable, purana text stack, par caption mein jo nahi likha wo kho gaya
# cross-modal -> visual detail bacha rehta hai, par fine text-in-image ab bhi mushkil

• Caption-then-search tab jeetta hai jab visual information text mein aasaani se aa jaati hai — aur tumhara purana text/RAG stack hi chal jaata hai.
• Cross-modal tab jeetta hai jab query ke words corpus ke words se match nahi hote — natural language likhi hui "feel" wali cheez ke liye.
• Cost dono ki ingestion pe aati hai: caption pipeline har image pe ek extra model call maangta hai; cross-modal bhi har image ka vector banata hai — par storage aur search sasta.
• Hybrid bhi ek option hai: caption text aur visual vector dono index mein daalo, dono results merge karo — aksar sabse robust.

Kahan fail hota hai — aur apne data pe eval

• Fine-grained detail: red vs maroon, exact model variant, chhote logos — general encoders ye pakadte nahi.
• Text inside images: signs, receipts, UI screenshots — image encoder ke liye ye sirf pixels hain; OCR ya document pipeline yahan behtar chalti hai.
• Counting aur spatial: "teen bottles wali shelf" — wahi vision failure modes jo L12.3 mein padhe, yahan bhi laagu hote hain.
• Domain imagery: medical scans, factory parts — encoder ne training mein nahi dekha toh uska space hi tumhare domain ka nahi.
• Benchmark trap: public scores general collections pe hote hain — tumhara corpus alag hai, apni images aur queries pe pehle chhota eval banao.

# Apna eval: 20-30 queries + gold images, wahi harness har candidate pe
cases = [
    {"q": "chai wali photo",           "gold": ["img-12", "img-44"]},
    {"q": "red sneakers on court",     "gold": ["img-07"]},
    {"q": "receipt with total 1240",   "gold": ["img-19"]},
]

def score(index, cases):
    hits = sum(len(set(top_k(index, c["q"])) & set(c["gold"])) for c in cases)
    return hits / sum(len(c["gold"]) for c in cases)   # simple recall

# Baseline (caption-then-search) ko same cases pe chalao — winner per-query dekho

Exercise

Common mistakes

• Vendor benchmark pe seedha model chun lena: Public scores general image collections pe hain; tumhari images aur queries alag hain. Fix: apne 20-30 cases + gold images se har candidate ko compare karo.
• Caption-then-search ko hamesha inferior maanna: Dono alag jagah jeette hain — caption path text-heavy queries pe strong hai. Fix: apne query mix pe dono chalao, per-query winner nikalo, ya hybrid jodo.
• Alag-alag models ke vectors mix kar dena: Image aur text encoders ek hi family ke hain — mismatched versions ke vectors compare hi nahi hote. Fix: ek model family, model change pe poora index re-embed.
• Text-in-image queries ko cross-modal pe chhod dena: Screenshots aur receipts par general encoders weak hain, ranking girti hai. Fix: OCR/text-extraction path side mein chalao aur hybrid results jodo.

cross-modal:         "chai wali photo" -> seedha image vectors -> top-k
caption-then-search: har photo -> caption -> text index -> top-k
jo caption mein nahi likha, wo second path mein kho jaata hai
best: dono index rakho, query decide kare kaun jeetta hai