Lesson 141 — Advanced

Choosing a Multimodal Model

Sahi model chunna ab sirf "sabse accha benchmark wala" sawaal nahi hai — modalities, pricing units, context cost, latency aur privacy sab ek saath decide karte hain. Is lesson mein hum ek shortlist-plus-own-data-test ka tareeka banayenge jo har candidate pe same chalta hai.

Modality se shuru karo

Pehla sawaal hamesha ye hota hai: input mein kya jaana chahiye aur output mein kya aana chahiye. Ye list hi aadhe candidates hata deti hai — kuch models image lete hain par video nahi, kuch audio input lete hain par output text hi hai, aur kuch jobs ke liye dedicated managed services hain jo general model se behtar chalti hain.

Pricing aur context cost

# Multimodal cost ka hisaab — text ke alawa bhi units hote hain
def turn_cost(images=0, audio_min=0, out_tokens=0, rate=None):
    c = 0.0
    c += images * rate["per_image"]        # resolution ke hisaab se bhi badal sakta hai
    c += audio_min * rate["per_audio_min"]  # silence bhi count hota hai
    c += out_tokens * rate["per_out_token"]
    return c

# Context window bhi image leti hai:
# 1 full-res screenshot = badi slice of window
# 5 screenshots = instruction + history + tool outputs squeeze

• Per-token pricing ke saath per-image aur per-audio-minute charges hote hain — bill ab text length se nahi, media volume se badhta hai.
• Image ka token cost resolution pe depend karta hai, file size (KB) pe nahi — chhoti file bhi badi resolution ho sakti hai.
• Ek image window ki badi jagah leti hai: kuch screenshots daalte hi system prompt, conversation history aur tool outputs squeeze hone lagte hain.
• Audio per-second/per-minute count hoti hai — meeting recordings ka batch bill control karna padta hai, "chhota audio hai" wala assumption kaam nahi karta.
• Apne typical request pe ek calculator banao (avg images per call, audio minutes) — list price se nahi, apne payload mix se compare karo.

Latency by modality aur privacy

• Latency modality-wise alag hai: image input ka prefill, TTS ka time-to-first-audio, document ka per-page turnaround — har ek ka apna budget.
• Streaming voice mein delay sunai deta hai (heard, not just measured) — head-to-head metric time-to-first-audio hai, poora response nahi.
• Privacy yahan text se bhi harder constraint hai: awaaz, chehre aur scanned documents ke consent aur data-residency obligations hote hain.
• Data residency: kuch workloads region se bahar hi nahi ja sakte — ye quality score se pehle ka filter hai, baad ka checkbox nahi.

Har candidate ke latency ko stages mein todo — upload, prefill, first token/first audio, full completion. Sirf average dekh ke user experience ka andaza nahi lagta; p95 pe dekho, kyunki worst call hi yaad rehti hai.

API vs self-host vs managed services

• API: zero infra, best models, per-call cost, data provider ke paas jaata hai — shuru karne ka sabse tez raasta.
• Self-host (esp. open speech/vision models): data aapke paas, per-request cost zero, par GPU, ops aur upgrades aapki zimmedari. High steady volume pe per-minute API cost operational effort se badh sakti hai.
• Managed cloud services (document, face, speech ke narrow jobs): ek hi kaam ke liye aksar sasta aur accurate — general multimodal model se pehle ye option check karo.
• Split deployment normal answer hai: STT local, vision API, TTS managed — per modality decide karo, poora stack ek provider pe mat pakdo.

Shortlist, phir apne data pe test

# Shortlist -> own-data test (2-3 candidates, exactly same harness)
cases = load_cases("eval/images") + load_cases("eval/audio")

def score(model, cases):
    hits, cost, latency = 0, 0.0, []
    for c in cases:
        r = model.ask(c.input, c.question)
        hits += int(judge(r, c.expected))
        cost += c.price(model)
        latency.append(r.ms)
    return {"acc": hits / len(cases), "cost": cost, "p95": p95(latency)}

for m in candidates:
    print(m.name, score(m, cases))
# public benchmarks aur demo clips: sirf shortlist tak

• Har candidate ko exactly same cases pe chalao — same images, same audio, same questions; alag inputs pe koi comparison nahi hai.
• Judge ko pehle se likho (exact match, rubric, ya LLM judge) — beech mein judge badalne se purane numbers compare hi nahi hote.
• Table mein rakho: quality, per-call cost, p95 latency, privacy flags. Ek column jeetne se decision nahi — constraints decide karti hain.
• Pilot ke baad production traffic ka chhota sample dobara chalao — eval set aur asli data hamesha alag hote hain.

Exercise

Common mistakes

• Benchmark leaderboard se seedha final choice: Public scores general data pe hain, tumhari images/audio alag hain. Fix: leaderboard se sirf shortlist, final decision apne eval set se.
• Cost ka hisaab sirf output tokens se karna: Per-image/per-minute units bill ko chhupa dete hain aur image tokens window bhi khaate hain. Fix: apne payload mix pe per-call calculator banao.
• Privacy ko last step pe sochna: Faces, voices aur scans consent/residency ke saath aate hain — end mein pata chale toh poora rework. Fix: privacy filter sabse pehle, uske baad quality round.
• Poora stack ek provider pe pakad lena: STT, vision aur TTS ki strengths alag hain; ek default har modality ki need nahi samajhta. Fix: per modality shortlist, hybrid deployment normal hai.

symptoms (modalities) -> filter -> options -> apne test cases -> final
inputs: image / audio / video?   outputs: text / audio?
constraints: privacy > quality > cost (apne case ke hisaab se)
benchmark scores = sirf shortlist, final prescription apne test se