Lesson 144 — Advanced

Capstone: Ship a Multimodal Build

L12 ke saare pieces ab ek jagah: ek feature jo image ya audio leta hai, sahi modality pipeline se guzarta hai, validated jawab deta hai aur production mein observe ho sakta hai. Is lesson mein hum scope se lekar ship checklist tak ka poora rasta ek blueprint ke roop mein dekhenge — taaki tum apna build isi pe banao.

Scope — kya banana hai

Scope pehle rakho: ek hi modality, ek hi user job, ek hi measurable success (jaise extracted fields ka exact-match count apne test set pe). Sab kuch karne wala capstone kuch bhi prove nahi karta — depth hi yahan impress karti hai.

Architecture — input se validated jawab tak

# Skeleton: accept -> modality handle -> model -> validate -> respond
def handle(request):
    payload = accept(request)                 # image / audio, size limits yahin
    if not within_budget(payload):            # resolution, duration, file count
        return reject("payload too large")

    artefact = to_model_input(payload)        # resize/crop ya raw audio
    draft = model.ask(artefact, prompt_with_schema())

    result = validate(draft, schema)          # types, ranges, required fields
    if result.low_confidence:
        return review_queue(payload, draft)   # human review, silent flow nahi

    return respond(result, source=payload.ref)  # page/region/time citation

• Accept stage pe hi limits lagao: max resolution, max duration, max files — token budget wahin banta hai, model call se pehle nahi.
• Native vs pipeline ka choice document karke rakho — kya detail kyun kho rahi hai, yahi tumhara system design discussion hai.
• Har stage pe timestamps log karo aur first-response target alag se define karo — latency budget ke bina "fast" sirf feeling hai.
• Fallback chain likho: model down / validation fail → queue ya simpler path. Silent failure kabhi nahi.

Eval set aur grounding

# Apna eval set — vendor demo se kaam nahi chalega
cases = [
    {"input": "img/invoice-03.jpg", "q": "total?",   "gold": "1,240"},
    {"input": "img/scan-11.png",   "q": "due date?", "gold": "2026-11-05"},
    {"input": "aud/call-07.wav",   "q": "mood?",     "gold": "frustrated"},
]
# Har case pe: match score + latency + cost log karo
# Grounding: har field ke saath source ref (page 4 / region / timestamp)
# Regression: model ya provider badalne pe yehi suite dobara chalao

• Eval set pehla deliverable banao — 20-30 apne real examples, hard cases ke saath (dense table, low-res scan, code-mixed audio).
• Grounding mandatory: har extracted field ke saath page/region/time ref — human bina poora document dobara padhe verify kar sake.
• Structured output (schema-typed JSON) zaroori hai — free-form summary kabhi validate hi nahi hoti, aur usme hallucination chup jaati hai.
• Regression runs: prompt, model ya provider badle toh eval suite dobara chalao — quality chup-chaap drift nahi honi chahiye.

Ship checklist aur interview delivery

• README: input/output contract, modality choice (native vs pipeline) aur trade-offs, known failure modes — sab likha hua.
• Cost model: per-call estimate apne payload mix pe, aur budget alerts — surprise bill nahi chahiye.
• Latency: stage-wise timings, p95 target, aur slow-path ka plan (queue, streaming, holding phrase).
• Privacy: consent, retention aur region — voice, faces aur scans ke liye explicitly stated.
• Observability: input sample, draft, validated result aur review decisions — sab traceable, bug repro mumkin.
• Eval in CI: vendor/prompt change bhi tumhe quality girne se pehle rok le.

Exercise

Common mistakes

• Capstone mein sab modalities ek saath ghusa lena: Image + voice + video = scope explode, kuch bhi deep nahi hota. Fix: ek modality, ek user job, ek measurable success — depth hi impress karti hai.
• Vendor demo pe hi demo chala dena: Apne data pe kabhi test nahi kiya toh interview ka ek sawaal hi gir jaayega. Fix: apna eval set capstone ka pehla deliverable banao.
• Uncertain values chup-chaap database mein daal dena: Hallucinated field production mein silent corruption hai. Fix: confidence gate + human review queue + page/region grounding har field pe.
• Logs, limits aur privacy bhool jaana: Sirf happy-path demo college project vibe deta hai. Fix: rejects, fallbacks, timings aur privacy note — checklist wala hissa hi ship karo.

counter (model) -> rule check (schema) -> galat ho toh cancel (review)
               -> roz ka hisaab log (observability)
bina rules ke machine chalti nahi — bina eval ke feature ship nahi