Lesson 103 — Advanced

Model Routing & Gateways

Production app mein sabse badi galti hoti hai — provider SDK seedha code mein thok dena. Gateway layer ek abstraction deti hai: ek jagah auth, limits aur usage tracking, plus har request ke liye model chunne ka rule — task, cost, risk aur context length dekh kar.

Gateway kya hai

Funda ek line mein: business logic kabhi provider SDK ko directly na chhue. Tumhara code "answer do", gateway decide karega kaun, kaise aur kitne mein.

Routing signals — task, cost, risk, context

• Task type: extraction/classification → chhota fast model; synthesis/complex reasoning → strong model.
• Context length: request bada hai toh us limit wala model hi chalega — gateway khud check kare.
• Latency class: interactive path chhota model, batch path badi freedom.
• Cost budget: feature-wise ceilings — free tier default par chhota model.
• Risk/modality: high-risk actions strong model + approval path; image/audio → vision/audio capable model.

def route(req):
    if req.modality == "image":     return MODELS["vision"]
    if req.task == "extract":       return MODELS["small"]   # structured, chhota kaam
    if req.context_len > req.ctx:   return MODELS["long"]    # window limit
    if req.risk == "high":          return MODELS["strong"]  # approval ke saath
    if is_simple(req):              return MODELS["fast"]
    return MODELS["default"]

Model cascades — sasta pehle, strong baad mein

Cascade ka matlab: cheapest option pehle try karo, mushkil request strong model par chadho. Easy queries ka bada chunk chhota model handle karta hai, hard queries hi strong tak pahunchti hain.

def cascade(req):
    for m in [MODELS["cheap"], MODELS["strong"]]:
        out = call(m, req)
        if validator(out) is None and confidence(out) >= THRESH:
            return out            # kaam ban gaya — ruk jao
    return escalate_to_human(req)  # dono fail — degrade (L9.6)

Provider fallbacks aur unka quality risk

• Outage/429: doosra provider try karo, par per-attempt timeout budget rakho — ek provider ka intezaar doosre ka time nahi kha sakta (L9.6 policies).
• Quality risk: fallback model ka style, format aur nuance alag hota hai. Blind fallback = silent quality drop.
• Format drift: ek model JSON deta hai, doosra markdown — output contract boundary par enforce karo, model pe trust nahi.
• Evals per model: har model apne eval set par (L7) — kabhi bhi "same toh hoga" assume mat karo.
• Rollback flag: naya model default banaya toh ek config flip se wapas — version pin (L9.8).

Auth, limits aur tracking — sab ek jagah

• Keys: gateway config/env mein, app code mein kabhi nahi — rotate karna aasaan.
• Rate limits: per provider per key central counter; client-side guessing se burst handled nahi hote.
• Retries: ek hi policy (budget, backoff, jitter) — har service apna retry laga degi toh storm ban jaayegi.
• Usage tracking: har call par model, task type, tokens, latency, outcome — yahi data L10 ke dashboards ka fuel hai.
• Budgets: per-team/per-feature ceilings — soft limit par alert, hard limit par block.

Exercise

Common mistakes

• Hardcoded model calls: Har function mein direct provider call — model badalna release ban jaata hai. Route() + config layer daalo, app code model ka naam kabhi na dekhe.
• Fallback ko quality-free samajhna: Doosre provider ka output style alag hota hai — schema validation aur per-model evals ke bina silent quality drop chala jaata hai. Contract boundary par check karo.
• Har request ko bada model dena: Simple query par strong model = cost aur latency dono bekaar. Cascade + validator se easy traffic chhote par jaati hai — par escalate ka signal measurable rakho.

simple query -> fast lane  (chhota model)
long context -> wide lane  (bada window wala)
image input  -> separate road (vision model)
provider down-> diversions ON (fallback)