Lesson 128 — Advanced

Model Routers & LLM Gateways

Do providers, teen teams, ek bill — aur har feature ke liye alag SDK. Gateway ek aisa layer hai jo sab providers ke peeche ek endpoint banata hai: credentials wahan, ek request format, aur ek usage view. Is lesson mein routing rules, failover, spend controls, gateway caching aur wo failure mode dekhenge jo gateway lagate hi introduce ho jaata hai.

Gateway kya hai — ek endpoint, anek providers

Mechanism: client gateway ko bulata hai, gateway upstream credentials apne paas rakhta hai, request ko ek standard format mein badalta hai, provider ko bhejta hai, aur usage log karta hai. Tumhara application code jaanta bhi nahi ki jawab kis model se aaya — jab tak tum rule ke through na dekho.

Hosted routers vs self-hosted proxies

gateway ke shapes:
  hosted router      -> third-party endpoint, aapka traffic unke through
                       (OpenRouter, Portkey jaise options)
  self-hosted proxy  -> tumhara infra, tumhara control
                       (LiteLLM jaisa open-source proxy)
  platform routing   -> cloud platform ka apna unified endpoint
  framework gateway  -> app framework ke andar built-in routing

har shape ka trade-off: setup effort vs control vs traffic boundary

• Hosted router: sabse tez start, bahut se providers ek account se — par traffic third-party ke through jaata hai, isliye boundary wali teams ke liye pehle poochhna zaroori.
• Self-hosted proxy: tumhare network mein chalta hai, credentials bhi wahin — control zyada, par availability aur updates tumhari zimmedaari.
• Platform routing: agar sab traffic wahi cloud platform pe hai toh uska unified endpoint bhi gateway jaisa kaam karta hai — extra layer ki zarurat nahi.
• Framework gateway: app ke andar hi routing — chhote app ke liye kaafi, par cross-team usage view nahi milta.

Routing rules aur automatic failover

# Routing rule ka shape (concept — har gateway ka syntax alag)
routes:
  - name: long-context
    match: { context_tokens: "> 64000" }
    target: long-context-model
  - name: cheap-bulk
    match: { task: "classification" }
    target: small-cheap-model
  - name: default
    target: primary-model
    fallbacks: [secondary-model, tertiary-model]
    retry: { on: [429, 5xx, timeout], backoff: "exponential+jitter" }

• Rule ke parameters: task type, cost ceiling, context length, latency target, region, risk level — sab routing keys ban sakte hain.
• Failover mechanism: upstream error ya throttle aaye toh gateway next target try karta hai — tumhare client ko 429 dikhta bhi nahi. Timeout budget gateway ke paas hi hona chahiye.
• Order matters: fallback list ka order hi priority hai. Primary ka p95 dekh kar rakho — fallback bhi latency budget mein aana chahiye, warna failover ne latency badha di.
• Retry ownership: gateway aur client dono retry karein toh double retry storm. Decide karo kaun karta hai — aksar gateway hi karta hai.

Virtual keys, budgets aur quotas

# Gateway spend controls ka shape (concept)
virtual_keys = {
    "team-search": { "monthly_budget": 5000, "rpm_limit": 60 },
    "feature-summarizer": { "monthly_budget": 800, "rpm_limit": 12 },
}
# app real provider key kabhi nahi dekhta — sirf virtual key
# budget hit -> gateway throttle ya block (policy par)
# per-key rate limit -> ek client doosre ki capacity nahi kha sakta

• Virtual keys: har team/feature ko gateway ki apni key milti hai — real provider credentials andar rehte hain, rotation bhi gateway se.
• Budgets + quotas: monthly budget aur per-key RPM/TPM limits — ek buggy client poori team ki capacity nahi kha jaata.
• Attribution: har request virtual key se judti hai, isliye "kaun kitna kharch kar raha hai" pehle se dikhta hai — shared provider key se mumkin nahi.
• Kill switch: ek key revoke karo, feature ruk jaata hai — blast radius control ka sabse tez tareeka.

Gateway ab critical path par hai — aur kuch farq chhupta nahi

Gateway lagate hi tumne ek naya single point of failure apne request path mein daal diya. Ab uski apni availability, timeouts aur bypass plan chahiye — warna provider theek hai par gateway gira toh sab ruk gaya.

• Gateway ki availability: self-hosted ho toh tumhara monitoring/scaling; hosted ho toh unki status page aur apna fallback (direct provider path bhi ready rakho).
• Bypass plan: gateway dead ho jaye toh config se provider endpoint seedha laga kar chalao — ye drill pehle se likhi honi chahiye.
• Jo farq gateway chhupa nahi sakta: tool-calling formats, structured-output support, context limits aur prompt behaviour — har model ka alag. Rule change = naya model = naya testing.
• Prompt portability ka dhoka: ek prompt do providers pe chal gaya — par tool schemas ya JSON mode alag ho sakte hain; integration tests hi pakadte hain.

Gateway par caching — exact aur semantic

# Gateway cache ke do levels
# 1. exact-match: same request hash -> cached response (sasta, safe)
key = hash(normalize(messages) + model + params)
if key in cache:
    return cache[key]      # zero token cost

# 2. semantic: similar meaning -> milta jawab (sasta, par risky)
#    similarity threshold + scope se control

# kabhi cache se mat lo:
# - personalized / user-specific data
# - real-time info (price, stock, time)
# - tool outputs jo har turn pe badalte hain

• Exact-match cache: identical request pe identical response — sabse safe, sabse sasta. Config-heavy prompts mein hit rate aksar accha hota hai.
• Semantic cache: alag wording ka same sawaal — similarity threshold se milta jawab. Par threshold galat ho toh purana ya galat jawab mil sakta hai.
• Kabhi cache nahi: personalized, real-time aur authenticated content — yahan cache hit hi incident ban jaata hai.
• TTL + scope: har cache entry ka time-to-live aur key-scope (model, user, tenant) hona chahiye — warna tenant boundary cache se cross ho sakti hai.

Router kab apni jagah banata hai — aur evals kyun chahiye

• Router earns its place: do se zyada providers, do se zyada teams, ya spend jo govern karna ho — teeno mein se ek condition ho tabhi layer add karo.
• Honest answer kabhi kabhi: ek provider, ek chhoti team — seedha SDK + chhota wrapper. Gateway ka setup/ops cost us complexity se zyada ho toh mat lo.
• Routing rule = model swap: rule badal diya, jawab ab doosre model se — app code change hua hi nahi. Isliye rule change bhi code change ki tarah review + eval se guzarta hai.
• Evals in the loop: har route target ko apne eval set pe chalao, fallback model bhi — emergency mein sabse pehle wahi toh jawab dega.

gateway rollout checklist:
  1. do providers configure + auth verify
  2. primary pe traffic chalao, usage view check
  3. primary ko fail karke fallback drill (latency + cost dono)
  4. virtual key budgets set (team/feature wise)
  5. exact-match cache on, semantic cache decide
  6. rule change ka eval gate — bina eval ke routing edit nahi
  7. bypass plan document — gateway down scenario

Exercise

Common mistakes

• Gateway laga toh providers interchangeable ho gaye: Tool-calling formats, structured output aur prompt behaviour phir bhi alag hain. Fix: rule change ko model swap maano — har change pe integration tests + evals.
• Bina drill ke failover trust karna: Diagram mein fallback hai, real outage mein pata chalta hai timeout galat hai. Fix: primary ko fault-inject karke fallback ki latency aur cost verify karo.
• Retry gateway aur client dono taraf: Double retry se storm — throttle aur badh jaata hai. Fix: retry ownership ek jagah decide karo, aksar gateway par, exponential backoff + jitter ke saath.
• Semantic cache sab request pe laga dena: Personalized ya real-time data cache hit ho gaya toh purana jawab user tak — seedha incident. Fix: exact-match se shuru, semantic sirf bounded domain mein, TTL + tenant scope ke saath.

visitor (request)  -> gate (gateway) -> guard ka record (usage log)
                    -> guest pass (virtual key) + budget
                    -> gate 1 band? -> gate 2 (failover)
andar: asli flat keys (provider credentials) bahar nahi jaatin