Lesson 127 — Advanced

Open-Weight Model Platforms

Open weights ka matlab yeh nahi ki GPU khud chalane padenge. Is lesson mein hum hosting ladder samjhenge — kisi aur ki API se lekar apne GPUs tak — aur har rung ka cost, latency, control aur effort ka trade-off dekhenge: model card trust se lekar vLLM ke continuous batching aur hardware sizing tak.

Open-weight hosting ladder

Ladder ka logic: jaise-jaise control badhta hai, billing per-token se per-GPU-hour badalti hai aur kaam tumhare sar aata hai. Serverless pe tum sirf call karo; GPU platform pe tumhara container chalta hai unke hardware pe; apne servers pe batching, scaling, updates sab tumhare. Koi rung "best" nahi — tumhare volume, latency budget aur boundary pe depend karta hai.

Model hubs aur checkpoint trust

model card mein padhne wala sab kuch:
  base model / architecture   -> kis family ka model hai
  licence                     -> commercial use allowed? conditions kya hain
  gated access                -> pehle terms/approval, phir download
  intended use + limitations  -> tumhare use case se match karta hai?
  card ke eval numbers        -> apne task pe khud phir bhi test karo
  official repo vs re-upload  -> checkpoint ki provenance check karo

• Hub se dhoondo, card padho: Hugging Face jaise hubs par model card hi licence, intended use aur limitations ka ghar hai — download se pehle wahi padho.
• Licence non-negotiable: har open-weight model commercial use allow nahi karta; kuch ke saath output-sharing ya attribution conditions hoti hain.
• Gated checkpoints: kuch models terms accept karne ke baad milte hain — ye approval step bhi onboarding checklist ka hissa hai, launch week ka surprise nahi.
• Trust: official provider repo se hi lo. Kisi ka re-uploaded checkpoint mein tumne jo nahi dekha — wahi problem audit mein pakdi jaati hai.

Serverless inference vs GPU compute platforms

# Rung 1: serverless inference provider — per token, GPU kabhi dikhta nahi
resp = inference_provider.chat(
    model="open-model-id",
    messages=[{"role": "user", "content": "Ek chhota summary do"}],
)
# billing: input + output tokens
# scaling, batching, hardware: sab provider ka

# Rung 2: GPU compute platform — tum serving stack laate ho
# image tumhara (vLLM/TGI container), schedule/scale/bill platform ka
# billing: GPU-hour, utilisation tumhari zimmedaari

• Serverless providers (Together, Fireworks, Groq, Replicate jaise options): per token pay, GPU kabhi nahi dekhna, cold start/scale provider ka kaam. Custom serving knobs limited rehte hain.
• GPU compute platforms (Modal, Baseten, RunPod jaise options): tum apna serving stack le kar aate ho — wo hardware schedule, scale aur bill karte hain.
• Farq control ka hai: serverless mein "model kaise serve hua" provider ka decision; GPU platform mein batching flags, parallelism, version pinning — sab tumhare haath mein.
• Portability bonus: GPU platform wala image kisi bhi compatible host pe chal jaata hai — rung badalna redeploy hai, rewrite nahi.

Khud serve karna — vLLM, TGI, Ollama

# Local dev ke liye (chhota model, laptop pe)
ollama serve
ollama run <model-name>

# GPU box pe production-style serving
vllm serve <model-name> --port 8000
# TGI ka bhi wahi shape: model path + port + limit flags

# client: tumhara app dono se OpenAI-style endpoint se baat karta hai

• Continuous batching: vLLM jaise servers har nayi request ko chal rahi hui batch mein daal dete hain — poore batch ka intezaar nahi. Isliye "ek request, wait, agla request" wala naive loop GPU ko aksar idle chhodta hai.
• KV-cache reuse: system prompt aur tool definitions har turn pe same hote hain — engine unka KV-cache reuse karta hai, tokens dobara compute nahi hote. Long conversations mein yahi sabse bada saving hai.
• Ollama vs vLLM/TGI: Ollama local dev aur light use ke liye; vLLM/TGI GPU production ke liye. Teeno OpenAI-style endpoint dete hain — client code same rehta hai.
• Naive loop ka nuksaan: serial calls pe GPU utilisation gir jaata hai — batching aur concurrent requests hi kiraye ke GPU ke paise vasool karte hain.

Per-token vs per-GPU-hour — utilization point

# Dono hisaab apne numbers pe chalao — vendor example pe nahi
def per_token_cost(in_tok, out_tok, rate_in, rate_out):
    return in_tok * rate_in + out_tok * rate_out

def per_hour_cost(gpu_hours, rate_per_hour):
    return gpu_hours * rate_per_hour

# crossover: volume badhne par per-hour fixed cost per token girta hai
# sawaal hamesha yahi: mera actual monthly volume kitna hai
# aur mera utilisation (idle hours) kitna?

• Per-token: fixed cost zero — traffic ruka toh bill ruka. Par high monthly volume pe per-token price guna volume kabhi kabhi GPU rent se upar nikal jaata hai.
• Per-GPU-hour: fixed monthly cost — jitna zyada utilisation, utna sasta per token. Par idle hours bhi bharte ho (scale-to-zero na ho toh).
• Crossover point kisi aur ka nahi, tumhara hai: ek hafta real traffic uthao, dono formulas chala kar dekho kahan lines cross karti hain.
• Price ke saath latency bhi: sirf rate pe mat lo — p95 latency, cold start behaviour aur failure rate bhi comparison ka hissa hain.

Cold starts, scale-to-zero aur hardware sizing

sizing ka basic math (weights ke liye):
  params (billions) x bytes-per-param = weight memory
  fp16 / bf16  -> ~2 bytes per param
  int8         -> ~1 byte
  int4         -> ~0.5 byte (quantized, quality validate karo)

upar se:
  KV-cache -> context length aur batch ke saath badhta hai
  overhead   -> runtime, activations, headroom

matlab: memory hi decide karta hai kaunsa GPU chahiye

• Cold start: scale-to-zero deployment pehli request pe model load karta hai — pehla answer slow. Frequent traffic wali service ko warm rakho (minimum instances) ya client-side se pehle ek warm-up call maaro.
• Sizing order: pehle weights (params × precision), phir KV-cache (context length × batch ke saath grow hota hai), phir headroom. Long-context workload aksar weights se pehle memory kha jaate hain.
• Quantization: memory kam, par quality task pe depend karti hai — hamesha apne eval set pe check, leaderboard pe nahi.
• Platform docs hi source of truth: hosting platform apne docs mein supported configs batata hai — forum post ya purana tutorial nahi.

Hosting ka asli kharcha — open vs closed apne task pe

• Hosting kya kharidti hai: volume pe predictable cost, data jo boundary se bahar nahi jaata, aur versions ka full control — teeno non-negotiable requirements ban sakte hain.
• Hosting kya maangti hai: serving ops, batching tuning, upgrades, security patches, aur evals har change pe. Ye cost nahi dikhti, par engineering time zaroor hai.
• Open vs closed ka faisla: dono ko apne task ke eval set pe daalo — quality, latency, cost per request, teen columns, same test.
• Hybrid bhi chalta hai: closed model tricky reasoning ke liye, open model high-volume simple calls ke liye — router se manage karo (agle lesson mein).

Sabse bada trap: "open hai isliye sasta" ya "closed hai isliye behtar" — dono assumptions hain. Asli numbers tabhi aate hain jab tumne dono ko apne corpus, apne prompts aur apne latency budget pe chala kar dekha ho.

Exercise

Common mistakes

• Open weights ko free samajhna: Licence alag cheez hai, hosting ka kharcha alag, ops ka time alag. Fix: per-token vs per-GPU-hour math apne volume pe chalao, "open = free" assumption nahi.
• Bina card padhe checkpoint le aana: Licence commercial use rok rahi hai ya gated model ka approval missing hai — audit mein pakda jaata hai. Fix: model card + revision hash apne records mein rakho.
• Kam traffic pe per-GPU-hour le lena: Idle GPU hours bhar kar bhi per-token se mehnga pad sakta hai. Fix: ek hafta real traffic se crossover nikalo, vendor example se nahi.
• Naive serial loop se serving benchmark karna: Ek-ek call karke throughput test karoge toh GPU idle milega aur galat conclusion milega. Fix: concurrency badha kar test karo — continuous batching tab dikhta hai.

kam traffic + kabhi kabhi   -> Uber (serverless, per token)
steady daily use            -> rental (GPU platform, per hour)
roz 24x7 + control chahiye  -> apni car (own servers)
sawaal: mileage kitna hai, aur maintenance kaun karega?