Lesson 104 — Advanced

Latency & Inference Optimization

System dheema kyun hai — ye guess nahi, measurement ka sawaal hai. Is lesson mein latency ko components mein todna, streaming se perceived latency ghatana, independent work parallelise karna, chhote model ka decision aur scale par throughput/utilization — sab decision rules ke saath.

Latency ko components mein toda

total latency = queue wait + network round trip
              + retrieval (embed + search + rerank)
              + TTFT (prompt processing + first token)
              + generation (output tokens / speed)
              + tool calls (serial steps add up hote hain)

har line par apna log timestamp daalo — andaza nahi, measurement

Streaming — perceived latency

Streaming se total time nahi badhta, par user ko pehla content turant milta hai — perceived latency ghat jaati hai. Chat, copilot, summarizer — jagah jahan padhna shuru ho sakta hai, streaming karo.

async def answer(request):
    async for chunk in client.stream(model=request.model, prompt=request.prompt):
        yield chunk                # user abhi se padh sakta hai
# total seconds same reh sakti hain — par time-to-first-content chhota
# retrieval ke turant baad streaming start karo, assembly baad mein mat roko

Independent work parallelise karo

import asyncio

# Independent steps ek saath; dependent steps sequential
results = await asyncio.gather(
    search_docs(q), fetch_profile(user), fetch_tickets(user))
# Serial chalate toh teen latency add hoti — gather se sabse lamba bacha

• Dependency graph banao: kya kis pe depend karta hai — wahi edges parallelism ki seema hain.
• Ek hi sawaal ke kai angles (3 searches, 2 formats) parallel; doosra call pehle ke output par = sequential.
• Do LLM calls ko chain karne se pehle poochho: kya ek call mein ho jaayega? Unnecessary chaining = double latency + double cost.
• Parallel ka limit bhi rakho — provider rate limit aur worker pool saturation naye bottleneck ban jaate hain.

Chhota model, chhota context, batching

• Chhota model: structured extraction, classification, simple summarization — eval green ho toh (L7 regression). Chhota model bhi tez aur sasta padta hai.
• Context reduction: prompt trim, output cap, history compaction (L6.3) — prefill chhota, TTFT chhota.
• Prompt caching: static prefix provider cache mein (L9.5) — prefill ka repeated hissa bachta hai.
• Batching: throughput ke liye service-side batches; par interactive requests ko waiting mat karao.
• Inference service opts: tumhara serving stack kya support karta hai (continuous batching, speculative decode) — docs padho, apne load test par verify karo.

Throughput aur utilization — scale ka alag khel

Do alag cheezein hain: per-request latency aur throughput (kitne requests). Utilization badhao toh queue lagti hai — lambi queue se latency upar jaati hai chahe model utna hi tez ho. Queue depth ko metric banao.

• Load test karo burst shape ke saath — steady traffic se queue behavior nahi dikhta.
• p95/p99 dekho, average mat — average aadhi problem chhupa deta hai.
• Shed load deliberately: queue cap + explicit "busy" response — andar queue mein slow badhne se bahar saaf jawab better hai.
• Autoscale ka trigger queue/latency par rakho, CPU nahi — LLM serving ka CPU kabhi bata hi nahi sakta.

Exercise

Common mistakes

• Andaze se optimize karna: Kisi ne bola "model slow hai" — aur infra upgrade ho gaya. Pehle per-stage timestamps log karo, wahi stage pakdo jo sabse zyada kha rahi hai.
• Sab kuch parallel kar dena: Jo calls ek doosre pe depend karti hain unhe parallel karna race deta hai — galat, ghatiya output. Dependency graph pehle, parallel baad mein.
• Bada model default rakhna: Har simple query par strong model = cost aur latency dono ka explosion. Eval gate ke saath chhota model try karo, quality trend nazar rakho.

total = walk + wait + ride + transfer + walk
walk zyada  -> ghar ke paas rehna (nai baat)
wait zyada  -> frequency/queue fix
ride zyada  -> route change (parallel kaam)