Lesson 126 — Advanced

Capacity, Quotas & Cost

Model call karna aasaan hai — mushkil hai wo call tab bhi chalna jab traffic badhe, aur bill tab bhi control mein rahe jab agent loop bhage. Is lesson mein hum platform ki quota mechanics, capacity ke options aur end-to-end cost model samjhenge.

On-demand vs provisioned/committed capacity

Mechanism: on-demand mein tum shared pool se maangte ho — platform jitna de sake utna deta hai, throttle kabhi kabhi. Committed capacity mein ek block physically tumhare naam ho jaata hai — uski rate limits alag hain, aur availability contract ke under aati hai.

Quota model padhna

platform quotas (common shape):
  RPM  -> requests per minute (kitni calls)
  TPM  -> tokens per minute (kitne input+output tokens)
  concurrency -> ek saath kitni requests chal sakti hain
  per-model limits -> har model ka apna quota, region-wise alag

note: limits account-level hain, region-level bhi —
      galat region mein quota check karoge toh numbers alag milenge

• RPM vs TPM: chhote prompts zyada RPM le jaate hain, bade prompts TPM. Dono alag se limit hain — sirf ek dekh kar size mat karo.
• Per-model, per-region: quota model specific hai — tumhara quota doosre region ya doosre model par alag ho sakta hai.
• 429 samjho: throttle (too many requests) ka matlab capacity nahi, rate. Client-side retry with backoff likho — har 429 pe crash nahi.
• Soft limits: shuruaati quota aksar chhota hota hai (new accounts) — production jaate waqt ye pehle se plan karo.

Burst traffic ki planning

# Burst absorb karne ka pattern: queue + backpressure
import asyncio

sem = asyncio.Semaphore(8)   # ek saath max 8 in-flight calls

async def guarded_call(prompt: str) -> str:
    async with sem:            # semaphore full toh request wait karti hai
        return await model_call(prompt)

# 429 aaye toh: exponential backoff + jitter, seedha retry nahi
# batch jobs ke liye: platform ke batch endpoints use karo (alag, sasta queue)

• Platform spike absorb nahi karega: quota tumhari hai — spike aaya toh 429 aayega. Queue laga kar requests ko line mein baithao, user ko error mat dikhao.
• Backpressure: app ka apna limit — queue lambi ho toh "abhi busy hai" dikhao, andhere mein 10x load mat bhejo.
• Batch endpoints: non-urgent workload (nightly summaries) ke liye platform batch options hain — queue lagti hai, rate kam hota hai, par bulk ke liye suitable.
• Jitter: retry mein random delay zaroor — sab clients ek saath retry karein toh thundering herd, spike aur badh jaata hai.

Quota increase — approval process, setting nahi

Quota badhane ka request platform ka approval process hai — setting nahi. Isliye evidence se bhara request chahiye: expected QPS, tokens per request ka average, growth curve, aur business case.

• Traffic evidence do — "hamare paas 500 users hain" nahi; "sustained 40 RPM, peak 120, growth 20% monthly" — numbers jo verify hon.
• Lead time maano — approval dinon ya hafte le sakta hai. Production launch se pehle request karo, uske baad nahi.
• Fallback rakho — quota milne tak batch/queue se kaam chalao, launch date capacity par dependent mat karo.
• Regional quota alag hota hai — jis region mein deploy kar rahe ho wahi mention karo.

Cost per request — end to end

# Cost per request ka model (rate platform se lo, guess mat karo)
def cost_per_request(
    in_tok: int, out_tok: int,
    rate_in: float, rate_out: float,   # per token rates (platform catalog se)
    retrieval_calls: int = 0, retrieval_rate: float = 0.0,
) -> float:
    llm = in_tok * rate_in + out_tok * rate_out
    retr = retrieval_calls * retrieval_rate
    return llm + retr

# ek RAG request ka real cost = LLM tokens + retrieval calls
# + agent loop ka multiplier (har step pe tokens dobara)

• Input vs output dono count hote hain — output tokens aksar input se rate mein alag (aur mahange) hote hain; verbose prompts ka asli kharcha yahin chhupa hota hai.
• Retrieval calls bhi paisa hain: knowledge base, reranker, embedding calls — sab per-request cost hain, LLM bill sirf hissa hai.
• Agent loops multiplier hain: ek user request 5 LLM calls generate kar sakti hai — cost per call × calls per request, ye product karna bhool jaana sabse common underestimate hai.
• Failure ka cost bhi gin lo: retries, timeouts, cached na hona. Production ka real cost test traffic se nikalta hai, doc example se nahi.

Attribution, budgets aur runaway alerts

# Cloud billing tags -> cost attribution
resource_tags:
  team: search
  env: prod
  cost-center: cc-1234
  feature: doc-qa

# Budget alert example (concept, platform setting)
budget:
  amount: "monthly cap (tumhara number)"
  alerts: [50%, 80%, 100%]
  action_at_breach: "throttle | notify | hard-stop"

• Tags ke bina attribution nahi: shared API key se team-wise split mumkin nahi; cloud tags/labels se har resource team, env aur feature ke hisaab se bill mein dikhta hai.
• Budget alerts: 50/80/100% alerts par owner ko notification — budget sirf ek number nahi, ek escalation policy hai.
• Runaway loops: agent bug ne 100x calls kar diye — spend monitoring hi wo alarm hai jo tumhe mahine ke bill se pehle bachata hai. Rate limit + budget dono rakho.
• Hard stop bhi socho: critical env mein budget breach par throttle/hard-stop — "alert bhej diya" enough nahi, blast radius control chahiye.

Exercise

Common mistakes

• Quota ko default pe chhod dena: New account ki chhoti quota production launch ke hafta todti hai. Fix: quota ek planning item hai — current limit check, needed limit calculate, request pehle se daalo.
• Sirf token price dekh kar cost forecast karna: Output tokens, retrieval calls aur agent loop ke multipliers bhool jaana — bill estimate 5-10x chhoti aati hai. Fix: real traffic ka hafta lo, end-to-end model chalao.
• Spike platform absorb kar lega sochna: Throttle aayegi hi — 429 ka retry-with-backoff aur queue bina error aata hai. Fix: semaphore/queue, jitter ke saath retries, batch endpoints for bulk.
• Tags bina billing dekhna: Bill total dikhta hai, kaun kitna kharch kar raha hai nahi — optimization mumkin hi nahi. Fix: har resource pe team/env/feature tags mandatory banao.

mehman kam + predictable  -> hall booking (committed capacity)
mehman uncertain           -> per plate (on-demand)
spike (achanak 2x log)     -> per plate extra + queue (throttle se bacho)
mahine ka hisaab           -> budget alert (50% par message)