Lesson 34 — Beginner

Async, Concurrency & Batching

Ek-ek karke 100 calls lagana = minutes. Parallel chala diya = seconds — lekin rate limit aur quota pe chhuri. Is lesson mein hum sync vs async, semaphore se bounded concurrency, rate limits aur throughput–cost trade-off dekhenge, provider SDK aur orchestration layer dono ke angle se.

Concurrency kab madad karta hai

Do words ka farq: parallelism = ek waqt mein kaafi kaam chal rahe hain; throughput = unit time mein kitne kaam poore hue. Streaming (M2.2) latency ka ehsaas badlati hai — concurrency throughput badlati hai.

Asyncio se bounded concurrent calls

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()
sem = asyncio.Semaphore(8)          # ek saath max 8 — bounded

async def one_call(text):
    async with sem:
        resp = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": text}],
        )
        return resp.choices[0].message.content

async def main(texts):
    tasks = [one_call(t) for t in texts]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    ok = [r for r in results if not isinstance(r, Exception)]
    failed = [r for r in results if isinstance(r, Exception)]
    print(f"done: {len(ok)} ok, {len(failed)} failed")

asyncio.run(main(texts))

• Sequential for-loop har call par poora wait karta hai — total time = sab ka sum. Gather mein total = sabse lambi call (plus overhead).
• Semaphore bounded concurrency deta hai — 500 tasks bana sakte ho, par ek waqt mein 8 hi chalenge.
• return_exceptions=True se ek failure baaki tasks ko na mare — failed list alag se handle karo (retry M2.9 mein).
• Sync SDK mein ThreadPoolExecutor ya asyncio.to_thread bhi chalte hain — concept wahi hai: wait overlap karna.

Rate limits aur bounded throughput

Concurrency badhao          : throughput up, per-request latency usually same
Limit paar (429 / TPM)      : failures up, retries up, effective throughput DOWN
Limit ke andar rehna         : steady throughput, kam errors, aasaan debugging

Rules of thumb:
- Concurrency limit = tumhara semaphore + provider ke rate limit ka beech
- 429 aaye toh concurrency THODO karo, badhao nahi
- Chhote calls vs bade prompts — token limit alag-alag pe lagti hai
- Provider docs mein apne tier ke limits dekho — numbers badalte rehte hain

• Rate limits do tarah ke hote hain — requests-per-time aur tokens-per-time. Bade prompts wale account mein token limit aksar pehle lagti hai.
• Bounded concurrency + backoff saath chalte hain — semaphore rokta hai, retry recovery deta hai (M2.9).
• Apne workload ka shape pata karo: chhote calls bahut saari ya bade calls kam — dono ka limit alag jagah lagta hai.
• 429 log karo — kitni baar aayi, kis concurrency par. Yeh tuning ka data hai, guess nahi.

Throughput, latency aur cost ka trade-off

• Cost per token wahi rehti hai — concurrency se sasta nahi hota. Haan, zyada speed = quota jaldi burn, aur daily budget jaldi khatam (jo alag ek planning problem hai).
• Latency: user-facing request ke andar concurrency sirf tab fayda deti hai jab uske apne steps independent hon (fan-out). Dependent chain mein koi faayda nahi.
• Resources: har concurrent request ka apna connection, buffers aur logs — memory bhi badhti hai.
• Batch execution: provider ke batch/queue endpoints (offline processing, standard rate) vs realtime API. Decision: zaroorat na ho turant toh batch.
• Orchestration layers batch/buffer abstractions deti hain — lekin semaphores aur limits wahi rehte hain, sirf chhup jaate hain.

Exercise

Common mistakes

• Unbounded gather — sab ek saath: — 500 tasks bina semaphore ke = rate limit ki barish. Semaphore lagao, limit ke andar raho.
• Concurrency ko speed-up samajhna: — ek user request ki latency nahi ghat-ti (model time wahi hai); fayda batch/throughput ka hai, single-turn ka nahi.
• 429 ke baad aur tez retry karna: — load badhata hi badhta hai. Backoff do, concurrency kam karo, phir aage badho.
• Dependency wale steps parallel chalana: — doosre step ka input abhi bana hi nahi. Pehle dependency graph, phir parallelism — ulta kiya toh kachra results.

1 chaiwale  : 10 order x 3 min = 30 min      (sequential)
4 chaiwale  : 10 order, ek waqt 4 = ~9 min   (gather)
40 chaiwale : stove hi nahi = 429 jaisa       (overload, rate limit)

Semaphore(4) = "ek waqt 4 hi order lo" — line sambhali rehti hai.