Lesson 28 — Beginner

Streaming LLM Responses

Poora jawab ek saath aane mein seconds lagte hain — user sochta hai app hang ho gaya. Streaming isi ka ilaaj hai: model har token generate karte hi chhota chunk bhejta hai aur screen render hoti rehti hai. Is lesson mein hum stream loop, TTFT, accumulation aur interruption handling dekhenge.

Streaming kya hai

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Chai ki recipe 4 line mein batao."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)   # jo aaya wahi turant dikhao
print()

stream=True hote hi response object ki jagah iterator milta hai. Har chunk mein sirf naya hissa (delta) hota hai — purana text kabhi dobara nahi aata. Tumhara kaam un hisso ko jodna aur dikhana hai.

TTFT, generation speed aur perceived latency

t=0.0s  request bheji
t=0.4s  pehla token aaya      <-- TTFT (time to first token)
t=0.5s  token 2 aaya
t=0.6s  token 3 aaya          <-- generation speed (tokens/sec)
...
t=4.2s  last token aaya       <-- total latency

User experience:
non-streaming : 0.0s ... kaali screen ... 4.2s -> EK SAATH poora jawab
streaming     : 0.4s pehla shabd, phir flow   -> LAGTA HAI turant jawab

• TTFT = request se pehle token tak ka time — queueing, network aur prefill isi mein aata hai. Streaming ka fayda tabhi dikhta hai jab TTFT kam ho.
• Generation speed = tokens per second — yehi UI ko smooth ya chunky banati hai.
• Total latency = poore jawab ka time — streaming se yeh ghat-ti nahi, sirf perceived latency (ehsaas) behtar hota hai.
• Interview line: streaming kaam jaldi nahi khatam karwati, bas intezaar ko bardaasht layak banati hai.

Accumulate karo, interruption handle karo

def stream_with_safety(model, messages):
    parts = []
    try:
        for chunk in client.chat.completions.create(
            model=model, messages=messages, stream=True, timeout=30
        ):
            delta = chunk.choices[0].delta.content
            if delta:
                parts.append(delta)
                yield delta
    except KeyboardInterrupt:            # user ne rok diya
        yield "\n[roka gaya]"
    except Exception as e:               # network drop, server error
        yield f"\n[stream toota: {type(e).__name__}]"
    finally:
        text = "".join(parts)            # jo bhi aaya tak — accumulate
        print(f"--- mila {len(parts)} chunks, {len(text)} chars ---")

• Chunks ko list mein append karo, end mein "".join(parts) se poora text nikaalo. Sirf print karte raho toh interruption par aadha jawab kahin nahi milega.
• Network failure beech mein aa sakta hai — partial text ko complete mat maano; clean end ya finish_reason ka check karo.
• User ne cancel kiya (Esc, button) toh partial output alag se mark karo — log mein completed: false likho.
• Har chunk ke baad client ko flush karna mat bhoolna — warna sab end mein ek saath aayega aur streaming ka faayda zero.

SSE — raw stream kaisa dikhta hai

data: {"choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"choices":[{"delta":{"content":"Chai"},"index":0}]}

data: {"choices":[{"delta":{"content":" ki"},"index":0}]}

data: {"choices":[{"delta":{},"finish_reason":"stop","index":0}]}

data: [DONE]

• HTTP response content-type text/event-stream hota hai — har event "data: " prefix ke saath aata hai aur khaali line par end hota hai.
• Har JSON line mein sirf naya delta hota — role pehle chunk mein, content baaki mein, finish_reason aakhri mein.
• [DONE] marker stream ka official end hai — uske baad kuch nahi aata.
• Tumhara HTTP client (httpx, fetch, EventSource) in lines ko tod kar JSON parse karta hai — SDK yeh sab internally karta hai, lekin raw dekhna debugging mein bachata hai.

Exercise

Common mistakes

• Sirf print karte jaana, accumulate na karna: — network tootne par output kahin milta hi nahi. parts list rakho, end mein join karo, partial ko mark karo.
• Streaming ko speed-up samajhna: — total latency same rehti hai; fayda sirf perceived latency ka hai. Marketing copy mein "2x fast" likhna galat hoga.
• Har chhote JSON output par streaming lagana: — parse karne mein dikkat aur code complexity badhti hai. 1-2 second ke calls mein plain response kaafi hai.
• flush karna bhool jaana: — print buffer mein baith kar sab end mein ek saath niklega, typing effect gayab. end="", flush=True ya yield karo.

Non-streaming : 0s ..... kaali screen ..... 4s -> "Match jeet gaya"
Streaming     : 0.4s "Bharat" ... "ne" ... "match" ... "jeet gaya"

TTFT = pehla word aane tak ka intzaar
Fayda: intzaar kam lage (perceived latency), total time wahi!