Lesson 166 — Advanced

Voice or Multimodal Build

Voice aur vision dono mein ek hi asli challenge hai: latency budget. Is lesson mein do paths hain — realtime voice agent (sunna → samajhna → bolna) aur vision pipeline (image → description/decision) — dono ka architecture aur trade-offs.

Project kya hai

Project ka demo point hai feel: voice agent kitna responsive lagta hai, vision pipeline kitna reliable. Do numbers hain jinse judge honge — turn latency (kitne seconds mein bolna shuru) aur task success rate.

Voice loop — sunna, bolna, rukna

Loop ke steps: VAD (bolna shuru/band) → STT streaming → LLM streaming → TTS streaming. Chaaron streaming honi chahiye — batch pipeline realtime nahi lagti.

async def voice_loop(mic, tts, stt, llm):
    speaking = False
    async for audio in mic.stream():          # chhote frames
        if vad.ends_utterance(audio):         # user ne bolna band kiya
            text = await stt.finalize()
            if not text:
                continue
            if speaking:
                tts.stop()                     # barge-in: user bole toh interrupt
            first_chunk = False
            async for token in llm.stream(text):
                if not first_chunk:            # pehla word turant bolo,
                    tts.start(token)           # poora response mat wait karo
                    first_chunk = True
                else:
                    tts.feed(token)
            speaking = True

Vision path — image kaise aati hai

Image ko model tak pahunchana simple hai (upload → resize → encode → request), asli decision hai: OCR/VLM kab:

• Sirf text nikalna hai (invoice number, form fields) → OCR/parsing: sasta, fast, structured.
• Samajhna/describe karna hai (chart trend, photo ka issue, UI screenshot debug) → VLM: mahanga, lekin semantic.
• Bada image batch → resize + compression pehle; token cost image size par depend karti hai.

import base64

def image_part(path, max_side=1024):
    img = Image.open(path)
    img.thumbnail((max_side, max_side))          # budget control
    buf = io.BytesIO(); img.save(buf, format="JPEG", quality=85)
    b64 = base64.b64encode(buf.getvalue()).decode()
    return {"type": "input_image", "image_url": f"data:image/jpeg;base64,{b64}"}

resp = client.responses.create(
    model=VLM,
    input=[{"role": "user", "content": [
        {"type": "input_text", "text": "Is receipt ka total aur vendor batao."},
        image_part("receipt.jpg"),
    ]},
],)

Latency budget

Total turn latency = VAD lag + STT finalize + LLM first token + TTS first audio. Ek example budget banao (apne numbers, apne targets) aur har stage par time measure karo:

• Target: user ke bolna band karne se pehla awaaz tak — seconds mein single digit, ideally ≤2s (yeh tumhara SLO hai, market claim nahi).
• STT finalize sabse bada lag maker hota hai — partial results stream karo.
• LLM: small model for chhote tasks, streaming on, prompt caching (L9.5) for system prompt.
• TTS: sentence-level synthesize — poora paragraph mat wait karo.

Testing — devices aur networks

• Low-end Android phone + hotel wifi par chala kar dekho — desktop demo sabse jhoota hota hai.
• Interruption tests: user bole beech mein, beech mein chhode, ekdum khali bole — teenon handle hon.
• Network drop:2 second ka disconnect — reconnect flow ya graceful message.
• Background noise: fan/music on karke VAD false positives count karo.

Exercise

Common mistakes

• Turn-based bana dena: User bole, agent poora soche-phir bole — IVR jaisa lagta hai. Streaming + barge-in se hi realtime feel aata hai.
• Sab kuch ek request mein wait karna: Poori response TTS ke liye ready karke phir bolna = lamba khaali gap. Sentence-level streaming karo — pehla word hi bol do.
• Desktop demo par hi rok dena: Voice products phones par toote hain (mic permission, background apps, networks). Kam se kam ek low-end phone par test karo.

while True:
    audio = mic.read()                 # sun rahe ho
    if vad.ends_utterance(audio):      # user ne bolna band kiya
        if speaking: tts.stop()        # barge-in
        reply = llm.stream(stt(audio)) # turant stream
        tts.start_first(reply)         # pehla word bolo, poora wait nahi