Lesson 139 — Advanced

Audio: Realtime Voice Agents

Voice agent kaam tabhi karta hai jab jawab insaan jaisi deri se aaye — aur deri har millisecond mein banti hai jo tum control kar sakte ho. Is lesson mein hum voice loop ka latency budget, cascaded vs speech-to-speech architecture, VAD endpointing, barge-in, partial transcripts, tool calls ke waqt ka silence aur bad call ka reconstruction dekhenge.

Voice loop kya hai

Ek turn hota hai: user bolta hai, tum uska bolna band hona pehchante ho, tabhi transcript model ko jaata hai, jawab banta hai aur playback shuru hota hai. Ye chain chhote-chhote timed parts mein tootti hai — aur jis deri se user ke beech gap sunai deta hai, wahi is domain ka asli product problem hai.

Latency budget — milliseconds kahan jaate hain

# Ek turn ka latency budget (ms) — illustrative targets, apna khud ka measure karo
capture   = 50     # mic buffering + frame collection
network   = 100    # upload + download legs
stt       = 300    # transcript banana (endpoint ke baad)
llm       = 500    # sochna (first token tak)
tts       = 200    # pehla audio chunk
--------------------------------------------
total     = 1150   # 1 sec+ = awkward silence

# Bachaane ke tareeke: overlap (niche), streaming, chhota prompt,
# aur waiting ke waqt holding phrase — sab mila kar budget antar

• Har stage pe timestamp lagao aur p95 pe dekho — average se pata nahi chalta ki user ko kab gap dikha.
• Sabse bada saving overlap se aata hai: STT user ke bolte hi chal sakta hai, aur TTS model ke partial output pe shuru ho sakta hai.
• Asli metric time-to-first-audio hai — poora jawab banne ka wait matter nahi jab tak pehla shabd bol diya gaya.
• User ko dikhta total hai: uske andar capture, network, STT, LLM aur TTS sab count hote hain — har stage ka apna quota chahiye.

Cascaded vs speech-to-speech

# Cascaded (tumhara control): teen steps, teeno inspect ho sakte hain
audio -> stt -> transcript -> llm -> text -> tts -> audio
# har stage ka apna log, apna eval, apna swap

# Speech-to-speech (ek call): audio in, audio out
audio -> realtime_model -> audio
# kam round-trips, par andar ka path chhupa rehta hai

# Compare karte waqt columns:
# latency | cost (per min vs per step) | debuggability | tool use

• Cascaded mein har stage apna latency aur apna failure deta hai — STT ne galat sun liya toh LLM ko bhi wahi galat text milega (garbage in, garbage out).
• Speech-to-speech mein andar ka reasoning path chhupa rehta hai — debugging, compliance logs aur citations dono mushkil.
• Cost ka shape alag hai: cascaded mein har step ka per-unit charge, speech-to-speech mein aksar per-minute billing — apne call volume pe dono ka hisaab lagao.
• Control: cascaded mein system prompt, tool calls aur guardrails bilkul waise hi lagte hain jaise text LLM pe — wahi code reuse.

Endpointing aur barge-in

# Endpointing: kya user ne sachmuch bolna band kiya?
def should_end_turn(audio_stream, silence_ms=700, frame_ms=20):
    silent = 0
    for frame in audio_stream:
        if is_speech(frame):
            silent = 0
        else:
            silent += frame_ms
        if silent >= silence_ms and last_hypothesis_stable():
            return True          # turn end — abhi transcript ko bhejo
    return False

# Barge-in: user ne agent ke jawab ke beech mein bol diya
def on_user_speech_during_playback():
    speaker.stop()               # playback rok do
    discard_unplayed()           # baaki jawab phenk do
    cancel_pending_generation()
    state.commit(heard_only())   # sirf jo suna wahi state mein rakho

• Fixed silence timeout sabse aasan aur sabse galat: sochne wala user (1-2 sec ka pause) aur background chatter dono turn-end jaisa dikhte hain. Adaptive threshold ya speech-resume handling chahiye.
• Partial transcript pe turant action risky hai: user abhi bol raha ho sakta hai. Long actions (tool call, order place) sirf endpoint ke baad start karo; partials ko sirf pre-fill rakho.
• Barge-in sirf playback rokne se khatam nahi: model ka planned response, TTS queue aur pending state teeno cancel/rollback karo — warna agent ko lagega usne wo bol diya jo user ne suna hi nahi.
• Isliye conversation history mein wahi jaata hai jo actually played out — state consistency after a cut-short turn hi iska test hai.

Silence, tool calls aur bad calls

• Tool call 3 sec ka hai toh silence nahi: pehle chhota acknowledgement bolo ("dekhta hoon"), phir result ya fallback bolo. Tool fail ho toh plan B ready — dobara silence nahi.
• Network loss / dead air: holding phrase + reconnect attempt + graceful exit. Bina wajah ki pause user ko call drop lagti hai.
• Noise, crosstalk, bad mics: VAD ko zyada aggressive mat banao, noise suppression pehle stage pe rakho, aur confidence low ho toh confirm-repeat ("pooch raha hoon: order cancel kar doon?").
• Reconstruction: har session ka audio, transcript, per-stage timings aur turn boundaries log karo — bug repro aur quality review isi data se hoti hai.

user bolta hai -> VAD sunta hai -> ruk gaya? (endpointing) -> STT -> LLM
LLM bolta hai -> TTS -> speaker
user bole beech mein -> playback stop + discard + state rollback (barge-in)

tool call chal rahi hai -> "dekhta hoon" (holding phrase) -> result/fallback
session khatam -> audio + transcript + timings + turn boundaries logged

Exercise

Common mistakes

• Fixed silence timeout se endpointing karna: Sochne wala pause bhi turn maan liya jaata hai aur user ke beech ka shabd kat jaata hai. Fix: adaptive threshold + speech-resume handling, apne typical pauses pe tune karke.
• Barge-in pe sirf playback rok dena: Model ka baaki response state mein chala gaya, agent dobara wahi dohrata hai jo user ne suna hi nahi. Fix: generation cancel, unplayed audio discard, state rollback — sirf actually-played wala hissa commit karo.
• Tool call ke waqt dead air chhod dena: Teen second ka silent gap user ko lagta hai call drop ho gayi. Fix: acknowledgement phrase pehle bolo, result aur failure fallback script likha hua rakho.
• Partial transcript pe action le lena: User bol raha hai, tumne galat guess par order place kar diya. Fix: actionable intent sirf endpoint ke baad lo; partials ko sirf UI pre-fill tak simit rakho.

caller bolta hai  -> chup suno (VAD) -> ruk gaya? (endpointing) -> aage badho
agent bol raha hai -> caller bole -> turant ruko (barge-in)
wait lambi        -> holding phrase -> warna caller phone rakh dega
accha receptionist = accha voice agent loop