Lesson 138 — Advanced

Audio: Speech to Text & Text to Speech

Speech-to-text aur text-to-speech dono hi production mein roz chalte hain — call transcriptions, captions, voice replies, IVR. Is lesson mein hum dekhenge STT model se actually kya milta hai (transcript + timestamps + confidence), batch vs streaming, diarization, WER ko kaise padhna hai, aur TTS mein voice, prosody, pronunciation aur streaming synthesis — hosted vs local ke trade-off ke saath.

STT se kya milta hai — transcript is not prose

# STT output ka typical shape — transcript + segments + timestamps
result = stt.transcribe("meeting.mp3", word_timestamps=True)

print(result.text)                    # ek blob: transcript
for seg in result.segments:            # time ranges ke saath segments
    print(round(seg.start, 1), "-", round(seg.end, 1), seg.text)
# segments ko downstream use karte hain: search, speaker join, highlights

Batch vs streaming transcription

• Batch: poori recording ek saath — best accuracy (context poora milta hai), simple integration. Offline jobs, meeting notes ke liye default.
• Streaming: bolte waqt hi partial text aata hai — captions, live agents, realtime commands ke liye zaroori. Par context limited hota hai, errors zyada.
• Partial vs final: streaming mein pehle partial hypothesis aati hai jo revise hoti hai — UI mein isliye text "settle" hone tak highlight karna chahiye.
• Word timestamps: dono modes mein mil sakte hain — subtitles, karaoke-style highlight, aur precise segment-joining ke liye zaroori.

# Streaming transcription — partial aur final hypotheses alag handle karo
for event in stt.stream(mic):
    if event.type == "partial":
        ui.show_live(event.text)        # user ko dikhaao, store mat karo
    elif event.type == "final":
        transcript.append(event)        # yahi asli record hai
        agent_pipeline.feed(event)      # downstream processing

Diarization aur WER

# Diarization ek ALAG step hai — kaun bol raha hai, ye STT nahi batata
text = stt.transcribe("call.mp3")
segments = diarizer.run("call.mp3")    # speaker A / speaker B ranges
merged = align(text, segments)          # "kaun kab bola" ka final record

# WER (word error rate) — transcription ka quality score
# WER = (substitutions + deletions + insertions) / total reference words
# 0.1 = 10% words galat — downstream task ke hisaab se acceptable ya nahi

• Diarization alag step hai: STT batata hai "kya kaha", diarizer batata hai "kisne kaha". Dono ko alag-alag run karke join karna padta hai — aur alignment bhi aksar galat hota hai (speakera overlap ho toh).
• Diaraation ki failures: ek hi speaker ke lamba gaps, do similar voices, background laughter — sab speaker boundaries bigadte hain.
• WER ko padho: overall WER se zyada important hai ki errors kahan hain — names? numbers? Hinglish code-switching? Downstream task sirf unhi se toota hoga.
• Errors ka asar: meeting summary ke liye 10% WER shayad chale; legal transcription ya medical notes ke liye nahi. Task-specific threshold rakho.

TTS — voice, prosody aur pronunciation

# TTS: text -> sampled audio in a chosen voice
audio = tts.synthesize(
    text="Your order total is 1,240 rupees. UPI ID: sales@shop",
    voice="female-a",          # provider ki voice list se
    speed=1.0,
)

# Pronunciation control: tricky words pehle se theek karo
text = text.replace("UPI", "U-P-I")            # acronyms spelled out
text = text.replace("1,240", "twelve forty")   # numbers bhi!

• Voice choice: aapke brand/persona se match karo; har voice ka apna range aur style hai — demo clips pe judge karo, naam pe nahi.
• Prosody: pace, pauses, emphasis — kuch models yeh sab expose karte hain (SSML ya parameters), kuch sirf ek voice+speed dete hain. Control kitni mil rahi hai, pehle check karo.
• Pronunciation traps: names, acronyms, numbers, code-mixed text — yahin TTS sabse zyada embarrass karta hai. Pehle se normalization/phonetic hints do.
• Streaming synthesis: poora utterance ka wait mat karo — sentence-level chunks stream karo, pehla audio sunte hi listener engaged rehta hai.

Hosted vs local speech models

• Hosted: zero infra, best-in-class voices/accuracy, per-minute/per-character cost. Data provider ke paas jaata hai — privacy reviews zaroori.
• Local: open models (Whisper-family, Kokoro-family jaise) GPU pe — data aapke paas, per-request cost zero, par hardware + maintenance aapki.
• Latency: local colo mein low latency ho sakta hai; hosted ke network round-trips add hote hain — realtime voice agents ke liye matter karta hai.
• Per-workload choose karo: customer calls (privacy-critical) local ya strict-DPA hosted; internal meeting summaries hosted batch; realtime agent mein latency budget decide karega.

Practical combo: ek modality ke liye ek hi provider mat pakdo — STT local, TTS hosted, ya ulta, depending on quality bar aur privacy. Har pipeline ke liye apna eval set rakho: 20-30 apni recordings (accents, noise, jargon ke saath) jo har candidate pe chalti hain.

Exercise

Common mistakes

• Transcript ko edited prose samajh lena: Filler words, galat punctuation, dropped words — sab aate hain. Fix: transcript ko noisy measurement maano; downstream validation rakho, critical values pe cross-check.
• Streaming partial hypotheses ko final maanna: Live captions mein text revise hota hai — store kar liya toh record hi galat. Fix: final events hi persist karo, partials ko sirf UI dikhao.
• Diarization ko STT ka hissa samajhna: STT words deta hai, speaker labels nahi — alag diarizer chahiye, aur unka alignment bhi fail hota hai. Fix: pipeline ke roop mein jodo aur boundary errors ka eval karo.
• TTS ko bina normalization ke chhod dena: Numbers, acronyms, names — TTS inhe galat bolta hai aur embarrassing results. Fix: text preprocessing (spell-out, phonetic hints) pipeline ka mandatory step banao.

recording -> likhne wala (STT) -> speaker tag (diarizer) -> notes
text -> bolne wala (TTS) -> audio
         numbers/names pehle spell-out karo, warna garbad