Lesson 137 — Advanced

Audio: How Sound Becomes Tokens

Audio bhi tokens ban jaata hai — par uska raasta image se thoda alag hai. Is lesson mein hum dekhenge waveform se spectrogram/learned tokens tak ka safar, audio ka per-second cost model, transcript-then-prompt vs audio-native architectures, aur transcript kya-kya gawa deta hai — tone, hesitation, sab kuch jo "theek hai" ko "theek hai…" aur "THEEK hai" mein baantta hai.

⏱ 10 min• Advanced• Vision: Image & Video Generation

Waveform se tokens tak

Kya hai?

Audio model ke andar jaane se pehle waveform (hawa ke pressure points ki sequence) ko spectrogram ya learned audio tokens mein badla jaata hai — phir ye bhi ek token stream ban jaati hai jo transformer ko milti hai.

Kab kaam aata hai?

Jab input mein call recordings, meetings, voice notes ya lectures hon — aur sawaal sirf transcribed text se nahi hal hote (tone, emphasis, speaker).

Kahan milta hai?

Speech-to-text models, audio-native multimodal models, aur realtime voice APIs mein — teeno jagah conversion hota hai, bas kahan hota hai wo alag hai (pipeline mein STT ke andar, native model mein andar).

Raasta simple hai: sample → frame → spectrum → tokens. Har second mein hazaron samples aate hain, unhe chhote frames mein baant kar unki frequency-content (spectrogram) nikaali jaati hai, aur ye visual pattern ek vision-jaisa encoder tokens mein badal deta hai. Isliye audio models aksar "vision on spectrograms" jaisi approach use karte hain.

Mental model: Spectrogram ek "time ka photo" hai — x-axis pe time, y-axis pe frequency, brightness pe loudness. Is photo ko transformer padhta hai, isliye audio bhi tokens ban jaata hai.

Cost per second, not per word

# Audio ka cost model: tokens ~ duration. Khamoshi bhi bill mein hai.
def audio_token_estimate(seconds, sample_rate=16000):
    tokens_per_second = sample_rate // HOP_LENGTH   # provider docs se constant
    return int(seconds * tokens_per_second)

print(audio_token_estimate(30))     # 30-sec clip
print(audio_token_estimate(3600))   # 1-hour meeting = poora context kha sakti hai
# isliye long recordings pehle chunk ya summarize karo, seedha mat bhejo

Per-second billing: tokens duration ke saath badhte hain — words ke saath nahi. 10 minute khamoshi wali meeting bhi utni hi mehngi hai jitni 10 minute baat ki hui.
Latency bhi isi se: zyada audio = zyada tokens = zyada processing time. Realtime paths mein duration budget pehle se hona chahiye.
Chunking: long recordings ko alag chunks mein transcribe karo (timestamps ke saath), phir unhe text-level pe process karo.

Practical rule: Hour-long meeting ko seedha audio-native model mat do — pehle STT se text banao, uske baad sawaal text pe karo. Audio-native tab jab tone/emphasis hi question ka jawab ho.

Transcribe-then-prompt vs audio-native

# Path A: pipeline — pehle transcript, phir LLM
def transcribe_then_prompt(audio, question):
    text = stt_model.transcribe(audio)          # tone, hesitation gayab
    return llm.ask(text + "\n\nSawaal: " + question)

# Path B: audio-native — sound hi model ko jaata hai
def audio_native(audio, question):
    return model.ask(audio=audio, text=question)  # tone bhi context mein hai

Transcript-then-prompt: sasta, debuggable (tum transcript padh sakte ho), replay-friendly. Par jo shabdon ke beech hai wo sab gawa.
Audio-native: tone, emphasis, overlapping speech, background context — sab milta hai. Par cost/latency zyada, aur debug mushkil (model ne audio ka kaunsa hissa use kiya, pata nahi).
Decision rule: "Kya kaha gaya?" → transcript kaafi. "Kaise kaha gaya?" ya sarcasm/hesitation matter karta hai → audio-native.
Product mein dono: kai systems pehle transcript banate hain, aur sirf low-confidence ya tone-sensitive turns pe audio-native escalate karte hain.

Transcript kya gawaata hai: Tone, emphasis, hesitation, do logon ka ek saath bolna, emotion — ye sab transcript mein nahi aate. "Theek hai" warm bhi ho sakta hai aur irritated bhi; text dono ko same likhega.

Latency-sensitivity aur kahan tootta hai

Conversation = latency game: doc summary mein 2 second koi nahi ginta; voice reply mein 2 second dead air conversation maar deti hai. Isliye audio paths hamesha budgeted hote hain.
Accented speech: regional accents aur unfamiliar pronunciations transcription mein sabse common errors laate hain.
Code-switching: Hindi-English mix (Hinglish!) — ek sentence mein dono languages. Pure-English models yahin par confused hote hain.
Domain jargon: medical terms, product names, slang — model ne nahi suna toh phonetically galat likh dega. Vocabulary hints (custom vocabulary, prompt mein terms) se sudhaarta hai.
Burri mic & noise: fan, traffic, echo — sab transcription quality giraate hain. Audio preprocessing (denoise) madad karta hai.

Engineering takeaway: audio pipeline ka har stage — capture, network, transcription, synthesis — latency aur error dono add karta hai. Isliye production systems stage-wise metrics rakhte hain: transcription ka word error rate, end-to-end ka reply delay, aur per-turn ka failure rate. Bina numbers ke voice experience guesswork rehta hai.

Rule of thumb: Jab bhi audio input aaye, pehle decide karo: sirf content chahiye ya delivery bhi? Content → STT pipeline. Delivery (tone/emotion) → audio-native. Ye do sawaal aadha architecture decide kar dete hain.

Exercise

Question: Waveform ko time-vs-frequency wale visual representation mein kya kehte hain?

Question: Audio tokens zyada tar words se nahi, kis cheez ke saath badhte hain?

Question: Speech pehle transcript mein badal kar phir LLM ko dena — is architecture ko kya kehte hain?

Question: Transcript sabse zyada kis cheez ko gawaata hai jo tone/emphasis mein hoti hai?

Common mistakes

Long recording ko seedha audio-native model dena: Per-second tokens kha jaati hai, latency bhi. Fix: pehle STT se text banao, sawaal text pe karo; audio-native sirf tone-sensitive turns pe.
Transcript ko poori conversation samajh lena: Transcript sirf shabd hai — tone, overlap, hesitation gayab. Fix: jab sawaal "kaise kaha" pe ho, transcript ke saath audio context bhi do.
Per-word cost intuition audio pe lagana: Audio per-second chalta hai — khamoshi, noise, background sab bill mein. Fix: duration budget rakho, silent segments trim karo, chunking se long files todo.
Hinglish recordings ko English model pe chhod dena: Code-mixed speech transcription mein systematically galti karta hai. Fix: multilingual models chuno, domain vocabulary hints do, aur per-accent eval karo.

Desi example: Meeting ka stenographer

Stenographer sirf shabd likhta hai — uske notebook mein "theek hai" aur "THEEK hai" same dikhta hai. Par jo banda kamre mein maujood tha, usne tone suni. Transcript stenographer hai; audio-native model woh banda hai jo poora mahaul sun kar jawab deta hai:

STT pipeline:  audio -> shabd -> LLM          (tez, sasta, tone gayab)
audio-native:  audio + sawaal -> model       (tone bhi andar)

kya kaha  -> STT kaafi
kaise kaha -> audio-native chahiye

Interview angle — yahan se sawaal aate hain

Q: Audio tokens kaise bante hain? — Waveform se spectrogram/learned tokens, phir transformer; visual-encoder-jaisa raasta.
Q: Audio ka cost kis se chalta hai? — Duration (per second) se, words se nahi; khamoshi bhi bill mein hai.
Q: STT pipeline aur audio-native model mein farq? — Pipeline sirf shabd deta hai; native tone/emphasis/overlap bhi leta hai, par mehnga aur mushkil debug.
Q: Hinglish transcription mein common problem? — Code-switching: ek sentence mein do languages, pure-English models confuse hote hain.

Lesson complete?

Audio ka base clear — waveform se tokens, cost model aur transcript ki limits samajh aa gayi. Agla lesson — Audio: Speech to Text & Text to Speech — mein inhe practical pipelines mein jodenge: timestamps, diarization, WER aur voices.