Lesson 143 — Advanced

On-device & Edge Modalities

Chhote speech aur vision models ko user ke device pe chalaana privacy, offline operation aur zero per-request cost deta hai — par saath mein memory, thermal aur quality ke limits bhi aate hain. Is lesson mein hum dekhenge quantization kya karta hai aur quality kahan girti hai, phone/laptop ki hardware ceilings, on-device se kya milta hai, aur device vs self-host vs API ka per-modality decision.

On-device modality kya hai

Do wajah se modalities device pe suit karti hain: models chhote ho sakte hain (specialized STT/vision models chhote batches mein chal sakte hain) aur input hi private hai — awaaz aur camera frames har waqt bahar bhejna consent-wise aur bill-wise dono mehnga. Plus network mein round trip aksar sabse slow part hota hai.

Quantization aur uska trade

# Quantization: precision kam -> size/speed behtar, par quality ka trade
params = 3_000_000_000        # example model size
fp16_bytes = params * 2       # -> 6 GB class
int8_bytes = params * 1       # -> 3 GB class
int4_bytes = params // 2      # -> 1.5 GB class

# Device pe zyada tar int4/int8 hi fit hota hai — par quality gir sakti hai
# Quality kahan PEHLE dikhti hai:
# numbers, rare words, accents, fine transcription, exact field reading

• Quantization matlab weights ko kam precision mein store karna — size aur compute dono ginte hain, isliye bada model chhota ho kar device pe fit ho jaata hai.
• Quality trade har task pe barabar nahi dikhta: pehle wo girti hai jahan exactness chahiye — digits, jargon, names, chhote text.
• Quantization ke baad thoda sa calibration/fine-tune karne se quality kaafi recover hoti hai — plain post-training se behtar chalta hai.
• Validate hamesha apne task pe: same audio/images pe full-precision vs quantized model chalao aur farq gino — spec sheet se nahi.

Phone aur laptop ki limits

• Memory ceiling: OS aur app ke baad jo bacha wahi model ka hai — peak requirement chahiye, average nahi; OOM matlab app crash.
• Thermal throttling: sustained inference se device garam hoti hai, clock girta hai, latency badhti hai — lambi streams mein yahi dikhta hai.
• Battery cost: continuous mic/camera + inference battery khaata hai — "always on" feature ke liye duty-cycle socho (wake-word pe hi chalao).
• Device matrix:

# Per modality decide karo — poora stack ek jagah mat pakdo
def place(small_model, privacy_strict, offline_needed, volume):
    if offline_needed or privacy_strict:
        if small_model:
            return "device"      # private + offline, chhota model fit hai
        return "self-host"       # private chahiye, par device pe fit nahi
    if volume_high and cost_sensitive:
        return "self-host"       # steady volume pe per-call API cost bachti hai
    return "api"                 # default: zero ops, best quality

api        -> bijli vibhag: zero maintenance, bill per use
self-host  -> borewell: privacy + volume pe sasta, par pump tumhara
device     -> overhead tank: offline, free, capacity limited
har modality ke liye alag decide karo — sab ek hi source nahi