Lesson 105 — Advanced

Caching Architecture

Caching ko afterthought mat banao — AI system mein chaar alag caches hain jo latency aur cost dono bachate hain. Is lesson mein har cache ka role, safe cache key kaise banti hai, semantic caching ka risk knob, aur hit rate/savings kaise measure karte hain — assume nahi, measure.

⏱ 10 min• Advanced• Latency & Inference Optimization

AI system ki chaar caches

WHAT

Exact-response cache (same request → stored answer), prompt/context cache (static prefix reuse), retrieval cache (query → top-k docs), semantic cache (similar intent → shared answer) — chaar alag layers.

WHEN

Repeated queries ho, static context har call jaata ho, corpus stable ho, ya cost/latency ceiling chhoot rahi ho — tab. Har system par default nahi.

WHERE

App layer (response store), provider layer (prompt caching), index layer (precomputed embeddings/results), similarity layer (embedding search) — har jagah alag TTL.

Exact-response: normalized request ka hash → stored answer. Sabse safe, jab jawab timeless ho.
Prompt/context: lamba static prefix (system prompt, guidelines) provider par cache hota hai — prefill cheaper/faster. L6.7 mein detail.
Retrieval: query → top-k docs store; index update par invalidate — embedding compute bachti hai.
Semantic: alag wording, same intent → shared answer. Sabse zyada savings, sabse zyada risk.

Yaad rakho: Chaar caches alag jagah kaam karti hain — ek doosre ki replacement nahi. Design mein likho kaun si kahan lagti hai, warna sab "caching" bol kar skip ho jaata hai.

Kya safe hai — key, scope, TTL

def cache_key(req):
    norm = " ".join(req.prompt.lower().split())      # whitespace normalize
    scope = f"{req.user_org}:{req.locale}"            # ACL scope key KE ANDAR
    return sha256(f"{norm}|{req.model}|{req.temp}|{scope}")

key = cache_key(req)
hit = cache.get(key)
if hit and fresh(hit, ttl=ttl_for(req.topic)):
    return hit.answer

Key design: normalized prompt + model + params + user/tenant scope. Scope bhoola toh cross-user leak — sabse bada caching bug.
TTL by volatility: rates roz, policy docs hafte, evergreen content lambi — ek global TTL har cheez ke liye galat hota hai.
Invalidation events: naya index build, prompt version change, model swap → tags se related entries flush.
Mat cache karo: personalized outputs, sampling wale answers, high-risk irreversible decisions wale results.

Stale jawab = support ticket: Freshness aur savings ka trade-off seedha hai. Topic-wise TTL lagao, aur jo topics legal/policy jaise sensitive hain unki TTL chhoti rakho.

Semantic cache — aur uska risk knob

Semantic cache mein query ko embed karke store mein dhoondhte hain — meaning same ho toh answer share. Savings aksar sabse zyada yahin hain, par risk bhi yahin: galat hit = galat jawab, jo no-answer se bhi bura hai.

def semantic_answer(q, threshold=SIM_THRESHOLD):
    v = embed(q)
    near = store.search(v, top_k=1)
    if near and near.score >= threshold:   # threshold APNE data par tune karo
        return near.answer, "semantic-hit"
    return None                            # miss -> full pipeline chalega

Threshold hi risk hai: Bahut low → ghalat answer mil jaata hai; bahut high → savings zero. Threshold ko apne query set par tune karo, har hit log karo aur sampled review karo.

Hit rate aur savings — assume mat karo

Cache ka ek hi score matter karta hai: hit rate = hits / total lookups. Aur savings tabhi asli hain jab hits, saved model calls aur saved latency log ho. "Caching se bachega" — hamesha nahi; measure karo.

lookup -> hit?  yes: hits++    (saved: model call + latency)
               no : misses++ -> full pipeline -> store

dashboard: hits, misses, hit_rate, saved_calls, saved_ms
savings = apne REAL traffic se measure karo — estimate nahi

Cache bhi mehnga hota hai: Memory, invalidation logic aur ops cost hai. Hit rate low aaye toh cache hataana bhi valid decision hai — caching architecture ka part measurement hai, faith nahi.

Exercise

Question: Hits ko total lookups se divide karke jo ratio aata hai — do shabd, English?

Question: Alag wording par bhi same answer share karne wala cache — do shabd, English?

Question: Lamba static prefix jo provider khud cache karta hai — do shabd, English?

Question: Naye index build par related cache entries hataana — ek shabd, English?

Common mistakes

Bina ACL scope ke response cache: Key mein user/tenant scope nahi toh ek ka doosre ko jawab mil sakta hai. Scope key ka compulsory field hai, option nahi — pehle security, phir savings.
Ek global TTL: Volatile data lambi TTL par purana, stable data chhoti TTL par miss hota hai. Topic-wise TTL lagao — freshness curve data ke hisaab se.
Hit rate na dekhna: Cache add kar diya, savings ka koi proof nahi. Hits/misses/saved_calls log karo — warna dead cache bina kisi ke puche repo mein baithe rahega.

Desi example: Chai wale ka order book — caching ka desi analogy

Ek jaise orders pehle se bana ke rakho toh turant serve — par customer naya maange toh fresh banana padega. Caching wahi: common orders ka stock, aur TTL matlab kharaab hone se pehle phenk do:

exact cache  = pahle se pack kiya same order
prompt cache = masala dabba refill na karna (prefix reuse)
semantic     = naam alag, order ek jaisa
TTL          = stale hone se pehle hatao (freshness)

Interview angle — yahan se sawaal aate hain

Q: AI system ki kaunsi kaunsi caches hoti hain? (exact-response, prompt/context, retrieval, semantic — alag layers, alag TTL)
Q: Cache key kaise banaoge? (normalized prompt + model + params + tenant scope — scope bhoolna leak hai)
Q: Semantic cache ka risk aur uska control? (wrong hit threshold se — tune + sampled review + logging)
Q: Caching ka benefit kaise prove karoge? (hit rate, saved calls, saved latency apne traffic par — measure, assume nahi)

Lesson complete?

Caches ab architecture ka hissa. Agla lesson — Reliability & Guardrails: jab kuch toote tab bhi system useful rahe.