Lesson 80 — Intermediate

Context Caching & Efficiency

System prompt, tool schemas, few-shot examples — yeh har call same jaate hain. Providers ka prefix caching mechanism isi ko exploit karta hai; tumhara kaam structure aisa banao ki cache hit ho, aur phir number se prove karo.

⏱ 10 min• Intermediate• Tool Context Management

Cache mindset

WHAT

Stable context ko prefix mein aage rakhna (cache-friendly order) + har call ke tokens/cost ko measure karna. Optimization se pehle metrics.

WHEN

Har call par same instructions/tools/examples ja rahe hain — chat apps, agent loops, RAG pipelines mein sabse bada reusable chunk yahi hai.

WHERE

Assembly layer ka ordering (stable first) + usage logging (prompt tokens, cached tokens, cost/day, latency).

Mechanism simple hai: provider ek bada common prefix pehchan kar us hisse ko repeated prompt tokens par discount deta hai. Order hi strategy hai — beech mein ek dynamic line daali, prefix toot gaya.

Rule: Stable upar, changing neeche. Timeline/user-specific content hamesha tail mein — wohi cache ka darwaza hai.

Stable vs changing

Prompt ko do hisson mein todo — jo kabhi na badle aur jo har turn badle:

system = RULES + TOOLS + FEW_SHOTS     # stable — har call same prefix
messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": dynamic_input},  # changing sirf tail mein
]
# resp.usage se prompt/cached tokens log karo — hit rate wahi se aayegi

Stable mein: rules, tool schemas, static docs, few-shot examples.
Changing mein: user text, latest retrieval, current state, timestamps.
Interleave bilkul mat karo — system ke beech mein date/user name = miss every call.

Ek line ka nuksaan: "Aaj ki date: ..." system prompt ke beech mein daal di — prefix badal gaya, discount gaya. Tail mein rakho, rules mein nahi.

Measure — blind nahi

Usage fields: prompt tokens, cached tokens (provider agar report kare), output tokens.
Cache hit rate: cached/total prompt tokens — dashboard ka metric.
Cost/day before vs after — same workload par comparison.
Latency p50 delta — cache se speed bhi aati hai, woh number bhi chahiye.

Experiment: Same eval/workload par before/after run karo, phir rollout karo. "Lagta hai kam ho raha hai" optimization nahi hai.

Redundant tokens kaatna

Instruction echo: har user turn mein rules dobara likhna band — system prompt se milta hai.
Verbose templates + politeness padding — chhote shabd, wahi matlab.
Schema duplication: tool JSON aur prompt dono mein same list — ek jagah hi rakho.
Bade fixed examples jo kabhi change nahi hote — trim ya cache-friendly order mein.
Retrieval dedupe: ek hi fact teen chunks mein aaya toh ek chhota.

Chhota cleanup, roz ka discount: Har redundant token roz hazaron baar kharch hota hai. Ek baar ka trim, perpetual saving — yehi efficiency hai.

Exercise

Question: Prefix caching ke liye prefix kaisa hona chahiye — stable ya badalta? (ek shabd, English)

Question: Optimization se pehle kya karo — guess ya measure? (ek shabd, English)

Question: Cache-friendly order: stable content pehle, changing content ___ mein? (ek shabd, English)

Common mistakes

Dynamic line beech mein: Date, user name, timestamp system ke andar daal diya — prefix badal gaya, har call par fresh billing. Tail mein rakho.
Metrics ke bina optimize: Anusaar se chhota kar diya par number nahi dekha. Hit rate, cost/day aur latency — teenon chahiye rollout se pehle.
Schema do jagah: Tool definitions + prompt dono mein same JSON list — double kharcha. Single source of truth rakho.

Desi example: Hotel ka menu board

Hotel ka menu board fix rehta hai, sirf daily special chalk se badhta hai — board roz nahi banate. Stable prefix wahi board hai: roz same, cache mein baith jaata hai.

menu_board = rules + tools + examples   # roz same — cached
special    = aaj_ka_user_input          # chalk — tail mein

Interview angle — yahan se sawaal aate hain

Q: Prefix caching mechanism kaise kaam karti hai? (common prefix pehchan kar repeated prompt tokens discount — order se exploit)
Q: Kya cache todta hai? (beech ki dynamic line, reordering, schema change)
Q: Kya measure karoge? (cached vs fresh tokens, hit rate, cost/day, p50 latency)
Q: Redundant tokens kaise kaatoge? (echo removal, single-source schema, trim examples, retrieval dedupe)

Lesson complete?

Caching + measurement toolset mein. Agli lesson: in sabko jod kar poore agent ko wrap karne wali layer — harness engineering.