Lesson 74 — Intermediate

The Context Window as a Resource

Context window har model call ka total input hai — system prompt, history, retrieved chunks, tools, working state. Yeh ek budget hai: jitna andar, utna cost aur latency, aur budget ke andar rehne par quality bhi sharp. Is lesson se budget mindset set hoga.

Budget mindset

Rule ek line mein: context ko input mat gin, budget gin. Har section ka apna kharcha hai aur sabka sum hi tumhara per-call bill banata hai.

Context rot & lost-in-the-middle

Bade context ka side effect — context rot: model ka attention distract hota hai, purane contradictory instructions mein ulajhta hai, aur beech ki baatein gira deta hai.

• Lost-in-the-middle: shuruat aur end ke facts sharp, beech wale hisse ka model ko kam yaad rehta hai — critical rules corners par rakho.
• Conflicting context: purana instruction naye ke saath takraya toh dono mein se ek follow hoga — decide kaun karega yeh koi nahi batata.
• Stale context: hafte puraana config, chhod diya gaya plan — sab abhi bhi window mein baithe hain model ko guide kar rahe hain.
• Growth: har turn par history, retrieval aur observations add hote hain — bina plan ke budget ek ghante mein phat jaata hai.

Budget audit

BUDGET = {
    "instructions": 600,    # system prompt + rules
    "history": 1500,        # recent turns
    "retrieval": 1800,      # evidence chunks
    "tools": 900,           # tool schemas
    "state": 300,           # plan / working notes
}   # total = 5100 tokens ka working budget

def audit(parts):
    for name, text in parts.items():
        used = count_tokens(text)
        cap = BUDGET[name]
        flag = "  <-- OVER" if used > cap else ""
        print(f"{name:>12}: {used:5d} / {cap:5d}{flag}")

Yeh numbers sirf example hain — apne use-case ka budget banao. Instructions stable hain isliye chhoti rakho, retrieval task par vary karta hai, history sabse zyada badhta hai. Deploy se pehle audit chalao.

Leak diagnose kahan karna

• Instructions + tools constant hain? → caching opportunity hai (L6.7).
• History per turn kitna badh raha hai? → compaction chahiye (L6.3).
• Retrieval har turn badh raha ya fixed? → RAG relevance ka masla (L4).
• Kya ek hi fact do sections mein duplicate hai? → redundancy kaat do (L6.2).

Exercise

Common mistakes

• Bada window hi solution maanna: Cost aur rot dono badhte hain. Pehle budget trim karo (summarize/offload), window size last resort hai.
• Har turn par poora history bhejna: Bina plan ke growth unbounded hai — ek ghante ki chat bhi20k tokens par. Rolling summary ka trigger do.
• Sections ka size track nahi karna: Blind assembly mein leak dikhta nahi. Per-part token audit logger mein daalo — yahi debugging ka pehla step hai.

thali = instructions + history + retrieval + tools + state
# Ek section over = doosra squeeze — isliye har section ka apna portion set karo