Lesson 47 — Beginner

Capstone — Optimize a Real Prompt

Is capstone mein tum poore level ko ek project mein jodoge: ek real support-ticket triage prompt lena, baseline banana, test set banana, ek-ek karke improve karna, aur quality/latency/tokens pe compare karke final version ship karna. Yehi workflow tum production prompts ke saath roz chalaoge.

Kya banayenge

Workflow poore level ka recap hai: anatomy → zero/few-shot → decomposition → system prompt → output spec → test set → version + cost. Ek-ek step mein inhi concepts ko lagao.

Step 1 — Baseline aur test set

BASELINE = """Tum ek support assistant ho. Ticket ka jawaab do."""

TESTS = [
    # (name, ticket, expected_category, expected_urgency)
    ("normal",  "Order 3 din late hai, kya karu?",                  "SHIPPING", "high"),
    ("edge",    "Refund nahi mila aur coupon bhi gayab",            "BILLING",  "high"),
    ("ambiguous", "Cheezein theek lag rahi hain bas confirm karna tha", "OTHER", "low"),
    ("oos",     "Kal cricket ka score batao",                       "OTHER",    "low"),
    ("lang-mix","Mera paisa nahi aaya, refund chahiye turant",       "BILLING",  "high"),
]

def score(prompt_fn):
    passed = 0
    for name, ticket, cat, urg in TESTS:
        out = prompt_fn(ticket)
        ok = cat in out and urg in out
        passed += ok
        print(f"{name}: {'PASS' if ok else 'FAIL'} -> {out}")
    return passed, len(TESTS)

• Baseline 0/5 ya 1/5 pe hoga — woh theek hai, woh tumhara starting point hai.
• Test set mein normal + edge + out-of-scope + code-mix language rakhi hai — M3.6 ki checklist yahin lagao.
• Expected values human judgement se aate hain — ticket dekh ke khud decide karo ki sahi category/urgency kya honi chahiye.

Step 2 — Iterate karo (ek-ek change)

# v2 — anatomy apply ki: task + labels + output spec + 1 example (few-shot)
SYSTEM: Tum "ShopKart" ticket-triage assistant ho.
Task: ticket ko (a) category — BILLING/SHIPPING/PRODUCT/OTHER, (b) urgency — low/medium/high
      classify karo aur ek-line next-action batao.
Rules: scope se bahar ho toh OTHER bhejo. Guess nahi, jaldi ki dhamki bhi maani nahi.
Format: sirf ek JSON line, extra text nahi.

Ticket: "Refund 10 din se nahi aaya, coupon bhi nahi mila"
{"category": "BILLING", "urgency": "high", "next_action": "refund status check karke 24h mein update do"}

Ticket: "<yahan live ticket aayegi>"

• Ek change per version: v2 = labels + output spec. Score dekho. v3 = examples add kiye. Phir dekho.
• Score same raha lekin output parse hone layak ho gaya — woh bhi improvement hai (reliability).
• Koi case bigda? Usse batao kis change se hua — rollback ya tweak, bina drama ke.
• Har version ke baad score + 1-line note prompt file ke header mein likho.

Step 3 — Compare, measure, ship

import json, time

def eval_prompt(prompt_fn, verbose=True):
    passed, total = 0, 0
    for name, ticket, cat, urg in TESTS:
        total += 1
        t0 = time.time()
        out = prompt_fn(ticket)
        lat = time.time() - t0
        try:
            data = json.loads(out)                 # parse fail = fail (schema rule)
            ok = data["category"] == cat and data["urgency"] == urg
        except (json.JSONDecodeError, KeyError):
            ok = False
        passed += ok
        if verbose:
            print(f"{name}: {'PASS' if ok else 'FAIL'}  {lat:.1f}s")
    print(f"score: {passed}/{total}")
    return passed

• Final decision rule: pehle quality (score max), phir tie-break — kam tokens, kam latency.
• Tokens dekhne ke liye API usage field check karo — input kitna ja raha hai woh bada lever hai (purani history, extra context hatao).
• Ship karte waqt: final prompt ko v-final naam se git commit karo, header mein model + score + date likho.
• Prompts/triage/current.md update karo — production wahi point karega.

Exercise

Common mistakes

• Bina baseline ke optimisation shuru karna: Pehle "achha" prompt likh liya, phir pata chala theek hi nahi tha. Baseline + test set pehle, polish baad mein — warna improvement prove hi nahi kar paoge.
• Ek improvement ko hi final maan lena: v2 dekh kar band kar diya. Do-teen versions try karo, har baar poora test set chalao — kabhi kabhi v3 chhota context le kar bhi behtar score karta hai.
• Sirf prompt likhna, measurement skip karna: Score, latency aur tokens na loge toh optimisation andheri hai. Har version ka header note (score + date + change) hi tumhara proof hai.

KOT (ticket):
  "Table 7 — paneer nahi mila, customer ko jaldi hai"
      |
      v
Waiter ka rule (prompt):
  section = STARTER, priority = HIGH, action = replacement bhejo
      |
      v
Kitchen (app code) sirf wahi karta hai jo KOT allow karta hai
  -> prompt decide karta hai, code permission deta hai. Done.