Lesson 44 — Beginner

Prompt Reliability & Debugging

Ek achha response dekh kar prompt ko final maanna sabse badi galti hai. Is lesson mein hum seekhenge prompt failures diagnose kaise karein, chhota test set kaise banayein (normal + edge + known failures), do versions ko same examples pe kaise compare karein, aur output variance/regressions kaise kam karein.

Failure diagnose karo — pehle symptom nahi, cause

Galat output dekho -> pehle yeh checklist chalao:
1. Instruction hi missing ya ambiguous hai?          (sabse common)
2. Do constraints ek doosre se takra rahe hain?       (short bhi chahiye + table bhi chahiye)
3. Example hi galat format dikha rahe hain?           (model example ko copy karta hai)
4. Input mein koi field/context hi missing hai?       (model guess kar raha hai)
5. Output requirement likhi hi nahi hai?              (format drift)

Yeh 5 causes hi 90% prompt bugs cover karti hain. Pehle list se tick karo, phir fix likho — bina diagnose kiye prompt badalna andheri mein teer chalana hai.

Chhota test set banao

TESTS = [
    {"name": "normal",     "input": "Parcel 5 din se nahi aaya", "expect": "SHIPPING"},
    {"name": "edge",       "input": "Refund ke saath coupon bhi nahi mila", "expect": "BILLING"},
    {"name": "known-fail", "input": "App mein discount dikh raha hai par bill mein nahi", "expect": "BILLING"},
]

def run_all(prompt):
    for t in TESTS:
        out = run(prompt, t["input"])
        ok = t["expect"].lower() in out.lower()
        print(t["name"], "PASS" if ok else "FAIL", "|", out)

• Normal cases: aam inputs jinke sahi jawab sabko pata hain — baseline.
• Edge cases: khali input, code-mix text, bahut lamba input, tricky wording — yahin se bugs nikalte hain.
• Known failures: purane bugs ke reproducer — fix ke baad woh dobara na tootein, isliye test set mein hamesha rehte hain.
• Shuruaat 10-15 cases se karo — badi harness ki zarurat nahi, roz chalne wali chhoti list kaafi hai.

Versions ko same examples pe compare karo

def compare(a, b, tests):
    """Dono prompt versions ek hi test set pe chalao."""
    for t in tests:
        out_a, out_b = run(a, t["input"]), run(b, t["input"])
        print(f'{t["name"]}: A={out_a!r} B={out_b!r}')

# Kabhi aisa mat karo: v1 ko ek ticket pe, v2 ko dusre ticket pe dekh kar decide karna.
# Inputs badal gaye toh fark input ka bhi ho sakta hai — confound ho jaayega.

A/B comparison ka rule ek hi line mein: inputs fixed rakho, sirf prompt badlo. Warna tum prompt ki quality aur input ki difficulty ko mix kar loge — aur galat conclusion nikaloge.

• Har change ke baad poora test set chalao — sirf woh 2 cases nahi jinpe tune dhyan diya tha.
• Ek waqt mein ek hi variable badlo — "3 cheezein badal di, ab better hai" se pata hi nahi chalega kisne improvement diya.
• Manual judging chahiye toh blind karo: model se "kaunsa jawaab behtar hai" poochho, order randomly badal ke.

Variance aur regression kam karo

• Variance: ek hi prompt, do baar chalaya — alag output. Temperature jaisi sampling settings kam karke (ya provider ke default reasoning pe chhod ke) output zyada stable hote hain — mechanism provider docs mein check karo.
• Format drift: kabhi bullet, kabhi paragraph. Output spec explicit likho ya schema lagao (pichhla lesson).
• Regression: naye fix se purana case toot gaya — isliye known-failures test set mein rehte hain, har change pe unhe bhi chalao.
• Prompt change log rakho: kya badla, kyun, kis case pe improve hua — warna 3 hafte baad khud nahi yaad rahega.

Exercise

Common mistakes

• Ek hi achha response dekh kar prompt final maan lena: Model stochastic hai — ek accha output luck bhi ho sakta hai. Poora test set chalao, tabhi confidence milega.
• Ek saath 3 cheezein badal lena: Prompt badalte waqt ek hi variable badlo. Warna improvement ya regression kis change se aaya — pata hi nahi chalega.
• Test set mein sirf normal cases rakhna: Edge cases (khali input, code-mix, lamba text) se hi bugs nikalte hain. Har test set mein kam se kam 2-3 edge aur known-failures rakho.

Question paper (test set) : 3 tickets — normal, edge, known-fail
Answer key (expected)     : SHIPPING, BILLING, BILLING

Teacher v1 (baseline): 2/3 pass
Teacher v2 (fix)      : 3/3 pass -> deploy
Teacher v3 (jugaad)   : 1/3 pass -> reject

Ek question pe acha aaya = kuch nahi; poora paper chalao.