Lesson 113 — Advanced

Experiment Tracking & Prompt Versioning

Bina history ke har change ek guess hai. Is lesson mein hum har run ko MLflow mein record karna, metrics aur prompts ko artifacts banana, runs ko compare karna, aur prompts ko ek registry mein version karke staging/production labels se deploy-rollback karenge.

Experiment kya hai aur kyun track karein

Sawaal simple hai: pichhle hafte wala behtar prompt ab kahan hai? Agar iska jawab "chat history mein tha" hai, toh tum experiment nahi, trial-and-error kar rahe ho. Tracking ka matlab hai har run ke saath ek named record jo baad mein compare ho sake.

MLflow mein run record karna

import mlflow

mlflow.set_experiment("rag-app")

with mlflow.start_run(run_name="chunk400-topk8"):
    # parameters — jo bhi behaviour badalte hain
    mlflow.log_params({
        "chunk_size": 400, "top_k": 8,
        "model": CFG.model, "temperature": 0.0,
        "tools": "search,fetch_doc",
    })
    # metrics — naapne wale numbers
    mlflow.log_metrics({
        "faithfulness": scores.faithfulness,
        "answer_relevance": scores.relevance,
        "p95_latency_s": scores.p95_latency,
        "tokens_per_request": scores.avg_tokens,
    })
    # artifacts — prompts, config, sample outputs
    mlflow.log_artifact("prompts/rag_system_v3.txt")
    mlflow.log_artifact("outputs/sample_run.jsonl")

• Params: chunk size, top-k, model, temperature, tool set — har wo knob jo jawab badal sakta hai.
• Metrics: eval scores, latency percentiles, tokens per request — numbers jo decide karenge kaun jeeta.
• Artifacts: exact prompt text, config snapshot, sample outputs — run ko dobara padhne layak banana.
• Run ka naam meaningful rakho — "chunk400-topk8" kabhi bhoolta nahi, "final_final2" kabhi yaad nahi aata.

Runs ka comparison aur decision

run              params                    faith  rel    p95    verdict
-------------------------------------------------------------------------------
baseline         chunk500 topk5            0.78   0.81   2.4s   reference
chunk400-topk8   chunk400 topk8            0.82   0.83   2.9s   quality up, slower
chunk400-topk8-r reranker on               0.86   0.84   3.6s   best quality
prompt-v4        v4 prompt, same retrieval 0.80   0.85   2.4s   relevance up, faith down?

# rule: ek metric upar, doosra neeche -> seedha promote mat karo,
# apna priority decide karo (quality-first ya latency-first)

• Side-by-side comparison ka fayda: pata chalta hai kaunsa change sach mein quality moved, kaunsa sirf noise tha.
• Do runs ka farq sirf ek parameter mein ho toh causal link clear hai; 4 changes ek saath = koi conclusion nahi.
• Har promote/reject ka ek line ka reason run mein likho — aadhi baad ki meetings isi se bachti hain.
• Quality upar par latency budget ke bahar — wo bhi "reject with reason" hai, failure nahi.

Prompt registry aur labels

prompt registry (Langfuse jaisa) — structure:

rag/system-prompt
  v1  (retired: faithfulness low)
  v2
  v3  <- label: staging
  v4  <- label: production

# PROMOTIONS sirf label hatti hai:
#   production: v3 -> v4     (text edit nahi, label move)
# ROLLBACK:
#   production: v4 -> v3     (ek API call, rebuild nahi)

fetch at runtime:
  prompt = registry.get("rag/system-prompt", label="production")
  # cache + pinned local fallback (next section)

• Registry ka fayda: prompt code deploy ke bahar independently version hota hai — chhote badlaav bina release ke chalte hain.
• Production text kabhi in-place edit mat karo — naya version banao, label move karo. In-place edit = rollback kabhi possible hi nahi.
• Har version ke saath author, date aur reason ho — review aur audit dono isi se hote hain.
• Traces mein prompt version likho — regression aaye toh seedha us version pe pahunch jaoge.

Runtime fetch, fallback aur rollback

import time

cache = {}

def get_prompt(name: str) -> str:
    try:
        p = registry.fetch(name, label="production", timeout=2.0)
        cache[name] = p          # cache update — kal bhi yahi milega
        return p
    except RegistryDown:
        if name in cache:
            return cache[name]   # stale par chal raha hai
        return PINNED_FALLBACK[name]   # repo mein versioned text

# rollback = ek call:
registry.set_label("rag/system-prompt", "production", to="v3")
# koi rebuild nahi, koi deploy nahi — agli request se hi purana prompt

• Registry down ho toh app bhi down nahi honi chahiye — cached version chalata rahe, cache na ho toh repo mein pinned fallback.
• Fallback text repo mein versioned hai — registry ke bina bhi app deterministic chal sakti hai.
• Rollback itna sasta hona chahiye ki koi "dekh lete hain" na bole — jitna rollback easy, utna jaldi fix.
• Rollback ke baad wahi post-mortem: ye prompt hua kyun, aur eval mein pakda kyun nahi.

Exercise

Common mistakes

• Baseline record kiye bina experiments chalana: Number down aaya toh pata hi nahi chalega ki naya change kharaab hai ya baseline hi weak tha. Fix: pehle current setup ka baseline run, uske baad hi koi badlaav.
• Ek run mein chaar changes lana: Kaunsa change kaam kar gaya — conclude hi nahi kar paoge. Fix: ek experiment, ek knob; baaki combos alag runs mein.
• Prompt ko code ke saath deploy karna: Chhote prompt fix ke liye poora build-deploy — aur rollback ke liye code revert. Fix: prompt registry, runtime fetch, label se promote/rollback.
• Registry ka fallback na rakhna: Registry outage ne poora application down kar diya. Fix: cache + repo mein pinned fallback, dono tested.

batch card  -> run record (params + metrics)
staging pass-> label: staging
approved    -> label: production (batch release)
complaint   -> usi batch ka formula dekho (trace),
              previous batch dobara issue karo (rollback)