Lesson 56 — Intermediate

Baseline RAG from Scratch

Framework se pehle pipeline apne haath se likhna zaroori hai — tabhi andar kya chalta hai dikhta hai. Is lesson mein hum retrieve → assemble → generate ka poora loop, grounded answers ke liye prompt design, weak/missing context handle, aur k/ordering/total context ke effects dekhenge.

Pipeline by hand

def retrieve(query, k=5, where=None):
    hits = hybrid_search(query, dense, sparse, k=k, where=where)
    return hits

def assemble(hits):
    blocks = []
    for i, h in enumerate(hits, start=1):
        blocks.append("[{}] {}\n{}".format(i, h["meta"].get("heading", ""),
                                            h["text"]))
    return "\n\n".join(blocks)

def rag(query, k=5):
    hits = retrieve(query, k)
    context = assemble(hits)
    prompt = GROUNDED.format(context=context, question=query)
    return chat(prompt), hits          # hits hamesha log ke saath

Grounded answers ka prompt

Tum ek assistant ho. Sirf neeche diye gaye CONTEXT se jawaab do.

Rules:
- Har claim ke saath [n] citation do, jahan n context block ka number hai.
- Context mein jawab na ho toh likho: "Insufficient information".
- Guess, extrapolation aur bahar ki knowledge mat jodo.
- User ki bhasha (Hindi/Hinglish) mein hi jawab do.

CONTEXT:
{context}

QUESTION: {question}

• Instruction upar, data neeche — delimiters ke saath (L3.1 ka wahi funda, ab retrieval ke saath).
• Citation contract explicitly likho: "[n]" format, block numbers kahan se aaye — model ko structure chahiye.
• Abstain option do hi nahi, zaroori hai — "Insufficient information" likhne ko bolo, warna model guess karega.
• Question ko context se alag section mein rakho — data aur instruction ka mix-up nahi hona chahiye (prompt injection bhi yahi hai).

Weak ya missing context handle karna

MIN_SCORE = 0.35     # apne eval se tune karo, kisi aur ka nahi

def safe_rag(query, k=5):
    hits = retrieve(query, k)
    if not hits or hits[0]["score"] < MIN_SCORE:
        return {"answer": "Insufficient information",
                "citations": [], "abstained": True}
    prompt = GROUNDED.format(context=assemble(hits), question=query)
    return {"answer": chat(prompt), "hits": hits, "abstained": False}

• Top score threshold sabse sasta guardrail hai — weak retrieval pe model ko jawab dene hi mat do.
• Threshold ka distribution dekh ke chuno: relevant queries ke scores aur out-of-scope queries ke scores beech ka gap.
• Retrieval kaafi weak ho toh query ko rewrite karke dobara try karo (M4.11) — pehla hi attempt final nahi hota.
• Log mein abstain rate track karo — bahut zyada abstain = retrieval weak, bahut kam = model guess kar raha hoga.

k, chunk ordering aur total context ka asar

# k sweep: har k pe apne set pe answer quality dekho
for k in [1, 3, 5, 10, 20]:
    acc = run_eval(lambda q: rag(q, k=k))
    print("k =", k, "score =", acc)
# aksar curve: 1 se 5 tez sudhar, 5-10 dheere, uske baad distract/flatten

• k badhao toh coverage badhti hai, par distractors bhi — aur tokens/latency dono badhte hain.
• Ordering: sabse relevant block upar rakho, aur LLM ko batao ki priority order mein diya gaya hai.
• Dedupe: ek hi doc se 4 similar chunks top-k mein ho toh unhe merge/limit karo — variety chahiye.
• Total context = k × chunk size. Ek bhi lever bigda toh prompt 2-3x ho sakta hai — cost aur quality dono pe nazar rakho.

Exercise

Common mistakes

• Bina weak-context guard ke chalana: Retrieval khaali ya weak ho toh model guess karega — hallucination seedha. Fix: score threshold + abstain option prompt mein, aur log mein track.
• Koi eval nahi, k=10 default: Bina sweep ke k chunna guess hai. Fix: k=1/3/5/10/20 pe apne set pe score, latency aur tokens compare karo.
• Hits na log karna: Answer galat aaya, par pata hi nahi retrieval sahi thi ya nahi. Fix: har request ke saath query + hit ids + scores structured log mein.

order note (query)
  -> saman uthao (retrieve top-k)
  -> recipe ke saath plate karo (assemble)
  -> chef banaye (generate)
saman missing? -> "aaj ye dish nahi ban sakti" (abstain) - better than guess