Lesson 54 — Intermediate

Capstone — Build a Hybrid Search Engine

Is capstone mein tum M4.1-M4.6 ko ek project mein jodoge: corpus se lekar hybrid search tak — ingest → chunk → dense index + BM25 index → RRF fusion → filters → apne query set pe eval. Ye pipeline agle half (RAG) ke liye tumhara retrieval backbone banegi.

Kya banayenge

Scope deliberately chhoti rakho: ek folder of documents, ek eval set, ek search function. Lekin andar ke sab parts real hon — chunking strategy, embeddings, BM25, RRF aur threshold. Yehi chhoti cheez hi production jaisi hai.

Step 1 — Pipeline banao

from collections import Counter

def build(corpus):
    dense, sparse = [], []
    for doc in corpus:
        pieces = heading_chunks(doc["text"], size=800)   # M4.2
        for i, piece in enumerate(pieces):
            cid = doc["id"] + "#" + str(i)
            dense.append({"id": cid, "vec": embed(piece),
                          "text": piece, "meta": doc["meta"]})   # M4.3
            sparse.append({"id": cid, "tf": Counter(piece.lower().split())})
    return dense, sparse

# fingerprint se re-run incremental: badle hue docs hi process (M4.1)

• Chunking: recursive/structural splitter + heading metadata (M4.2) — ek function, dobara mat likho.
• Dense side: embeddings batch mein, normalize ke saath (M4.3); sparse side: token counts + df stats (M4.6).
• Har record ka id same rakho dono lists mein — fusion isi id se hoga.
• Dedup: fingerprint se duplicate docs pehle hi hata do (M4.1).

Step 2 — Search, filters aur fusion

def hybrid_search(query, dense, sparse, k=10, rrf_k=60, where=None):
    d_rank = dense_rank(query, top=50, where=where)      # M4.4/M4.5
    s_rank = bm25_rank(query, sparse, top=50, where=where)
    fused  = rrf([d_rank, s_rank], k=rrf_k)              # M4.6
    return fused[:k]

# CLI
# python search.py "refund kab milega" --k 5 --dept finance

• Har retriever se 50 (wide) candidates lo, fusion ke baad 10, LLM ko aage 4-5 — ye layering isi lesson ki hai.
• Metadata filter dono retrievers pe same hona chahiye — warna fusion mein mismatch aayega.
• k=10 se pehle threshold check: top result ka score bahut weak ho toh "koi nahi mila" bhi ek jawab hai.
• CLI flags (--k, --dept, --exact) se tumhari queries test harness chal sakti hain.

Step 3 — Eval karo aur ship karo

QSET = [
    ("refund policy kya hai",              "doc-12#0"),   # paraphrase
    ("INV-4821 status",                    "doc-7#2"),    # exact ID
    ("kya return 7 din mein hota hai",     "doc-3#1"),    # Hinglish
    ("kya kal ka cricket score batao",     None),         # out of scope
]

def hit_at_k(engine, k):
    hits = 0
    for q, gold in QSET:
        if gold is None:
            continue
        top = engine.search(q, k=k)
        hits += gold in [h["id"] for h in top]
    return hits / 3        # ye TUMHARA score hai, koi benchmark nahi

• Query set mein variety rakho: paraphrase, exact ID, Hinglish, aur ek out-of-scope — ye chaar types hamesha.
• Dense-only, sparse-only aur hybrid teeno chalao — kis ka hit-rate kitna, wahi proof hai hybrid chunne ka.
• Har run ka result log karo (query, top ids, scores) — regressions pehle se pata chal jaate hain.
• Ship: module + CLI + eval script git mein, README mein setup aur do example queries.

Exercise

Common mistakes

• Pehle code phir query set: Bina gold queries ke tumhe nahi pata engine chal bhi raha hai ya nahi. Fix: pehle 15-20 queries likho, phir engine banao, har change pe wahi chalao.
• Sirf dense wala hybrid: BM25 side integrate hi nahi, "hybrid" sirf naam ka. Fix: do lists, same ids, RRF — teeno steps alag-alag test karo.
• Fusion ke baad filter bhool jaana: Filtered query pe rank lists hi alag hain — filter pehle ya baad, decide karke dono jagah lagao.

photo search (dense) : [Rahul? , Rahul2? , Rahul3?]
naam search (sparse) : [Rahul exact, Rahul-old]
        | RRF = sum 1/(k+rank)
final : Rahul -> sabse upar (dono lists mein top)