Lesson 53 — Intermediate

Sparse & Hybrid Retrieval

Embeddings kaam karte hain, par exact IDs, codes aur rare terms pe dense search chhod deta hai — aur wahi queries aksar production ki hote hain. Is lesson mein hum lexical retrieval aur BM25, sparse vs dense ka farq, RRF se hybrid fusion, aur exact-word wali queries pe hybrid ka asar dekhenge.

⏱ 20 min• Intermediate• Vector Databases & ANN Search

Lexical retrieval aur BM25

Kya hai?

Lexical (sparse) retrieval sirf shabd pe match karta hai — kya document mein wahi terms hain. BM25 iska classic formula hai: rare term ko zyada weight, baar baar aane ko cap, lambi document ko penalty.

Kab kaam aata hai?

Exact IDs, error codes, product codes, proper nouns, technical jargon — jahan matlab nahi, shabd chahiye. Aur jab corpus chhota/keyword-rich ho (logs, tags, specs).

Kahan milta hai?

Full-text search engines (Elasticsearch, OpenSearch) mein, Postgres full-text search mein, aur hybrid RAG pipelines mein ek retriever ke roop mein.

import math

def bm25(query, doc_tf, doc_len, df, n_docs, avgdl, k1=1.5, b=0.75):
    score = 0.0
    for term in query:
        tf = doc_tf.get(term, 0)
        if tf == 0:
            continue
        idf  = math.log(1 + (n_docs - df[term] + 0.5) / (df[term] + 0.5))
        sat  = tf * (k1 + 1) / (tf + k1 * (1 - b + b * doc_len / avgdl))
        score += idf * sat      # rare term -> zyada idf; length norm -> b

Teen factors: IDF = rare term ko zyada weight (dictionary mein kam aane wala shabd). TF saturation = term 50 baar aaye toh 5 baar wale se 10 guna nahi. Length norm = lambi document ka unfair faayda khatam.

Sparse vs dense — kya farq hai

                 sparse (BM25)              dense (embedding)
match kis pe      exact terms                meaning ke coordinates
tokens            sirf query ke words        poori vector, har dim
IDs/codes         strong                     weak
synonym/Hinglish  weak                       strong
index kya         inverted term lists        vectors + ANN index
build cost        cheap (counting)           embedding compute
kab naya term     turant (statistics)        model retrain/fine-tune

Sparse = statistics: query aur doc ke terms ka overlap. Naya shabd aaya toh usi waqt kaam karta hai, training ki zarurat nahi.
Dense = geometry: meaning ka pattern. "paisa wapas" aur "refund" ko ek jagah laata hai par "INV-4821" vs "INV-9917" nahi dekh paata.
Do alag failure zones hain — isliye sawaal ka ek tareeka kaam kare toh doosra miss karta hai.
Hybrid = dono ko ek saath chalao, phir unke answers fuse karo — coverage dono ki.

Muh jawab: Q: "BM25 outdated hai kya?" A: Nahi — exact terms wali queries pe abhi bhi top performer hai. Aaj ki badi retrieval systems bhi dense + sparse hybrid hi chalate hain.

Hybrid retrieval aur RRF

def rrf(rankings, k=60):
    """rankings = [[dense top ids], [sparse top ids], ...]
    RRF: har doc ka score = sum of 1/(k + rank) across lists"""
    scores = {}
    for rank_list in rankings:
        for rank, doc_id in enumerate(rank_list, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

# dense  top-50  +  bm25 top-50  ->  rrf  ->  top-10 bhejo

RRF (Reciprocal Rank Fusion): sirf rank pe kaam karta hai, raw scores ki scaling ki fikr nahi — isliye dense aur sparse ko bina normalize kiye milana aasaan.
k ek smoothing constant hai — classic default 60 rakha jaata hai; extreme top ranks ko control karta hai.
Doc dono lists mein top pe = uska score sabse uncha — yahi chahta hai hybrid.
Doosra tarika: normalized scores ka weighted sum — par weights tune karne padte hain, RRF zero-tuning kaafi chalta hai.

Scores ka trap: Dense score (0.0-1.0) aur BM25 score (jaise 14.2) ko seedha jod do toh BM25 hamesha jeetega — scales alag hain. Fusion ke liye rank-based RRF (ya normalization) zaroori hai.

Hybrid kab help karta hai

Exact IDs aur numbers: invoice, order, ticket numbers — sparse inhe pakadta hai jo dense chhodta hai.
Technical terms aur rare jargon: error codes, product SKUs, drug names, standard IDs.
Exact phrases: "zero trust architecture" jaise terms jahan word-overlap matter karta hai.
Dense ka strong zone bhi bacha rehta hai: paraphrase, synonym aur Hinglish sawaal — fusion dono worlds deta hai.
Jahan dono retrievers ek jaise fail hon (bahut ambiguous query), hybrid bhi nahi bachayega — wahan query transformation chahiye (M4.11).

Decision rule simple hai: pehle ek chhota eval set banao (dense-only, sparse-only, hybrid), teeno chalao, compare karo. Zyada tar real corpora mein hybrid top pe aata hai — par ye assume karke nahi, apne numbers se prove karke.

Shuruaat kahan se: Pehle baseline dense chalao, phir hybrid add karo, phir reranking (M4.10). Har step pe apna eval set — isi rhythm se poora L4 aage badhta hai.

Exercise

Question: BM25 kis type ka retrieval hai — dense ya sparse?

Question: Hybrid results ko sirf ranks se combine karne wali technique ka naam?

Question: Rare word ko zyada weight dene wala BM25 factor kya hai? (do akshar)

Question: Query mein exact error code aaye toh kaunsa retriever strong chalta hai — dense ya BM25?

Common mistakes

Har query pe sirf vector search: IDs, codes aur numbers wali queries pe dense chhod deta hai. Fix: exact-match zone wali queries identify karke unpe sparse chalao, ya hamesha hybrid.
Dense aur BM25 scores ka seedha sum: Scales alag (0.9 + 14.2) — bade-score wala retriever hamesha jeetega. Fix: rank-based fusion (RRF) use karo ya scores normalize karo.
BM25 ko outdated samajhna: Naye models aaye iska matlab nahi wo bekaar ho gaya. Fix: apne eval set pe dense vs sparse vs hybrid compare karo — numbers decide karenge.

Desi example: Telephone directory vs dost ki salah (ab dono)

Pehle lesson mein humne do tareeke dekhe the — ab dono ko ek saath lagaate hain. Ek banda naam-bilkul-wahi dhoondhta hai (sparse), doosra matlab samajh ke (dense); fusion committee dono ki list lekar final list banati hai:

sparse list : [INV-4821 policy, error log E503]
dense  list : [refund timeline, reimbursement doc]
                 |
              RRF: 1/(k+rank) jod do
                 |
fused top-3 : [INV-4821 policy, refund timeline, error log]

Interview angle — yahan se sawaal aate hain

Q: BM25 ke teen factors? — IDF (rare term weight), TF saturation (term frequency cap), length normalization.
Q: Sparse vs dense ka core farq? — Sparse exact terms pe match, dense meaning ke coordinates pe; dono ke alag failure zones.
Q: RRF kaise kaam karta hai? — Har list ke rank se 1/(k+rank) score, phir sum — scales ki fikr nahi, sirf rank.
Q: Kab hybrid chuno? — Corpus mein exact IDs/numbers/phrases hain ya paraphrase dono — apne eval set se verify karo.

Lesson complete?

Sparse aur hybrid clear — ab tum sirf vector pe dependent nahi ho. Agla module capstone hai: in sab ko jod kar ek Hybrid Search Engine khud banoge.