Lesson 51 — Intermediate

Similarity & Semantic Search

Vector database se pehle samajh lo ki andar kya chalta hai: query embed karo, sab vectors se score nikaalo, sort karo, top-k lo. Is lesson mein hum cosine/dot/Euclidean ka farq, scratch se semantic search, keyword search ka difference, aur dense search ke failure cases dekhenge.

⏱ 20 min• Intermediate• Embeddings Fundamentals

Cosine, dot product aur Euclidean

Kya hai?

Similarity metrics do vectors ko compare karke ek number dete hain. Cosine angle dekhta hai (direction), dot product magnitude ke saath mul karta hai, Euclidean seedhi doori — chhoti doori = paas.

Kab kaam aata hai?

Har retrieval request pe — query vector aur har document vector ke beech score banate hi top results nikalte hain. Reranking aur clustering mein bhi yehi metrics kaam aate hain.

Kahan milta hai?

Vector DB ke ORDER BY distance clause mein, hand-written search scripts mein, aur similarity calculators mein. Default aksar cosine ya dot hota hai.

import numpy as np

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

def dot(a, b):
    return float(a @ b)

def l2(a, b):
    return float(np.linalg.norm(a - b))   # chhoti = paas

Normalize ho toh cosine = dot: Embeddings normalize kar diye (unit length) toh cosine similarity aur dot product same ranking dete hain — isi liye production search dot product karta hai, wo zyada sasta computation hai. Euclidean tab natural jab geometry matter karti hai.

Scratch se semantic search

import numpy as np

class MiniSearch:
    def __init__(self, texts, embs):
        self.texts = texts
        self.mat = np.array(embs, dtype="float32")
        self.mat /= np.linalg.norm(self.mat, axis=1, keepdims=True)

    def search(self, query_vec, k=5):
        q = query_vec / np.linalg.norm(query_vec)
        scores = self.mat @ q                 # cosine = dot (normalized)
        top = np.argsort(-scores)[:k]
        return [(self.texts[i], float(scores[i])) for i in top]

# flow: query -> embed -> dot with all -> sort -> top-k

Matrix ka har row ek document vector, har column ek dimension — ek matrix multiply se saare scores ek saath.
argsort(-scores)[:k] : descending sort karke pehle k — yahi vector DB ka top-k hai, bas wo ye brute force optimized form mein karti hai.
Ye poora loop hi vector database ka kaam hai — chhote corpus pe tum khud chala sakte ho, bade pe ANN index chahiye (agla lesson).
Har result ke saath score aur doc id wapas aao — score threshold lagane ke liye chahiye.

Threshold ka fayda: Top-k hamesha k results deta hai — khaali corpus mein bhi kuch na kuch deta hai. Score threshold se bol sakte ho "itna weak match chahiye nahi" — abstention isi se banti hai.

Semantic similarity vs keyword match

Query: "mera paisa wapas kab milega"

Keyword/BM25:  "paisa wapas" exact words dhoondhta hai
               -> refund/return wale docs miss ho sakte hain

Semantic:      meaning ki taraf jaata hai
               -> "refund timeline", "reimbursement policy" pakadta hai

Exact ID:      "INV-4821"
               -> semantic: INV-9917 bhi lagbhag same lagta hai
               -> keyword: sirf INV-4821 wala doc milega

Keyword: exact terms, IDs, codes, numbers pe pakka. Synonym chhod deta hai — "auto" aur "car" alag.
Semantic: paraphrase, synonym, Hinglish phrasing. Exact strings ke saath dhundla padta hai.
Dono ke apne alag failure zones hain — isi liye production retrieval aksar hybrid hota hai (M4.6).
Tumhari awaaz mein deciding test: 5 exact-ID queries aur 5 paraphrase queries likho, dono tareekon se chalao, results compare karo.

Decision rule: Sawaal exact identifiers ya technical codes maangta hai → keyword/BM25. Sawaal bhavna, synonym ya Hinglish mein hai → dense. Confusion ho toh dono chalao aur fuse karo — aage ke lesson mein RRF se.

Semantic search kahan fail hota hai

q = "INV-4821 refund status"
# dense model: INV-4821 aur INV-9917 ko lagbhag same samajhta hai
# exact keyword: "INV-4821" wahi doc pakadta hai jo chahiye

for query in ["INV-4821", "error code E_CONN_503", "SHA-256 kaise"]:
    print(query, "-> dense ke liye numbers/ids ek jaise hain")

Exact names aur IDs: invoice, order, ticket numbers — dense inhe differentiate nahi kar paata.
Numbers aur rare codes: versions, error codes, chemical formulas, sha codes — tokenization bhi inhe todta hai.
Bahut rare terminology: jo term model ne training mein kam dekhi, uska place vector space mein bekaar ho sakta hai.
Short queries: 1-2 shabd ki query embedding thodi noisy hoti hai — rewriting se sudhar sakta hai (M4.11).

Fail = feature nahi, fix: Ye sab failure modes waqt pehle hi expect karo. Exact-ID wale queries ke liye hybrid retrieval rakho, aur apne eval set mein inhi failure queries ko zaroor shaamil karo — warna production mein yahi se aag lagti hai.

Exercise

Question: Normalized vectors pe cosine similarity kaunsa doosra metric ban jaata hai?

Question: Semantic search ka basic flow — query ko pehle kya banate hain?

Question: Exact invoice ID wale query pe kaunsa retrieval reliable — dense ya keyword?

Question: Euclidean (L2) distance chhoti ho toh do vectors kitne similar hain? (ek shabd)

Common mistakes

Cosine score ko percentage samajhna: 0.8 ka matlab 80% match nahi — ye relative score hai. Fix: threshold ko apne corpus ke score distribution se set karo, absolute number pe bharosa mat rakho.
Normalization chhod ke dot product: Bade vectors ka dot product unhe hamesha top pe laata hai, meaning se nahi. Fix: encode karte hi normalize, ya cosine use karo.
Har query pe sirf dense search: IDs, codes aur numbers wali queries pe dense chhod deta hai. Fix: exact-match queries identify karo aur unpe keyword/BM25 chalao — ya hybrid se dono.

Desi example: Telephone directory vs dost ki salah

Directory mein exact number dhoondhna = keyword search — naam bilkul wahi chahiye. Dost se poochhna "wo movie jisme train wala scene tha" = semantic search — matlab se dhoondhta hai. Dono kaam ke, par phone bill ka hisaab sirf number se hi niklega:

query "refund kab milega"
  keyword -> exact "refund" wale docs     (ID/code pe pakka)
  semantic-> meaning wale docs            (synonym/Hinglish pe pakka)
  hybrid  -> dono rankings fuse           (production default)

Interview angle — yahan se sawaal aate hain

Q: Cosine vs dot product kab? — Normalized embeddings pe same ranking; cosine direction-focused, dot magnitude bhi leta hai.
Q: Semantic search ka pipeline? — Query → embedding → similarity scores → sort → top-k, yehi vector DB automate karti hai.
Q: Dense search ki 3 klasik failures? — Exact IDs, numbers/error codes, aur rare terminology.
Q: 0.9 cosine score ka kya matlab hai? — Akele mein kuch nahi; relative hai, threshold apne corpus ke distribution se chuniye.

Lesson complete?

Metrics aur scratch search clear — ab tum bina vector DB ke bhi search chala sakte ho. Agla lesson — Vector Databases & ANN Search — mein dekhenge laakhon vectors ko tez tareeke se kaise query karte hain.