Lesson 52 — Intermediate

Vector Databases & ANN Search

Chhote corpus pe NumPy wala search chal jaata hai — par laakhon vectors aate hi brute force dheere padne lagta hai. Is lesson mein hum vectors ko database mein store karna, metadata filters ke saath top-k, ANN kyun zaroori hai, HNSW/IVF ke trade-offs, aur index ki daily operations dekhenge.

Vector database kya hai

Schema simple hai: id, text chunk, embedding, aur metadata columns. Ek query aati hai → embed hoti hai → distance ke hisaab se order hoti hai → top-k wapas. Jo tumne scratch search mein likha tha, wahi ab index aur optimized code karega.

Store karna aur metadata filters ke saath search

CREATE TABLE chunks (
  id        bigserial PRIMARY KEY,
  doc_id    text NOT NULL,
  heading   text,
  body      text NOT NULL,
  embedding vector(768)
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- metadata filter + top-kSELECT id, doc_id, heading, body
FROM chunks
WHERE doc_id = $1
ORDER BY embedding <=> $2
LIMIT 5;

• <=> pgvector ka cosine distance operator — chhota score = zyada similar; dot aur L2 operators bhi hain metric ke hisaab se.
• Metadata filter pehle ya baad mein? Chhote filtered sets pe filter pehle (exact), bade pe ANN ke candidates ke baad (post-filter) — dono ke trade-offs hain.
• Dedicated stores (Qdrant waghera) wahi kaam collections, payload metadata aur distance config ke roop mein dete hain.
• Schema migration: dimension ya metric badli toh table rebuild — purane vectors naye index mein nahi chalte.

Brute force kyun nahi chalta — ANN

import numpy as np

n, d = 1_000_000, 768
ops = n * d                      # har query pe: 10^8 multiply-adds
print(f"{ops:,} ops per query")
# bade corpus + har request = ANN ki zarurat
# ANN: thoda recall chhod ke tez (approximate nearest neighbor)

Brute force = exact: har vector compare, best guarantee. Par iska kharcha corpus ke saath linear badhta hai. ANN (Approximate Nearest Neighbor) index pehle se "kareeb wale" dhoondh leta hai — tez, par kabhi kabhi ek do asli neighbours chhoot sakte hain.

HNSW aur IVF ke trade-offs

EXACT  : sab vectors compare     -> 100% recall, corpus ke saath dheere
HNSW   : layered graph walk      -> tez, RAM zyada, recall thoda chhota
IVF    : nearest clusters mein   -> tez, clustering quality matters

HNSW knobs : M (graph connectivity)  ef_search (query time depth)
IVF knobs  : nlist (clusters)        nprobe (kitne clusters scan)

• HNSW: small-world graph — chhoti layer se coarse search, phir neeche precise. Bade corpus pe best recall/latency, par index RAM leta hai.
• IVF: vectors ko clusters (nlist) mein baanta, query sirf nearest clusters (nprobe) mein dekhti hai. Nprobe badhao = tez nahi, par recall zyada.
• Trade-off teen taraf: speed ↔ memory ↔ recall. ek ko behtar karte ho toh doosre pe asar padta hai.
• Recall ka matlab: top-k jo exact search deta, usme ANN ke results kitne aaye — ye number tumhara hai, koi universal nahi.

Indexing, updates aur deletes

• Inserts: incremental adds chalte hain, par index periodically rebuild/compact karna parta hai — purana data hataate rehna warna waste.
• Updates: embedding badla toh vector replace; text badla toh re-embed + upsert (M4.1 ka fingerprint yahan kaam aata hai).
• Deletes: deleted doc ke vectors hatao — warna purana content abhi bhi top-k mein aayega (common production bug).
• Filters + ANN: pre-filter exact par slow, post-filter fast par kabhi kam results. Filter selective ho toh pre, generic ho toh candidates badhao.
• Operations: index size, build time, query latency aur recall — in 4 cheezon pe nazar rakho; metric/model change = poora reindex.

Exercise

Common mistakes

• Chhote corpus pe hi ANN laga dena: Kuch hazar vectors pe brute force hi fast hai; ANN ka build/RAM overhead bekaar. Fix: pehle exact chalao, latency dekho, tabhi index.
• Distance metric change karna bina reindex ke: Cosine se dot pe switch, par purane index pe dot — ranking chhitam. Fix: metric = schema change, poora rebuild.
• Deletes ko ignore karna: Doc hata diya par vector pada raha — purana content abhi bhi jawab mein aata hai. Fix: purge job + periodic compact.
• Recall bina tune kiye defaults pe chalna: ef_search/nprobe default pe silent quality drop ho sakti hai. Fix: ground-truth sample se recall curve banao, budget ke hisaab se choose karo.

brute force: 1,000,000 dukaan ek-ek puchho     -> exact, slow
HNSW naksha : graph pe walk  (ef_search)       -> tez, RAM
IVF mohalle : nearest blocks (nprobe)          -> tez, clustering

tumhara rule: chhoti mandi -> seedha; badi -> naksha + recall test