Lesson 50 — Intermediate

Embeddings Fundamentals

Embedding text ko numbers mein badalta hai jahan meaning paas wale numbers ke roop mein banta hai — isi se "bolo jo matlab hai, dhoond wahi milega" wala search mumkin hota hai. Is lesson mein hum embedding kya represent karti hai, kaise generate hoti hai, model kaise chunta hai, aur do models ke vectors kyun nahi milte.

⏱ 20 min• Intermediate• Chunking Strategies

Embedding kya hai

Kya hai?

Embedding text (ya image, audio) ka fixed-length numeric representation — ek list of floats. Iske paas coordinates hain: meaning ka jo hissa similar hai uske points vector space mein paas hote hain.

Kab kaam aata hai?

Semantic search, clustering, recommendation, RAG retrieval — jab bhi "same matlab" dhoondhna ho. Har document chunk aur har user query dono ko embed karna padta hai.

Kahan milta hai?

Embedding models mein (local Python library ya hosted API), aur vector databases mein stored vectors ke roop mein. Index ka har row isi vector par khada hota hai.

Ek embedding mein har number ek dimension hai — jaise x, y, z, par hundreds mein. Model ye coordinates kaise decide karta hai? Training ke dauran similar text ke paas aur alag text ke door khada karne ki practice se. Isliye embedding "similarity" ka kaam karta hai, exact match ka nahi.

Yaad rakho: Embedding mein meaning ka pattern hota hai, spelling ya exact words nahi. "Return policy" aur "refund rules" paas baithe honge; "INV-4821" jaise exact IDs ke saath density search kamzor padta hai (agle lessons mein, hybrid se solve hoga).

Embeddings kaise generate karte hain

import os
from sentence_transformers import SentenceTransformer   # example library

model = SentenceTransformer(os.environ["EMBED_MODEL"])

docs = [c["text"] for c in chunks]
vecs = model.encode(docs, batch_size=64, normalize_embeddings=True)
# har vector apne chunk record ke saath store hua: id + vec + metadata

qvec = model.encode("refund policy kaise hoti hai",
                    normalize_embeddings=True)
# query ko bhi usi model se hi embed karna hai

Batch mein encode karo — ek-ek call karne se ingestion 10x dheere chalti hai.
normalize: vector ko unit length pe laao — phir dot product = cosine similarity, search tez ho jaata hai.
Query aur doc dono usi model se — alag-alag model ke vectors ek doosre se compare hi nahi hote (niche detail).
Har vector ke saath apna id, text, metadata store karo — vector kabhi akela nahi chalta.

API vs local: Hosted embedding API se zero infra, par har text pe per-call cost aur data bahar jaata hai. Local model se data aapke paas, par GPU/RAM kharcha. Chhote corpus aur budget dono ke liye local models kaafi strong chalte hain.

Embedding model kaise choose karein

Dimensions: chhoti dimension = kam storage aur tez search, par bahut fine-grained meaning khone ka darr. Badi ulta. Apne corpus pe thoda test hi batayega.
Language: agar queries Hinglish/English mix hain toh multilingual model chuno — English-only model Hinglish sawaal ko theek se place nahi karega.
Domain: legal, medical, code — domain text pe trained/fine-tuned model general model se aksar behtar chalta hai.
Cost & latency: API per-text billing vs local GPU. Ingestion ek baar chalti hai, query har request pe — query path ka latency zyada matter karta hai.
Instruction-tuned models: kuch models query aur document ke liye alag prefixes lete hain — model card mein likha hota hai, wahi follow karo.

Sabse bada practical rule: apna chhota eval banao — 15-20 apne sawaal, unke sahi docs ke saath, aur 2-3 candidate models ko uspe compare karo. Public leaderboard numbers tumhare domain aur language pe translate nahi hote.

Model card padho: Dimensions, max input length, language support aur recommended usage — sab model card mein likha hota hai. Wo stale benchmark numbers se zyada trustworthy hai jo directly tumhare kaam ke hain.

Vector spaces mix nahi hote

# GALAT: do alag models ke vectors ek jagah
v1 = model_a.encode("refund policy")
v2 = model_b.encode("refund policy")
# cosine(v1, v2) ka matlab nahi — dono alag coordinate systems

# SAHI: model badla toh poora index re-embed karo
if loaded_model_name != current_model_name:
    reembed_all()   # purana index hatao, naye model se dobara banao

Har model ka apna coordinate system hai — uske numbers ki koi cross-model meaning nahi.
Model upgrade = migration: poora corpus dobara embed karo, tabhi naya index lagao.
Ek hi index mein do models kabhi mix mat karo — ek query kab model A se, kab B se embed hogi, ranking chhitam.
Embedding version ko apne manifest mein likho — kaunsa model, kab, kis dimension ke saath index bana.

Interview favourite: Q: "Model upgrade kaise karoge?" A: Dimension alag hui toh table schema bhi; phir batch re-embed + atomic swap (naya index banao, test karo, phir switch). Downtime isi tareeke se bachta hai.

Exercise

Question: Embedding vectors ko unit length pe laane ko kya kehte hain, jisse dot product cosine ban jaata hai?

Question: Query aur document ko embed karne ke liye kis cheez same honi chahiye? (ek word)

Question: Embedding model badal diya — ab kya karna padega purane vectors ke saath?

Question: Embedding mein numbers ki total count ko kya kehte hain?

Common mistakes

Do models ke vectors mix karna: Ek index mein alag-alag models ke vectors daal diye toh similarity ka matlab hi khatam. Fix: har index = ek model + ek version, model change toh full re-embed.
Dimension ko quality samajhna: Badi dimension = behtar search? Zaroori nahi — kabhi kabhi chhoti dimension hi behtar generalization deti hai. Fix: apne eval set pe compare karo, size ko proxy mat manno.
Query ko alag tareeke se embed karna: Doc normalized, query nahi — dot product rankings bigad jaati hain. Fix: dono ko same pipeline, same normalization, same model se guzaro.
Bina model card dekhe model chun lena: Language support aur max input length pata nahi, ingestion mein hi surprise milta hai. Fix: model card pehle padho, phir chhote test pe daalo.

Desi example: Classroom ka seating chart

Classroom mein seating arrangement ek embedding hai — jo log aapas mein baat karte hain wo paas baithe hain, section A aur section B door. Do sections ke alag charts mix kar doge toh seat numbers ka koi matlab nahi rahega:

section A chart:  Ravi__ Priya   Amit       <- meaning ke hisaab se paas
section B chart:  Zoya__ Imran  Sara

A + B charts mix -> "Ravi seat 3" ab kis class ka?  ERROR
wahi do models ke vectors = do alag charts, kabhi mix nahi

Interview angle — yahan se sawaal aate hain

Q: Embedding kya represent karta hai? — Fixed-length numeric vector jahan similar meaning paas, alag meaning door (vector space).
Q: Cosine aur dot mein normalization ka kya rishta hai? — Unit vectors pe dono same ranking dete hain; isliye encode karte hi normalize karte hain.
Q: Model kaise choose karoge? — Language, domain, dimensions, cost/latency, phir apne 15-20 sawaal ke eval pe compare.
Q: Model upgrade hua toh index? — Full re-embed + atomic swap; kabhi bhi do models ka data ek index mein nahi.

Lesson complete?

Embeddings clear — vector, normalization aur model selection ka funda samajh aa gaya. Agla lesson — Similarity & Semantic Search — mein dekhenge in vectors ko compare karke search khud kaise banate hain.