Lesson 24 — Beginner

Classical NLP: Cleaning, Tokenization & TF-IDF

Model ko text nahi, numbers chahiye — aur wo number banane ka purana, sasta, explainable raasta hai classical pipeline: raw text → clean tokens → numbers, jahan TF-IDF ka poora khel hai "is corpus ke liye important shabd kaunse". Is lesson mein tum cleaning ke trade-offs, word/sentence/subword tokenizers, stemming vs lemmatization, ek 5-document corpus ka TF-IDF haath se, aur pipeline tooling (NLTK/spaCy style) dekhoge — saath mein wo signal jo batata hai kab embeddings ki zarurat pad gayi.

⏱ 20 min• Beginner• Deep Learning Foundations: CNN, RNN & LSTM

Text pipeline — code ko numbers chahiye

Kya hai?

Classical NLP ka pipeline: raw text ko clean karo, tokens mein todo, normalize karo (stem/lemma), phir numbers mein badlo (TF-IDF) taaki machine use sake. Har step kuch na kuch tabah karta hai — isliye har step ka trade-off samajhna zaroori hai.

Kab kaam aata hai?

Spam filter, keyword search, dedupe, topic counts — jahan speed, explainability aur kam data matter karta hai. Aur jab budget mein embeddings/LLM fit na hon.

Kahan milta hai?

NLTK aur spaCy pipelines mein, scikit-learn ke TfidfVectorizer mein, aur production ke chhote services mein jo milliseconds mein jawab dete hain.

Chain hamesha wahi rahegi: raw text → cleaning → tokenization → normalization → vectorization (numbers) → model. Har arrow pe tum kuch decide karte ho — kya hataana hai, kya rakhna hai. Aur TF-IDF ka ek-line meaning: jo shabd is corpus mein kam milta hai (rare) aur is document mein maujood hai, wahi asli signal hai.

Order matter karta hai: Cleaning pehle, tokenization uske baad, normalization phir, vectorization aakhir mein. Aksar log cleaning vectorizer ke andar chhupa dete hain aur baad mein "output mein tags kyun aa rahe hain" mein ghonte hain — pipeline stages alag-alag rakho, debug aasaan.

Cleaning — har step kuch na kuch tabah karta hai

import re, html

raw = "<p>Great! FREE Shipping... order NOW</p>"

def clean(s):
    s = html.unescape(s)                 # entities wapas lao
    s = re.sub("<[^>]+>", " ", s)         # <p> jaise tags hatao
    s = s.lower()                         # case folding
    s = re.sub("[^a-z0-9 ]", "", s)       # punctuation hatao
    s = " ".join(s.split())               # extra spaces normalize
    return s

print(clean(raw))   # great free shipping order now

Case folding: "Free" aur "free" ek ho jaate hain. Par case ka signal bhi tha (PROPER nouns, ALL CAPS spam cues) — wo chala jaata hai. Decide karo task ke against.
Punctuation: hashtags, numbers, contraction wale marks hat jaate hain — structured IDs (order#4521) bhi toot sakte hain. Har punctuation ko kachra mat samjho.
HTML aur noise: tags, URLs, emoji, menu text — search/spam tasks mein aam hataaye jaate hain. Par emoji sentiment ka hissa hon toh mat phenko.
Stopwords: the/a/is jaise aam shabd hataate ho toh matrix chhoti hoti hai — par negation ("not good") ka "not" hi chala gaya toh meaning ulta ho sakti hai.
Normalization side-effects: numbers ko category (NUM), dates normalize, slang map — har step ke baad sample padho ki meaning zinda hai ya nahi.

Har cleaning step ek trade hai: Lowercase se proper nouns ka case signal gaya; stopwords se negation chhoot sakti hai; punctuation se IDs toot sakti hain. Isliye cleaning step ko default mat manno — downstream task pe sample le kar verify karo ki kya ukhda nahi.

Tokenization — word, sentence, subword

sentence: "Order #4521 shipped, arriving Tuesday."

word tokens (nltk style): [Order, #4521, shipped, ,, arriving, Tuesday, .]
subword style (LLM-ish):  [Order, #, 45, 21, shipped, ,, arriving, Tuesday, .]
                            (approx idea — asli splits tokenizer pe depend karte hain)

token types (spaCy style): Order/NOUN  #4521/NUM  shipped/VERB  Tuesday/DATE

sentence tokenizer: ek paragraph -> alag-alag sentences (clause-level tasks)

Word tokenizer: spaces aur punctuation pe todo. Sabse simple, par "unhappiness" ek hi token — morphology chhup jaata hai.
Sentence tokenizer: clause/summary level tasks ke liye boundaries. Ek galat dot (2026.5) boundary tod sakte hain — rule-based fix zaroori.
Subword tokenizer: chhota vocab rakh kar rare shabd ko tukdon mein toda (un|happi|ness). OOV problem khatam, numbers/misspellings handle, par tokens = words nahi hote.
LLM tokenizer ≠ NLTK word list: LLM wale BPE-style hain (chars → subwords), isliye token count, cost aur context windows alag hisaab se chalte hain.
Token types: spaCy jaisi pipelines NOUN/VERB/DATE wale tags deti hain — ye agle lesson (NER, POS) ke liye features hain.

Token ka hisaab alag hota hai: "Kitne tokens" wala budget word count se nahi chalta — tokenizer kaunsa hai wo config mein dekho. Do models ko token counts pe compare karna jaise feet aur metre se ek hi naapna.

Stemming vs lemmatization — normalization ke do tareeke

word      stemming (rule)      lemmatization (dictionary + POS)
--------  -------------------  --------------------------------
studies    studi                study
running    run                  run
flies      fli  (breaks!)       fly
boxes      box                  box
better     better (todta nahi)  good (irregular, adjective POS ke saath)

stemming: chhote rules, tez, blind — spam/search mein aksar kaafi
lemma: POS-aware lookup, behtar shabd — par dheere aur dictionary-bound

Stemming kaise kaam karta hai: suffix katne ke rules (-ing, -ies, -ed). Grammar nahi dekhta — isi liye tez hai par kabhi-kabhi shabd tod deta hai.
Lemmatization kaise: dictionary + POS tag se base form nikaalta hai. Output insaan ko sahi lagta hai, par jo shabd dictionary mein nahi (domain slang, brand) waisa ka waisa reh jaata hai.
Kab stemming kaafi: keyword recall wale kaam (spam, search) jahan "studi" se kaam chal jaata hai aur speed matter karti hai.
Kab lemma chahiye: feature quality matter karti ho, output insaan ko dikhna ho, ya downstream dictionary lookup ho.
Dono ka breaking point: OOV words, proper nouns, aur morphology jo rules/dictionary dono ke bahar ho — normalize hoke bhi alag reh jaate hain.

Normalized vs original — dono rakho: Features ke liye normalized form chahiye (ship/shipped/shipping ek column), par analysis aur display ke liye original text. Normalized shabd ko dictionary/search mein dhoondhoge toh nahi milega — isliye mapping wapas jodne wala step pipeline mein rakho.

TF-IDF — haath se gin ke dekho (5 documents)

from math import log

docs = ["free shipping", "free money", "shipping offer",
        "money back", "free shipping offer"]
N  = 5
df = {"free": 3, "shipping": 3, "money": 2, "offer": 2, "back": 1}

idf = {w: round(log(N / d), 2) for w, d in df.items()}
# free 0.51, shipping 0.51, money 0.92, offer 0.92, back 1.61 (ln(5/df))

def tfidf(doc):
    toks = doc.split()
    return {w: round(toks.count(w) * idf[w], 2) for w in set(toks)}

print(tfidf(docs[0]))   # {free: 0.51, shipping: 0.51}
print(tfidf(docs[4]))   # {free: 0.51, shipping: 0.51, offer: 0.92}
# matrix ki shape: 5 docs x 5 terms, zyada cells ZERO (sparse)

TF = term frequency (is document mein kitni baar) aur IDF = log(N/df) (corpus mein kitna rare — rare to unchi). Dono ka product = us shabd ka score. "back" sabse unchi idf wala shabd hai (sirf 1/5 docs) par agar wo tumhare doc mein hai hi nahi toh score 0 — isi liye TF-IDF ka poora khel "rare + present-here" hai.

from sklearn.feature_extraction.text import TfidfVectorizer

v = TfidfVectorizer()
X = v.fit_transform(docs)         # 5 x 5 sparse matrix
print(X.shape, X.toarray()[0])
print(v.get_feature_names_out()) # ["back", "free", ...]

# dhyan: sklearn smooth_idf aur L2 normalization lagata hai — exact values
# thodi alag hongi; formula yehi hai, ranking wahi rahegi

Sparse matrix ki shakal: Docs × vocab — aur 99% cells zero (har doc mein chhante hi shabd hain). Isliye ye CSR jaisi sparse formats mein store hoti hai, dense mein nahi. Vectorizer ka output dekho: shape, non-zero count, aur top terms — yahi tumhara model ka input hai.

Pipeline tooling — aur kab TF-IDF hi sahi hai

spacy / nltk style pipeline:
  raw text -> tokenizer -> cleaner -> normalizer -> vectorizer -> model

blank vs pretrained:
  blank  : sirf tokenizer + apne rules  -> control zyada, knowledge nahi
  pretrained: POS/NER/lemma attach     -> turant features, par domain fit check

NLTK  : padhai ke liye saaf modules, har step dikhta hai (learning)
spaCy : production speed, pipeline objects, rule + model ka mix
sklearn: vectorizers + chhote models, TF-IDF ka ghar

TF-IDF sahi hai jab: fast search/dedupe, spam/keyword routing, explainable keyword matching, aur data kam hai — minutes mein chal jaata hai, GPU nahi chahiye.
Embeddings ki zarurat ke signals: user ek hi cheez ko alag-alag shabdon mein likhta ho (laptop vs notebook), paraphrase miss ho rahe hon, multilingual ho, ya TF-IDF ke misses aksar synonym mein hon.
Aksar hybrid: TF-IDF se pehle shortlist (sasta, tez), phir embedding/LLM se re-rank (semantics). Dono ek dusre ko replace nahi, complete karte hain.
Official docs zaroor kholo: sklearn TfidfVectorizer, spaCy pipelines, NLTK tokenizers — do line padhne se kaafi assumptions saaf ho jaati hain (docs par official links hain).

Decision rule: Keyword match hi kaam kar raha hai toh TF-IDF raho — chhota, tez, auditable. Error sample padho: agar misses "meaning ke farq" mein hain (synonym/paraphrase) tab embeddings; agar logic/labels mein hain toh embedding se bhi nahi hoga — model/labels sudharo.

Exercise

Question: TF-IDF mein IDF value kis tarah ke shabd ke liye zyada hoti hai — common ya rare?

Question: Hamare 5-document corpus mein "free" ka document frequency (df) kitna hai? Sirf number daalo.

Question: Rule-based truncation (studies -> studi) kya kehlata hai — stemming ya lemmatization?

Question: Stopwords ki category ka ek aam English example ek shabd mein likho.

Question: TF-IDF matrix ki rows kya hote hain — documents ya terms?

Common mistakes

Blind cleaning — meaning hi tabah: Bina soche lowercase, stopwords aur punctuation hata diye — proper nouns, negation aur IDs chala gaye. Fix: har step ka asar downstream task pe sample le kar dekho, default nahi.
LLM tokenizer ko word tokenizer samajhna: Tokens ≠ words; token count, cost aur context budget galat ho jaate hain. Fix: tokenizer kaunsa hai wo config/pipeline mein dekho, NLTK wale list se compare mat karo.
Stemmed shabd ko display/search mein dhoondhna: "studi" ko insaan nahi samjhega aur dictionary lookup bhi fail. Fix: normalized sirf features mein; output aur search ke liye original text wapas jodo.
TF-IDF ko semantic search samajhna: Synonym/paraphrase miss ho jaate hain — "laptop" query, "notebook" doc nahi milta. Fix: keyword wala kaam ho toh TF-IDF; meaning wala ho toh embeddings (ya hybrid shortlist + re-rank).

Desi example: Exam se pehle important questions chunna

Teacher ne kaha — jo sawaal tumhare notes mein baar baar aate hain wo common hain, par jo sawaal sirf ek hi chapter mein milta hai wo special hai. TF-IDF wahi karta hai: term frequency (mere notes mein kitni baar) guna inverse document frequency (poore corpus mein kitna rare). Aam filler ko neeche kheenchta hai, rare wale ko upar:

notes = "free free shipping offer"    # tf: free = 2
idf = {"free": log(5/3), "back": log(5/1)}   # 0.51 vs 1.61

score_free = 2 * 0.51          # 1.02
score_back = 1 * 1.61          # 1.61  -> kam shabd, zyada wazan
# ranking isi wazan se hoti hai, shabd ki lambai se nahi

Interview angle — yahan se sawaal aate hain

Q: TF-IDF kaise calculate hota hai? — TF × IDF; IDF = log(N/df). Rare aur present-ha wale shabd ko sabse zyada wazan milta hai — common filler ko nahi.
Q: Stemming aur lemmatization ka farq? — Rule-based suffix katna (tez, blind) vs dictionary + POS se base form (dheere, sahi). Kab kaunsa — task ki quality/speed demand se.
Q: "the" jaise stopwords kyun hataate hain aur kab nahi? — Aam shabd se matrix badi; par negation/phrase search mein nuksaan. Task pe decide, default nahi.
Q: TF-IDF aur embeddings kab chunoge? — Keyword/explain/speed/data-kam → TF-IDF; synonyms/semantics/multilingual → embeddings; aksar hybrid.
Q: Subword tokenization kyun? — OOV shabd, chhota vocab, numbers/misspellings handle — isi liye LLM tokenizers subword pe chalte hain (words nahi).

Lesson complete?

Pipeline saaf — ab tum raw file ko clean tokens aur sparse matrix mein badal sakte ho aur bata sakte ho ki us matrix mein kya likha hai. Agla lesson — Classic NLP Tasks — mein isi matrix ko spam, sentiment aur NER par lagayenge.