Lesson 25 — Beginner

Classic NLP Tasks: Classification, NER & Sentiment

Spam filter, ticket routing, sentiment, entity extraction — ye sab wo kaam hain jo LLM se pehle se chal rahe hain aur aaj bhi saste, tez aur auditable rehte hain. Is lesson mein tum ek poora mental model chaloge — Task → labels → features → model → error analysis — jahan ek LLM bas ek (mehnga) row hai features column mein. Har task ke saath uska failure mode bhi milega: threshold traps, sarcasm, domain shift, aur wo decision table jo decide karti hai kahan chhota model aur kahan LLM.

⏱ 20 min• Beginner• Classical NLP: Cleaning, Tokenization & TF-IDF

Classic tasks — saste, tez, auditable

Kya hai?

Transformer-era se pehle se chale aa rahe NLP tasks: text classification (spam/topic routing), sentiment analysis, NER (naam/organisation/location pehchanna), POS tagging (shabd ka grammatical kaam), aur extractive summarization (source se hi lines chunna). Feature engineering + chhota model wala approach.

Kab kaam aata hai?

Bulk kaam ke liye — hazaar documents routing, har ticket ka sentiment, har invoice se entity nikaalna — jahan latency, cost aur audit trail matter karta hai, aur labels already maujood hain.

Kahan milta hai?

Spam filters, inbox routing, support triage, invoice/contract parsing, search results ke tags — aur hybrid systems jahan chhota model bulk karta hai, LLM sirf mushkil cases.

Poora loop ye hai: task → labels → features → model → error analysis. Model banana sabse aasaan hissa hai; asli kaam error analysis hai — har miss ka reason likhna (label galat? domain shift? threshold?). Aur LLM ka role isi loop mein hai: wo bas features column ka ek mehnga row hai — usse pehle chhota model try karo, jab wo ruke tab use karo.

Error analysis hi progress hai: Model 10 minute mein ban jaata hai, behtar miss-categories hafte bhar mein banti hain. Har run pe 10 misses uthao, unhe bucket karo (label/domain/threshold/format) — agli improvement isi list se aayegi, kisi leaderboard se nahi.

Text classification — labels, thresholds, multi-label

# classifier output: probability, jawab nahi
p_spam = 0.62
print(p_spam >= 0.50)   # True  -> default threshold
print(p_spam >= 0.80)   # False -> high-precision setting

# threshold ko error cost se chuno:
#   wrong block (FP) mehnga  -> threshold upar, precision upar
#   spam chhoot gaya (FN) mehnga -> threshold neeche, recall upar

# multi-class : ek hi row, ek category  (billing / tech / other)
# multi-label : ek hi row, kayi categories saath (finance + regulation)

Label design pehle: kam, crisp categories; ek hi cheez ka do naam nahi. 20 examples do labelers se karwao — disagreement dikhe toh guideline sudharo, model baad mein.
Threshold ek business knob hai: model probability deta hai, line insaan khinchta hai. Curve dekh kar line chuno — 0.5 koi sacred value nahi.
Multi-class vs multi-label: ek hi row mein ek hi category chahiye toh softmax-style competition; kayi categories saath chahiye toh har label ka independent score + threshold.
Routing use-case: support tickets ko department mein bhejna — yahan top-1 accuracy se zyada matter karta hai ki "unassigned" wali bucket bhi ho (model ko nahi pata toh reject karo).

Label set chhota rakho: 15 categories jo kisi ne spreadsheet mein sochi hain, usmein se aadhi overlap wali hongi. Pehle 5-7 alag-alag categories banao, aur "ye dono kyun hain?" ka jawab likho — label taxonomy hi model ki ceiling hai.

Sentiment — lexicon, model, aur asli traps

LEX = {"great": +2, "awesome": +2, "slow": -1, "down": -1}
def lex_score(text):
    return sum(LEX.get(w, 0) for w in text.lower().split())

print(lex_score("great product"))              # +2 -> positive (sahi)
print(lex_score("great, the server is down"))  # +1 -> "positive"?! galat!

# traps: sarcasm, negation ("not good" ko bhi positive maanega)
# aspect-level: "camera awesome, battery average" -> do verdict chahiye

Lexicon approach: positive/negative shabd ki list se score. Zero training data, fully transparent — par context nahi dekhta.
Trained model: labelled examples se context seekhta hai — sarcasm/negation kabhi-kabhi pakad leta hai, par data chahiye aur domain shift phir bhi tootta hai.
Sarcasm aur neutral traps: "great, the server is down" lexicon ko positive dikhega; neutral sentences (status updates) ko bhi force-positive/negative karna common bug hai.
Aspect-level sentiment: asli useful version — review ke alag aspects (camera, battery, delivery) ke alag verdicts. Ek hi sentence mein do sentiment ho sakte hain.
Scope define karo: sentiment kis cheez pe — product, delivery, brand? Scope ke bina label hi galat hoga, model kuch bhi seekhe.

Sarcasm ka sasta fix (jaadoo nahi): Sarcasm rokna mushkil hai — isliye production mein 3-class (pos/neg/neutral) rakho, confidence kam ho toh neutral do, aur aspect-level pe focus karo jahan "kis cheez ka" clear hota hai. Ek global "positive/negative" number har cheez pe mat lagao.

NER — BIO tags, domain shift, fallback

# "Ravi joined Infosys in Pune" -> BIO tagging
tokens = ["Ravi", "joined", "Infosys", "in", "Pune"]
tags   = ["B-PER", "O", "B-ORG",  "O",  "B-LOC"]
# B- = entity ki shuruaat (Beginning), I- = andar (Inside), O = bahar
# multi-word: "New Delhi" -> B-LOC, I-LOC  (do token, ek entity)

# fixed formats ke liye dependable fallback: regex / gazetteer
import re
print(re.findall("[0-9]{6}", "pincode 560001"))   # ["560001"] — har jagah chalega

BIO tagging: har token ko label — B-ORG shuru, I-ORG andar, O bahar. Model span kaise banata hai, yahi uska mechanics hai (aur evaluation bhi isi pe hoti hai).
Entity types: PERSON, ORG, LOC, DATE, MONEY — type list hi project ka contract hai. Naya type chahiye toh examples chahiye, sirf naam badalne se nahi hoga.
Domain shift = sabse aam failure: general model ne "HDFC Bank" ko ORG kabhi dekha, kabhi nahi; medical/legal jargon mein toh aur bhi. Training aur production ka distribution alag ho toh NER tootta hai.
Regex/gazetteer fallback: pincode, phone, order IDs, invoice numbers — fixed formats hamesha rules se pakdo. Sasta, deterministic, aur audit-friendly.
Error analysis ke buckets: miss (entity chhooti), span galat (aadha pakda), type galat (ORG→LOC), spurious (naam tha hi nahi). Har bucket ka fix alag hai.

10 sentences, haath se tag karo: Pipeline chalane se pehle 10-20 apne domain wale sentences khud PERSON/ORG/LOC se tag karo, phir pipeline output se side-by-side diff karo. Har miss ka reason likho — domain shift pehle yahin pakda jaata hai, dashboard mein nahi.

POS tagging aur extractive summarization

POS tagging (shabd ka grammatical kaam):
  "bank" -> NOUN (river bank) vs VERB (bank the money)
  the/DET  cat/NOUN  sat/VERB ... -> rules, gazetteers, chunking ke features
  legacy pipelines mein aaj bhi: pre-neural NER/parse isi pe bane the

extractive summarization (source SE hi lines chuno):
  doc -> sentences -> score (TF-IDF weight / TextRank style graph) -> top-k lines
  output: quotes hi quotes -> har line source mein milti hai

abstractive (LLM style): nayi line banati hai -> flow behtar, par faithfulness check chahiye

POS tags kyun hain: downstream rules ko pata chalta hai kaun shabd noun hai (chunking, gazetteer joins, disambiguation). Transformer era mein embeddings ne kaam aasaan kar diya, par legacy code aur rule systems mein aaj bhi milte hain.
Extractive mechanics: sentences ko score karo (unmein ke important words ka wazan, ya graph-centrality style selection), top-k lines uthao. Kuch naya "likha" nahi jaata — sirf chuna jaata hai.
Verifiability ka faida: extractive output ki har line source mein milti hai — audit mein defendable, hallucination ki gunjaish hi nahi.
Keywords extraction: top TF-IDF terms hi tumhara keyword tag hai — tagging, dedupe aur search suggestions mein isi se kaam chal jaata hai.
Abstractive ka trade: flow behtar, coverage behtar — par har line source se check karni padti hai. Jahan compliance important hai wahan extractive default rakho.

Quote karne wala output sasta padta hai: Production mein faithfulness check ek extra layer hai. Extractive summary ko seedha dikha do — har line source ke saath link. Abstractive chahiye toh judge/human se faithfulness verify karwao, bina nahi.

Classic vs LLM — decision table, hybrid setups

TASK                         CHHOTA MODEL          LLM
---------------------------- --------------------- ---------------------
bulk routing (bahut rows)     sasta + tez + audit   mehnga + dheere
hard 5% cases                 accuracy adhuri       judgement behtar
explainability                features + weights    mushkil defend
labels available              supervised se chalta  few-shot bhi
log/audit trail               feature + score       prompt + response

default flow (cascade):
  bulk -> chhota model -> confident? -> ship
                              nahi -> LLM / insaan -> judgement

Tin columns hamesha dekho: cost, latency, auditability. Har request pe LLM call tab tak justified nahi jab tak do teeno columns clear na hon.
Cascade mechanics: chhota model apna confidence de, neeche wali ghadi ko LLM bhejo. Bulk sasta, mushkil chehra expert ke paas — dono ke logs alag.
LLM ko feature banao: kabhi-kabhi LLM ka output hi ek feature hai chhote model ke liye (structured extract → classical classifier). Ye bhi hybrid hai.
Labels aa gaye toh raste badalte hain: shuru LLM/few-shot se, thode labels bante hi chhote model pe shift — cost aur latency dono girti hain.
Decision rule: throughput, audit aur cost priority ho toh classical pehle; samajh ambiguous ho aur volume chhota ho toh LLM. Confusion ho toh dono chala kar same sample pe error analysis karo.

LLM default nahi, exception: Har ticket, har mail, har row pe LLM call karna sabse mehnga sabse dheera raasta hai. Pehle chhota model banao, uske misses LLM ko do — kaunsa kaunsa miss hua, ye data hi agla deciding factor hai.

Exercise

Question: BIO tagging mein B ka full form kya hai? (ek shabd)

Question: Spam classifier ka threshold 0.50 se 0.25 kar diya — recall kya hota hai, badhta hai ya ghatta hai?

Question: Ek hi document mein do topics ek saath lag rahe hain — wo multi-class hai ya multi-label?

Question: "great, the server is down" wale sentence pe lexicon sentiment kis famous trap mein fas jaati hai? (ek shabd)

Question: Pincode, order ID jaise fixed formats pehchanne ka sabse dependable classical fallback kya hai — trained model ya regex?

Common mistakes

Label design skip karna: Anjaan labelers alag-alag karenge, model unhi inconsistency ko seekhega. Fix: guidelines likho, 20 examples pe agreement check karo, taxonomy chhoti rakho.
0.5 threshold blind rakhna: Model probability deta hai, line tum khichhte ho — default line par kaam nahi karta. Fix: FP/FN ki cost decide, phir precision-recall curve se threshold chuno.
General NER ko domain pe blind trust: Domain shift se entity types hi badal jaate hain — training wale patterns production mein nahi. Fix: haath se tag kiye 10-20 samples pe test, aur regex/gazetteer fallback rakho.
Har document pe full LLM call: Bulk task pe cost aur latency dono explode, audit trail bhi mushkil. Fix: cascade — chhota model bulk, LLM sirf low-confidence/complex ghadi.

Desi example: Thane ka workflow — constable filter, DSP mushkil cases

Agar har complaint seedhe DSP ke paas jaaye toh dheere aur mehnga hoga. Thane mein kaam chalta hai: pehle clerical filter (chhota model — bulk, sasta), jo serious/confusing wo upar (LLM ya insaan — judgement), aur har decision register mein likha jaata hai (audit). Wahi cascade tumhari pipeline bhi hai:

inbox (bulk)    -> chhota classifier -> confident? -> seedha decision
                               nahi -> LLM / insaan (mushkil ghadi)
har row pe log  -> features + score -> audit trail
rare miss dekho -> bucket karo      -> agli improvement isi se

Interview angle — yahan se sawaal aate hain

Q: Multi-class aur multi-label mein farq? — Ek row ek category (ek hi score se competition) vs ek row mein kayi categories (har label ka independent score + threshold).
Q: Threshold kaise chunoge? — Error cost se: FP mehnga ho toh upar (precision), FN mehnga ho toh neeche (recall). Curve dekh kar, 0.5 koi rule nahi.
Q: BIO tagging kya hai? — B/I/O prefixes se entity spans: B-ORG shuru, I-ORG andar, O bahar; multi-word entities do token (B + I).
Q: Domain shift NER ko kaise todta hai? — Production ke entity patterns training se alag. Fix: haath se tagged sample se test + regex fallback fixed formats pe.
Q: LLM kab use karoge vs chhota model? — Labels/throughput/audit → chhota; ambiguity + kam volume → LLM; default cascade — bulk chhota, mushkil LLM.

Lesson complete?

Classic tasks aur unke failure modes ready — ab agla lesson Classical Metrics: BLEU, ROUGE & Perplexity mein dekhenge generation ke output ko naapne wale purane metrics kaise kaam karte hain aur kahan fail hote hain.