Lesson 48 — Intermediate

Data Ingestion & Document Processing

RAG ki quality us data se banti hai jo index mein jaata hai. Is lesson mein hum PDF, HTML aur DOCX se text nikalna, OCR ka decision, structure ke saath cleaning, duplicates pehchanna aur incremental re-indexing — sab ek chhote Python pipeline ke saath dekhenge.

Ingestion kya hai

Funda simple hai: extraction sahi nahi hua toh chunk sahi nahi banega, chunk sahi nahi banega toh embedding bekaar. Isliye ingestion par debugging pehle karo — har doc pe ek sample nikaal ke padho, andhere mein batch mat chalao.

PDF, HTML aur DOCX se extract karna

from pathlib import Path

def extract(path: Path):
    suffix = path.suffix.lower()
    if suffix == ".pdf":
        return extract_pdf(path)      # text layer ho toh seedha, warna OCR
    if suffix in (".html", ".htm"):
        return extract_html(path)     # tags hatao, main content chhodo
    if suffix == ".docx":
        return extract_docx(path)     # headings + paragraphs ke saath
    raise ValueError("unsupported format: " + suffix)


def needs_ocr(sample_text: str) -> bool:
    letters = sum(ch.isalpha() for ch in sample_text)
    return letters < 30               # page bhar ka text khali = scan hai

• PDF: text layer ho toh extraction tez; image-only pages pe OCR chahiye (scanned bills, purane manuals).
• HTML: script/style/nav hatao, main content chhodo — warna menu ka text retrieval mein ghul jaata hai.
• DOCX: heading style ke saath paragraphs nikalo — yahi tumhara future structural chunking ka daana hai.
• Kab OCR: text layer khali ya padhne layak nahi — tabhi OCR, poore document pe andhe tareeke se nahi.

Cleaning — structure bachaate hue

import re

FOOTER = re.compile(r"Page \d+ of \d+")

def clean(text: str) -> str:
    text = text.replace("\u00a0", " ")      # non-breaking space
    text = FOOTER.sub("", text)             # har page ka footer hatao
    text = re.sub(r"[ \t]+", " ", text)     # extra spaces -> ek space
    text = re.sub(r"\n{3,}", "\n\n", text)  # 3+ newlines -> 2
    return text.strip()

• Cleaning ke baavjood headings, list markers aur page breaks mat udao — citation aur structural chunking isi pe chalti hai.
• Har doc ke saath metadata rakho: doc_id, heading path, page number, source path — baad mein citation isi se aayega.
• Regex lagane se pehle ek real sample pe chalao — regex kabhi kabhi asli content bhi uda deta hai.
• Karacters ki normalization: smart quotes, extra dashes, tabs — ye sab embedding ko noise dete hain.

Duplicates, updates aur re-indexing

import hashlib

def fingerprint(text: str) -> str:
    norm = " ".join(text.lower().split())   # case + space normalize
    return hashlib.sha256(norm.encode()).hexdigest()[:16]

# Har doc ke liye stored state (manifest):
# old = {"doc-7": fingerprint, ...}
# naya fingerprint same  -> skip (embedding bach gaye)
# naya fingerprint alag  -> re-extract + re-chunk + re-embed

naya doc      -> fingerprint naya    -> extract -> chunk -> embed -> index
updated doc   -> fingerprint alag    -> purane chunks delete -> naye embed
del doc       -> sirf chunks delete (embedding compute hi nahi)
same doc      -> skip (poora embedding bach gaye)

• Duplicates: same content do baar index mein ho toh top-k ka space waste hota hai aur answer ek hi baat dobara dohrata hai.
• Fingerprint text pe lagta hai, filename pe nahi — wahi PDF do naam se aayi toh bhi duplicate pakda jaayega.
• Updates: doc-level fingerprint diff karo, phir sirf badle hue docs ke chunks replace karo — incremental re-indexing.
• Deletes: doc delete ho toh uske saare chunk IDs purge karo, warna purana content abhi bhi retrieve hoga.

Ingestion ke common traps

• Sab kuch ek hi text blob mein dump karna — structure aur metadata khatam, citation baad mein mumkin hi nahi.
• Scanned PDF ko seedha embed kar dena bina OCR ke — khali ya garbage text ka embedding bhi garbage hoga.
• Header/footer/table ke repeated cells ko content maan lena — har chunk mein wahi noise baar baar aayegi.
• Extracted text kabhi padha hi nahi — batch chal gaya, numbers acche dikhe, par retrieval bekaar nikla.

Exercise

Common mistakes

• Sab kuch ek text blob mein dump karna: Structure aur metadata mita diya toh citation, heading-aware chunking aur page reference sab khatam. Fix: headings, page numbers aur doc metadata alag fields mein bachao.
• OCR chalaye bina scanned PDF embed karna: Image-only PDF se nikla khali text embed hota hi nahi — retrieval mein wo doc kabhi nahi aayega. Fix: pehle needs_ocr check, phir sirf required pages pe OCR.
• Har chhote update pe poora corpus re-embed karna: Ek file badli, poora index rebuild — time aur cost dono barbaad. Fix: content fingerprint se diff nikaal ke sirf badle hue documents re-process karo.

raw page  -> entry nikalo (extract)
          -> daag-dhabbe hatao (clean)
          -> fingerprint match? (dedupe)
          -> register mein daalo (index)
warna: photostat = ganda data register mein, baad mein dhoondhna impossible