Lesson 135 — Advanced

Vision: Documents, Charts & Visual RAG

Duniya bhar ke business documents PDF mein band hain — kuch clean digital, kuch scanned photographs. Is lesson mein hum text-layer aur scan ka farq, OCR vs native vision reading, layout aur tables ki mushkilein, schema-first field extraction, page-region grounding, aur visual RAG — sab ek practical pipeline ke saath dekhenge. Ye L4 ki RAG knowledge ka visual extension hai.

Text-layer PDF vs scan

from pypdf import PdfReader

def route(pdf_path):
    r = PdfReader(pdf_path)
    first = r.pages[0].extract_text() or ""
    if len(first.strip()) < 30:          # text layer lagbhag khali
        return "scan", len(r.pages)      # OCR ya vision path
    return "digital", len(r.pages)      # seedha text extraction

kind, pages = route("invoice.pdf")
# har page ka route alag bhi ho sakta hai — mixed documents aam hain

OCR vs native vision reading

• OCR: tez, sasta, batch-friendly — par layout aksar bigad jaata hai; columns, headers aur tables ke relationships flatten ho sakte hain.
• Native vision reading: multimodal model page image dekh kar padhta hai — layout samajhta hai, par page image tokens kha jaati hai (cost & latency zyada).
• Mixed approach: text-layer ko guide ke roop mein use karo (kya likha hai) aur layout ke liye image bhejo — dono ka best.
• Reading order: multi-column pages, headers, footers, footnotes, sidebars — galat reading order sahi page ko bhi nonsense bana deta hai.

Decision rule: clean digital text ke liye text pipeline hi kaafi hai; scans ke liye vision path, par har page ko blind mat bhejo. Pehle ek sample page pe manually padho ki extraction kaisa nikla — reading order, tables, special characters. Batch shuru karne se pehle ye checkpoint compulsory hai.

Tables, forms aur charts

# Table extraction — merged cells, multi-page rows sab dhyan mein
# Pehle: schema define karo (free-form table dump kabhi nahi)
TABLE_SCHEMA = {
    "headers": ["item", "qty", "unit_price", "total"],
    "rows": [[str, int, float, float]],
    "continues_on_next_page": bool,
}

# Charts: exact values chahiye toh underlying data maango, model se guess nahi
# Model axes, labels aur legend padhta hai — number pixel se naapta nahi

• Merged cells: row ya column span ho toh value kis cell ki hai — ambiguity sabse common failure.
• Multi-page tables: header repeat hota hai, rows break hoti hain — page break pe row ko dobara mat jodo (ya jodo toh verify karo).
• Forms: checkbox, key aur value ke beech doori, signature block — har field ka label uske aas-paas nahi hota.
• Charts: model axes aur legend padh kar trend bata deta hai; exact value aksar estimate hoti hai. Precise numbers chahiye toh chart ke saath underlying table ya CSV lo.

Schema before prompt — extract & validate

SCHEMA = {
    "invoice_no": {"type": "string",   "required": True},
    "issue_date": {"type": "date",     "required": True},
    "total":      {"type": "number",   "required": True},
    "currency":   {"type": "string",   "required": True},
    "page":       {"type": "integer",  "required": True},   # grounding
}

def extract_and_validate(page_img):
    fields = model.ask(page_img, "schema ke mutabiq fields nikalo", schema=SCHEMA)
    problems = validate(fields, SCHEMA)        # type, range, date format, required
    problems += sanity(fields)                 # subtotal + tax == total?
    if problems:
        review_queue.append((page_img, fields, problems))   # DB mein mat daalo
    return fields if not problems else None

• Schema pehle, prompt baad mein: fields, types aur required list define karo — phir model se wahi maango. "Free-form summary" se DB-ready values kabhi nahi aati.
• Validate: type checks (number actually number?), range (total > 0), date format, required fields — aur cross-field sanity (subtotal + tax = total?).
• Ground: har field ke saath page number ya region bhejo, taaki human verify kar sake bina poora document dobara padhe.
• Route low confidence: validation fail ya model ki uncertainty — human review queue, silent DB write kabhi nahi.

Pages & regions as retrieval units

# Visual RAG: retrieval unit = page image (ya page ka region)
# Embedding kaise? — multimodal embedding model se: image aur text dono
# usi vector space mein aate hain (L12.9 mein detail)

def index_document(pdf):
    for i, page in enumerate(pdf.pages):
        text = extract_text_layer(page) or ocr(page)
        vector = multimodal_embed(page.image(), text=text)   # hybrid signal
        store(vector, {"doc": pdf.name, "page": i + 1, "image": page.image()})

# Query: user ka sawaal text hai, top-k mein page images aati hain,
# phir vision model un pages se jawab deta hai — citation ke saath

• Kab visual RAG: jab text layer nahi hai ya layout hi information hai (forms, diagrams, scanned contracts) — retrieval unit page image banna padta hai.
• Hybrid retrieval: extracted text (ya OCR) aur page-image embeddings dono search karo; ek sawaal ka answer text se aata hai, doosra layout se.
• Citation mandatory: jawab ke saath page number ya region do — reader document usi jagah khol ke check kar sake.
• Cost reality: page image tokens bahut zyada khaati hain — har query pe full-res pages bhejne se pehle region crop karo. Clean digital text kabhi kabhi visual pipeline ki demand hi nahi rakhta.

Practical combo: retrieval ke liye text embedding se search karo (sasta, tez), aur jo pages mile unka region crop karke vision model se padhwao. Poore corpus ko page-images mein re-embed karne ki zarurat tabhi hai jab layout retrieval ke liye hi zaroori ho.

Exercise

Common mistakes

• Har PDF ko OCR se guzar dena: Text-layer wale clean PDF ki zarurat hi nahi — beech ki quality aur layout dono kharab hote hain, cost bhi. Fix: pehle route karo (text layer hai ya nahi), phir decide karo.
• Extraction ke baad validation na karna: Model blank cell mein bhi value de sakta hai — bina checks ke wo chup-chaap DB mein chala jaata hai. Fix: schema validation + sanity (dates, totals, ranges) + low confidence pe review.
• Fields ke saath grounding na dena: Galat value nikli toh pata hi nahi chalega kahan se aayi. Fix: har field ke saath page/region cite karo — human verification isi se fast hoti hai.
• Poore corpus ko page-images mein embed kar dena: Visual RAG ka bill query-time pe explode karta hai. Fix: text-first retrieval + top-k pages pe visual reading; full visual indexing sirf layout-critical corpora ke liye.

printout  -> seedha text extraction (sasta, tez)
photostat -> vision reading (mehnga, par layout bachta hai)

har jawab ke saath: "dekh lo, yehan likha hai" (page + region cite)