Project 3 — Intermediate

Web Data Extraction Bot

Competitor roz prices change karta hai. Manual check = impossible. Is project me bot banayenge jo product page scrape kare, pagination handle kare, aur daily price report banaye.

⏱ 28 min• Intermediate• UiPath Studio (Community)

Project kya karega?

WHAT

Bot target website kholta hai (jaise ek e-commerce category page), har product ka naam, price, rating nikaalta hai, "Next" button se pagination karta hai, aur sab DataTable me collect karke CSV export karta hai. Daily schedule pe chalega.

WHEN

Competitor price monitoring, job portal scraping, real estate listings, lead generation. Jab bhi website se structured data chahiye aur API nahi hai — RPA web extraction ka area hai.

WHERE

E-commerce companies (pricing teams), digital agencies, market research firms. "Web scraping UiPath" searches me ye project sabse common requirement hai.

Ethics note: Sirf public data scrape karo, website ke robots.txt aur Terms of Service ka respect karo, rate limit maintain karo (delay lagao). Credential-protected ya private data scraping illegal hai — enterprise me bhi compliance team isi ko check karti hai.

Step-by-step build

Main.xaml (Sequence)
 |
 +-- Use Application/Browser  (Chrome, url: in_StartUrl)
 |    +-- Navigate To  in_StartUrl
 |    +-- Assign  intPage = 1
 |    +-- While  intPage <= in_MaxPages
 |         +-- Extract Structured Data  (template: product cards)  ->  dtPage
 |         +-- Merge Data Table  dtAll <- dtPage
 |         +-- Element Exists  (selector: Next button)  ->  blnNext
 |         +-- If  blnNext = True  AND  dtPage.Rows.Count > 0
 |              +-- Click  Next button
 |              +-- Delay  2000 ms   (throttling + page load wait)
 |              +-- Assign  intPage = intPage + 1
 |         +-- Else
 |              +-- Break   (last page)
 +-- Write CSV  dtAll  "C:\Output\prices_" + DateTime.Now.ToString("yyyyMMdd") + ".csv"

Data extraction ke 2 tarike hain — Data Scraping Wizard (Studio me Data Scraping button, 2 samples do, wizard selector khud bana deta hai — structured data ke liye best) aur Screen Scraping (unstructured text ke liye). Is project me Extract Structured Data use hoga.

Selectors jo tute nahi

# Fragile selector (ek class change = bot toot gaya):
<web ctrl name='₹1,299' class='price-tag-v2'/>

# Robust selector (wildcards + stable attributes):
<web ctrl id='product-list' tag='li' aaname='*Laptop*' />

# Rules:
# 1. id, tag, name attributes pe bharosa - class me versioning hoti hai
# 2. Wildcard (*) lagao variable parts pe (price, dynamic class)
# 3. UI Explorer me "Highlight" karke verify karo
# 4. Website redesign-proof nahi hota - Element Exists check zaroori hai

Error handling

Element not found: page load slow → Retry Scope (3 retries, 2 sec interval) lagao Click/Extract pe.
Ad/pop-up: cookie banner click hata do pehle — "Close" button ka selector banao, Element Exists se check karo.
Bot detection: 429 ya captcha aaye to bot ruk jaye, log karo, next run pe retry. Delay badhaao (3-5 sec).
Zero rows: page hi nahi khula — selector template galat. Blanking check: dtPage.Rows.Count = 0 pe log + continue.

Enterprise tips

Output CSV me scrape_date column add karo — historical price trend isi se banta hai.
Orchestrator me Trigger lagao daily 8 AM — schedule bot ko trigger better hai.
Proxy rotation aur rotating user-agent enterprise level pe hote hain — entry level pe delay + clean behavior kaafi hai.
Target selectors ka smoke test banao: pehle 1 page pe run karke validate, phir full pagination.

Interview question: "Website ka design badal gaya, bot kaise handle karega?" — 3 layer defense: (1) stable selectors (id/tag pe, wildcards variable parts pe), (2) Element Exists check + graceful skip with log, (3) selector ko Config.xlsx me externalize — bot bina code change ke selector update ho sakta hai. Selector healing (Modern folder structure) bhi enterprise feature hai.

Exercise

Question: Pages ke beech wait/throttle lagane ke liye kaunsi activity use hoti hai? (Ek word)

Question: Pagination kab rukna hai ye check karne ke liye "Next" button ke liye kaunsi activity use hoti hai? (2 words max)

Common mistakes

Delay nahi: 1 sec me 50 requests = website block. Delay rakho, respectful scraping karo.
Browser visible rakha production me: unattended bot ke liye Open Browser me headless (no-browser) mode on karo.
Infinite loop: Next button milta hi rehta agar content repeat ho — row-count check zaroori.
Selector copy-paste: UI Explorer se nikalo, wildcards samjho, Highlight karke verify karo.

Lesson complete?

Project ban gaya? Ab next project pe chalte hain — Email Triage & Attachment Bot.