Data Cleaning � Real Data par Jo Aata Hai

Jo 80% time data scientist market mein data cleaning hi hota hai � aur zyada log isliye fail hote hain kyunki unke datasets gande hote hain. Is lesson mein seekhoge: missing values, duplicates, outliers aur inconsistent formats kya hote hain, aur unhe pandas se kaise saaf karte hain. � Hinglish mein, with code.

WHY: Data Cleaning kyun zaroori hai?

Model ka result utna hi accha hota hai jitna uska data. Agar aapke data mein 30% missing rows hain, duplicates present hain aur date formats alag-alag hain � koi bhi machine learning model sahi kaam nahi karega. Ranges: GIGO � Garbage In, Garbage Out. Ek clean dataset model accuracy ko easily 20-30% improve kar deta hai.

Real companies measurement karti hain � Excel rakhne waale 80% time bad data par fail ho jaate hain. Isliye data cleaning ko skill ki tarah practice karo.

HOW: Few step-to-step data cleaning approaches with pandas

1. Profile the data � pehle data ko dekho

df.info(); df.shape; df.head()

Pehle samjho kitne rows/columns hain, kaunsa column kitna null hai, kya dtypes sahi hain. Ye steps code likhne se pehle must hai.

2. Missing values � drop ya fill?

df.dropna(subset=['salary']) # important rows khali to hataya
df['age'].fillna(df['age'].median()) # numeric � median fill
df['city'].fillna('Unknown') # categorical � fill string

Decision rule: missing kam to drop, missing zyada par rows important hain to fill karo. APna decision ek note mein likh kar justify karo.

3. Duplicates hatao

df.duplicated().sum()
df.drop_duplicates(subset=['email'], inplace=True)

4. Inconsistent formats

df['date'] = pd.to_datetime(df['date'], format='%d/%m/%Y')
df['category'] = df['category'].str.strip().str.lower()
df['amount'] = df['amount'].str.replace('?','').str.replace(',','').astype(float)

Same cheez alag format mein (?4,999 vs 6999) chhut jati hai to analysis galat hota hai � standardise karo.

5. Outliers ko samjho, delete nahi

q1 = df['amount'].quantile(0.25); q3 = df['amount'].quantile(0.75)
iqr = q3 - q1
df = df[(df['amount'] > q1 - 1.5*iqr) & (df['amount'] < q3 + 1.5*iqr)]

Outliers ko hamesha context ke saath handle karo � senior analyst jobs mein report karna hota hai, blindly delete nahi.

Try it: Data cleaning practice karo

Ek kenchaa dataset banao jisme 10% missing values aur 5 duplicates ho, phir upar ke 5 steps ko ek Jupyter notebook mein apply karo. Final sheet mein batao � kitne rows the, kitne saaf hue, kya decision le liye.

Pro cleaning tips

• Har step ke saath code ekdum small keep karo � ek cell mein ek hi task.
• Original data ka backup hamesha rakho, processed dataframe alag naam se save karo.
• Log karo (comments) ki kisi column ko kyun saaf kiya � 3 mahine baad wahi reasoning yaad nahi rehti.
• Practice data cleaning on raw CSV files from real sources (not perfect toy datasets).

Common data cleaning mistakes

• Outliers ko bina context delete karna � poora customer base hata sakta hai.
• Missing-drop karna jab wo rows aapke analysis mein zaroori hain.
• Feature engineering mein train aur test data ko salah mix karna (data leakage).
• Dummy variables banana with too many categories � memory aur model size badh jate hain.