Lesson 05 � Intermediate
EDA: EXPLORE
YOUR DATA
EDA data samajhne ka sabse achha tareeka hai � patterns, trends, anomalies sab EDA se pata chalta hai. Data Cleaning ke baad sabse pehla kaam hota hai data ko explore karna. Jaise detective evidence dhundhta hai, waise hi data scientist EDA se insights dhundhta hai.
WHY: EDA kyun zaroori hai?
Data scientist ka kaam hai data se insights nikalna. Lekin insights tab tak nahi milenge jab tak data ko samajh nahi loge. EDA ek structured approach hai jismein aap data ko dekhte ho, uski summary nikalte ho, distribution check karte ho, aur relationships dhundhte ho. Bina EDA ke modeling karna jaise andheri kamari mein taala kholna hai � kuch samajh nahi aayega.
Data ki summary statistics nikalna � mean, median, mode, min, max, std dev. `df.describe()` se ek line mein poora data ka snapshot mil jaata hai. Sabse pehla step EDA ka.
Data kaise spread hai � normal hai, skewed hai, ya bimodal hai. Histogram aur KDE plot se pata chalta hai. Distribution samjho toh model choose karna aasan ho jaata hai.
Do columns ka relationship � ek badhta hai toh doosra bhi badhta hai ya girta hai. `df.corr()` se correlation matrix milta hai. Heatmap se visually samajh aata hai.
Data ko charts se dikhana � bar, scatter, box, histogram. Numbers samajhna mushkil hai, charts se patterns turant dikh jaate hain. Matplotlib aur Seaborn se banao.
HOW: EDA ka workflow
EDA ek systematic process hai. Randomly data dekhne se kuch nahi hota � ek step-by-step approach follow karo. Yeh 4-step framework yaad rakho.
EDA ka 4-step framework:
1. DESCRIBE ✓ Summary statistics nikalo � df.describe(), df.info(), df.shape
2. DISTRIBUTION✓ Har column ki spread dekho � histogram, value_counts()
3. CORRELATION ✓ Relationships dhundho � df.corr(), scatter plots
4. VISUALIZE ✓ Charts banao � bar, box, scatter, heatmapHOW: Python code se EDA karo
Ab practical dekhte hain. Neeche students ka data hai aur hum step by step EDA karenge. Har step ka code aur output hai.
import pandas as pd
import matplotlib.pyplot as plt
# Data load karo
df = pd.read_csv('students.csv')
# Step 1: Data ka overview
print("=== HEAD ===")
print(df.head())
print("\n=== INFO ===")
print(df.info())
print("\n=== DESCRIBE ===")
print(df.describe())
print("\n=== CORRELATION ===")
print(df.corr())
# Step 2: Groupby se analysis
print("\n=== COURSE WISE MARKS ===")
print(df.groupby('course')['marks'].mean())
# Step 3: Visualization
df.groupby('course')['marks'].mean().plot(kind='bar')
plt.title('Average Marks by Course')
plt.ylabel('Marks')
plt.show()DESCRIBE: Data ki summary statistics
Summary statistics EDA ka pehla step hai. `df.describe()` se mean, median, min, max, quartiles sab ek saath mil jaata hai. `df.info()` se data types aur missing values pata chalti hain. Yeh fast overview deta hai ki data kaisa hai.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'name': ['Aman', 'Priya', 'Rahul', 'Sneha', 'Vikram'],
'age': [22, 21, 23, 20, 24],
'marks': [85, 92, 78, 95, 60],
'salary': [50000, 55000, 48000, 62000, 45000]
})
# Basic info
print("=== SHAPE ===")
print(f"Rows: {df.shape[0]}, Columns: {df.shape[1]}")
print("\n=== INFO ===")
print(df.info())
print("\n=== DESCRIBE ===")
print(df.describe())
print("\n=== MEAN MARKS ===")
print(f"Average: {df['marks'].mean():.2f}")
print("\n=== MEDIAN MARKS ===")
print(f"Median: {df['marks'].median()}")
print("\n=== VALUE COUNTS (age) ===")
print(df['age'].value_counts())DISTRIBUTION: Data kaise spread hai
Distribution batata hai data kaise spread hai � kahan zyada values hain, kahan kam. Normal distribution hai toh parametric tests use kar sakte ho. Skewed hai toh non-parametric ya transformation chahiye. Histogram aur value_counts se check karo.
import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame({
'marks': [45, 52, 58, 62, 65, 68, 70, 72, 75, 78,
80, 82, 85, 85, 88, 90, 92, 95, 98, 100]
})
# Histogram � distribution dikhata hai
plt.figure(figsize=(10, 6))
plt.hist(df['marks'], bins=10, edgecolor='black', color='steelblue')
plt.title('Marks Distribution')
plt.xlabel('Marks')
plt.ylabel('Frequency')
plt.show()
# Skewness check
print(f"Skewness: {df['marks'].skew():.2f}")
# Positive = right skewed, Negative = left skewed
# Value counts for quick distribution
print("\n=== MARKS RANGES ===")
print(pd.cut(df['marks'], bins=[0,40,60,80,100],
labels=['Fail','Average','Good','Excellent']).value_counts())CORRELATION: Columns ka relationship
Correlation batata hai do columns ka relationship � ek badhta hai toh doosra bhi badhta hai (positive) ya girta hai (negative). Correlation 1 se -1 ke beech hota hai. 0 ka matlab koi relationship nahi. Heatmap se visually samajh aata hai.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
df = pd.DataFrame({
'study_hours': [2, 4, 6, 8, 10, 3, 5, 7, 9, 1],
'marks': [40, 55, 65, 80, 90, 45, 60, 75, 85, 35],
'attendance': [70, 80, 85, 90, 95, 75, 82, 88, 92, 65],
'salary': [30000, 45000, 55000, 70000, 85000, 35000, 50000, 65000, 80000, 25000]
})
# Correlation matrix
corr = df.corr()
print("=== CORRELATION MATRIX ===")
print(corr.round(2))
# Heatmap � visual correlation
plt.figure(figsize=(8, 6))
sns.heatmap(corr, annot=True, cmap='coolwarm', center=0)
plt.title('Correlation Heatmap')
plt.show()
# Strong correlations identify karo
print("\n=== STRONG CORRELATIONS (|r| > 0.7) ===")
for i in range(len(corr.columns)):
for j in range(i+1, len(corr.columns)):
if abs(corr.iloc[i, j]) > 0.7:
print(f"{corr.columns[i]} <-> {corr.columns[j]}: {corr.iloc[i, j]:.2f}")VISUALIZATION: Charts se patterns dhundho
Charts numbers se zyada fast samajh aate hain. Bar chart se comparison hota hai, scatter plot se relationship dikhti hai, box plot se outliers nazar aate hain. Har chart ka apna use case hai � sahi chart sahi jagah lagao.
import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame({
'course': ['Python', 'SQL', 'ML', 'DL', 'Python', 'SQL', 'ML', 'DL'],
'marks': [85, 78, 92, 70, 88, 82, 95, 75],
'students': [120, 95, 80, 45, 110, 90, 75, 40]
})
# 1. Bar Chart � comparison
plt.figure(figsize=(8, 5))
df.groupby('course')['marks'].mean().plot(kind='bar', color='steelblue')
plt.title('Average Marks by Course')
plt.ylabel('Marks')
plt.tight_layout()
plt.show()
# 2. Scatter Plot � relationship
plt.figure(figsize=(8, 5))
plt.scatter(df['students'], df['marks'], color='coral', s=100)
plt.title('Students vs Marks')
plt.xlabel('Number of Students')
plt.ylabel('Average Marks')
plt.show()
# 3. Box Plot � spread aur outliers
plt.figure(figsize=(8, 5))
df.boxplot(column='marks', by='course')
plt.title('Marks Distribution by Course')
plt.suptitle('')
plt.show()
# 4. Pie Chart � proportion
plt.figure(figsize=(8, 5))
df.groupby('course')['students'].sum().plot(kind='pie', autopct='%1.1f%%')
plt.title('Students by Course')
plt.ylabel('')
plt.show()Try it: EDA practice karo
Neeche ka editor Python jaisa hai. Yahan students data diya hai � EDA karo. describe(), corr(), groupby(), aur plots use karo. "Run Python" dabao aur output dekho.
Quick check
Data ka describe nikalo � ek line Python code likho jo students data ke liye summary statistics dikhaye.
Yaad karo � `df.describe()` method use hota hai jo mean, median, min, max, quartiles sab dikhata hai.
Common EDA mistakes
- Bina cleaning ke EDA mat karo: Missing values aur outliers hain toh describe() ka result galat aayega. Pehle data clean karo, phir explore karo.
- Sirf numbers pe mat jao: Describe se sirf summary milta hai. Distribution aur relationships samajhne ke liye charts zaroor banao. Numbers aur visuals dono chahiye.
- Correlation samjho: Correlation 0.8 hai toh matlab nahi ki ek dusre ka cause hai. Hamesha domain knowledge use karo. Correlation ✓ Causation.
- Har column ka EDA karo: Sirf target variable pe focus mat karo. Har feature ka distribution aur correlation check karo � kabhi kabhi unexpected features useful nikalte hain.
- Visualizations mat chhupo: EDA ka point hai insights dhundhna. Agar chart bana ke rakh diya aur analysis nahi kiya toh koi fayda nahi. Har chart ka interpretation likho.
Ab Visualization par chalo � Matplotlib aur Seaborn se advanced charts banao. EDA mein visualization sabse powerful tool hai.