Lesson 12 · Intermediate

DIMENSIONALITY REDUCTION
FEATURES KAM KARO, INFORMATION RAKHO!

Jab bahut saare features hote hain toh model slow ho jaata hai aur overfitting ka risk badh jaata hai. PCA (Principal Component Analysis) naye components banata hai jo maximum information capture karte hain � original features se kam dimensions mein zyada variance rakh sakta hai. Visualization bhi easy ho jaati hai jab 10 features ko 2-3 components mein compress karo.

◛ 22 min◉ Intermediate✓ Prerequisite: Clustering

WHY: Dimensionality Reduction kyun zaroori hai?

Real world datasets mein 100+ features hote hain � data, images, text. Har feature ek dimension hai. Zyada dimensions = zyada computation time, zyada storage, aur sabse bada problem: Curse of Dimensionality. Jaise jaise dimensions badhte hain, data points ek doosre se door ho jaate hain aur model ko patterns dhundhna mushkil ho jaata hai. PCA ek smart tareeka hai kam dimensions mein zyada information rakhne ka � jaise ek big file ko ZIP karna bina important data loss kiye.

PCA

Principal Component Analysis � ek technique jo original features ko naye components mein transform karta hai. Har component original features ka linear combination hai jo maximum variance capture karta hai.

VARIANCE

Kitna information ek feature rakhta hai. High variance = zyada information. PCA hamesha pehla component highest variance wala banata hai, doosra second highest, aur so on.

COMPONENTS

Naye features jo original features se bante hain. Principal Components unidirectional hain (orthogonal) aur ek doosre se independent hote hain. Isliye multicollinearity bhi solve hota hai.

SCREE PLOT

Ek graph jo dikhata hai har component kitna variance explain karta hai. Elbow point tak components choose karo � jahan curve suddenly flat ho jaata hai, wahan se kam dimensions milte hain bina information loss ke.

HOW: PCA kaise kaam karta hai?

PCA step by step kaam karta hai � pehle data ko standardize karta hai, phir covariance matrix nikalta hai, phir eigenvectors (directions) aur eigenvalues (magnitude) nikaalta hai. Har eigenvector ek principal component hai aur uska eigenvalue batata hai kitna variance capture hota hai.

PCA ka step-by-step process:

STEP 1: DATA STANDARDIZE KARO
 Har feature ka mean = 0, std = 1 banao
 Kyunki PCA magnitude pe depend karta hai
 Agar scale different hai toh wrong components aayenge

STEP 2: COVARIANCE MATRIX NIKALO
 Har feature pair ke beech ka relationship dekho
 Diagonal = variance of each feature
 Off-diagonal = covariance between features

STEP 3: EIGENVALUES & EIGENVECTORS NIKALO
 Eigenvectors = new directions (components)
 Eigenvalues = kitna variance hai har direction mein
 Zyada eigenvalue = zyada important direction

STEP 4: COMPONENTS CHOOSE KARO
 Eigenvalues ko descending order mein sort karo
 Top k components choose karo jo 95%+ variance explain karein
 Scree plot se elbow point dhundho

STEP 5: TRANSFORM KARO
 Original data ko naye components pe project karo
 X_pca = X_scaled @ eigenvectors[:, :k]
 Ab 100 features = 3 components!

Example:
 Original: 100 features (100 dimensions)
 After PCA: 3 components (3 dimensions)
 Variance explained: 97%
 Data: same information, 97% less storage!

✦

Mental model: PCA jaise 3D cheez ki 2D photo lena hai � aap ek angle se dekhte ho jo sabse zyada dikhata hai. Original cheez 3D mein hai lekin photo mein sab important features dikh rahe hain 2D mein. Har principal component ek naya angle hai jo zyada information capture karta hai.

HOW: Variance Explained kya hai?

Har principal component kitna variance (information) capture karta hai � yeh batata hai kitna important hai woh component. Agar pehla component 80% variance explain karta hai toh aap sirf ek component rakh ke bhi 80% information preserve kar sakte ho.

Variance Explained ka formula:

Total Variance = sum of all eigenvalues

Variance Explained by PCi = eigenvalue_i / total_variance

Example:
 5 features with eigenvalues: [3.5, 1.2, 0.5, 0.3, 0.2]
 Total variance = 3.5 + 1.2 + 0.5 + 0.3 + 0.2 = 5.7

 PC1 variance = 3.5/5.7 = 61.4%
 PC2 variance = 1.2/5.7 = 21.1%
 PC3 variance = 0.5/5.7 = 8.8%

 Cumulative:
 PC1: 61.4%
 PC1 + PC2: 82.5%
 PC1 + PC2 + PC3: 91.3% <- 3 components = 91% information!

Rule of thumb:
 95% variance explained = good
 90% variance explained = acceptable
 80% variance explained = minimum threshold

Scree Plot:
 X-axis: Component number (1, 2, 3, ...)
 Y-axis: Variance explained (%)
 Elbow point = jahan curve suddenly flat ho jaata hai
 Us point se pehle ka components choose karo

WHAT: Python code - PCA apply karo

Ab code karte hain! Sklearn mein PCA available hai. Pehle data standardize karo (StandardScaler), phir PCA fit karo. Variance ratio dekh ke decide karo kitne components rakhne hain.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

# Random data with 10 features
X = np.random.rand(100, 10) # 100 samples, 10 features
print(f"Original shape: {X.shape}")

# Step 1: Standardize
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Step 2: Apply PCA - 3 components
pca = PCA(n_components=3)
X_pca = pca.fit_transform(X_scaled)
print(f"Reduced shape: {X_pca.shape}")

# Step 3: Variance explained
print(f"\nVariance per component: {pca.explained_variance_ratio_}")
print(f"Total variance: {sum(pca.explained_variance_ratio_):.2%}")

# Step 4: Components dekho (kaise bane)
print(f"\nComponents shape: {pca.components_.shape}")
print("Component 1 weights:")
for i, w in enumerate(pca.components_[0]):
 print(f" Feature {i}: {w:.3f}")

Original shape: (100, 10)
Reduced shape: (100, 3)

Variance per component: [0.11 0.10 0.10]
Total variance: 31.54%

Components shape: (3, 10)
Component 1 weights:
 Feature 0: 0.321
 Feature 1: -0.187
 Feature 2: 0.445
 Feature 3: 0.098
 ...

HOW: Scree Plot se components choose karo

Scree plot batata hai kitne components rakhne hain. Elbow point dhundho � jahan curve suddenly flat ho jaata hai. Wahan se pehle ka components choose karo. Neeche code mein visualize karte hain.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

# Data with varying importance
np.random.seed(42)
X = np.random.rand(200, 8)
# Add some signal to first 3 features
X[:, :3] += np.random.randn(200, 3) * 2

# Standardize
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# PCA with all components
pca = PCA() # n_components = min(n_samples, n_features)
pca.fit(X_scaled)

# Variance explained
print("Variance explained per component:")
for i, var in enumerate(pca.explained_variance_ratio_):
 print(f" PC{i+1}: {var:.2%}")

# Cumulative variance
cumulative = np.cumsum(pca.explained_variance_ratio_)
print(f"\nCumulative variance:")
for i, cum_var in enumerate(cumulative):
 print(f" PC{i+1}: {cum_var:.2%}")
 if cum_var >= 0.95:
 print(f" -> 95% reached at PC{i+1}")
 break

# Scree plot data
print(f"\nScree Plot points:")
print(f"X: {[f'PC{i+1}' for i in range(len(pca.explained_variance_ratio_))]}")
print(f"Y: {[f'{v:.2%}' for v in pca.explained_variance_ratio_]}")

✦

Pro tip: Scree plot mein dekho � agar PC1 aur PC2 ka variance 90%+ hai toh aap sirf 2 components rakh sakte ho aur baki ke features discard kar sakte ho. Elbow point ke baad ka components noise hai, information nahi.

Try it: Python playground

Neeche ka editor Python jaisa hai. Yahan PCA ka code likho aur "Run Python" dabao. Screen par output dikhenge � yeh browser-based execution hai, real Python chalega.

Python playgroundPCA - Dimensionality Reduction visualization

main.py

# PCA - Dimensionality Reduction visualization
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
import numpy as np

# Iris dataset (4 features -> 2 components)
iris = load_iris()
X = iris.data
y = iris.target

print(f"Original: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Classes: {list(iris.target_names)}")

# Standardize
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# PCA - 4 components se 2 tak reduce
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

print(f"\nReduced: {X_pca.shape[0]} samples, {X_pca.shape[1]} components")

# Variance explained
print(f"\nVariance per component: {pca.explained_variance_ratio_}")
print(f"Total variance: {sum(pca.explained_variance_ratio_):.2%}")

# Component weights
print(f"\nPC1 weights: {pca.components_[0]}")
print(f"PC2 weights: {pca.components_[1]}")

# Class-wise centers
for i, name in enumerate(iris.target_names):
 mask = y == i
 center = X_pca[mask].mean(axis=0)
 print(f"\n{name} center: ({center[0]:.2f}, {center[1]:.2f})")

print("\n-> 4 features ko 2 components mein reduce kiya")
print("-> 95%+ information preserved!")

Optional input

Code ko apni info se update karke run karein

Quick check

PCA kaise kaam karta hai?

Sochho: PCA original features ka ek linear combination banata hai jo sabse zyada variance capture kare. Pehla component highest variance, doosra second highest. Isliye kam dimensions mein zyada information mil jaata hai.

Dimensionality Reduction vs Feature Selection

Dimensionality Reduction vs Feature Selection:

DIMENSIONALITY REDUCTION (PCA):
 Original features ko naye components mein transform karta hai
 Components original features ka combination hain
 Sab features use hote hain (koi discard nahi)
 Better jab features correlated hain
 Visualization ke liye best (2D/3D plotting)
 Example: 100 features -> 3 components

FEATURE SELECTION:
 Original features mein se important select karta hai
 Selected features same rehte hain (transform nahi)
 Irrelevant features discard hote hain
 Better jab features independent hain
 Interpretability better (original feature names)
 Example: 100 features -> 10 best features choose

When to use what:
 Features correlated hain -> PCA (dimensionality reduction)
 Features independent hain -> Feature selection
 Visualization chahiye -> PCA (2D/3D components)
 Interpretability chahiye -> Feature selection (original names)
 Training time kam chahiye -> Dono (kam features = faster)
 Overfitting kam chahiye -> Dono (regularization effect)

Common beginner mistakes

Standardize na karna: PCA ke pehle data standardize karna zaroori hai. Agar features different scales mein hain toh PCA galat components banayega. Hamesha StandardScaler use karo.
Sab components rakhna: PCA ka fayda hi hai ki kam dimensions mein zyada information rakho. Agar sab components rakhoge toh reduction ka koi fayda nahi. Elbow point peh components choose karo.
Categorical data pe PCA lagana: PCA sirf numerical data pe kaam karta hai. Categorical features ko pehle encode karo (one-hot, label encoding) phir PCA lagao.
PCA ke baad interpretation bhoolna: PCA components original features ke combinations hain. PC1 = 0.5*age + 0.3*income - 0.2*score � isko interpret karna zaroori hai model samjhne ke liye.
Non-linear data pe PCA lagana: PCA sirf linear relationships capture karta hai. Agar data non-linear hai toh t-SNE ya UMAP try karo � woh non-linear dimensionality reduction karte hain.

Dimensionality Reduction clear?

Ab Model Evaluation par chalo � accuracy, precision, recall, F1-score, confusion matrix sab kuch. Model kitna acha hai yeh evaluate karna seekho.

DIMENSIONALITY REDUCTIONFEATURES KAM KARO, INFORMATION RAKHO!