Lesson 13 · Intermediate

HYPERPARAMETER TUNING
MODEL KE BEST SETTINGS DHOONDHO!

Jaise car ke liye sahi gear combination zaroori hai waise hi ML model ke liye sahi hyperparameters zaroori hain. Grid Search sab combinations try karta hai, RandomizedSearch random picks karta hai � dono se best parameters milte hain jo model ko fastest aur most accurate banate hain.

◛ 20 min◉ Intermediate✓ Prerequisite: Model Evaluation

WHY: Hyperparameter Tuning kyun zaroori hai?

Model train karna toh easy hai � model.fit(). Lekin model ko truly efficient banane ke liye hyperparameters set karna padta hai. Default values hamesha best nahi hoti. Jaise Random Forest mein n_estimators=100 default hai, lekin shayad 200 trees pe accuracy zyada aaye. Ya max_depth=5 pe overfitting kam ho. Hyperparameter tuning automated tareeka hai sahi values dhundhne ka � manual try karne se better hai kyunki computer sab combinations test kar leta hai.

GRID SEARCH

Sab possible combinations try karta hai � har parameter ki har value ke saath. Thorough hai lekin slow. Small parameter space ke liye best.

RANDOMIZED SEARCH

Randomly kuch combinations pick karta hai � fixed iterations tak. Fast hai aur bade parameter space ke liye kaam karta hai. Best nahi milta hamesha, lekin time bachta hai.

CROSS-VALIDATION

Data ko K folds mein todta hai � har fold baar ek part validation ke liye, baaki training. Isse model ka true performance pata chalta hai, sirf ek train-test split se better estimate milta hai.

BEST PARAMS

Tuning ke baad jo parameters sabse zyada score dein � woh best_params_ mein milte hain. In params se final model train karo aur deploy karo.

HOW: Hyperparameters vs Parameters kya hai?

Dono alag cheezein hain � parameters model seekhta hai training se, hyperparameters hum set karte hain training se pehle. Samajhna zaroori hai kyunki tuning sirf hyperparameters ki hoti hai.

Hyperparameters vs Parameters:

PARAMETERS (model seekhta hai):
 Training ke dauran automatically set hote hain
 Data se directly learn hote hain
 Examples:
 - Linear Regression: coefficients (weights), intercept
 - Decision Tree: split points, feature thresholds
 - Neural Network: weights, biases
 Hum inhe nahi set karte � model ka kaam hai

HYPERPARAMETERS (hum set karte hain):
 Training se pehle set karte hain
 Model ki learning process control karte hain
 Examples:
 - n_estimators: kitni trees (Random Forest)
 - max_depth: tree kitni deep jaaye
 - learning_rate: kitna seekhe har step mein
 - C: regularization strength (SVM, Logistic)
 - k: kitne neighbors (KNN)
 Hum inhe tune karte hain best performance ke liye

Key difference:
 Parameters = model ka ANDAR ka kaam
 Hyperparameters = model ke BAHAR ka control

✦

Mental model: Hyperparameters jaise car ka dashboard settings hain � AC temperature, seat position, steering adjustment. Yeh car chalane se pehle set karte ho. Parameters jaise engine ka internal kaam � fuel combustion, piston movement � yeh car automatically karta hai. Tuning matlab sahi dashboard settings dhundhna jo driving best banaye.

HOW: Grid Search kaise kaam karta hai?

Grid Search sab possible combinations try karta hai. Agar 2 parameters hain � n_estimators: [50, 100, 200] aur max_depth: [3, 5, 10] � toh 3 x 3 = 9 combinations test hongi. Har combination pe cross-validation hoti hai aur best score wala winner hota hai.

Grid Search ka process:

STEP 1: PARAMETER GRID BANAO
 Dict of lists � har parameter ki possible values
 Example:
 param_grid = {
 'n_estimators': [50, 100, 200],
 'max_depth': [3, 5, 10, None]
 }
 Total combos: 3 x 4 = 12

STEP 2: GRIDSEARCHCV SETUP KARO
 grid = GridSearchCV(
 estimator=model,
 param_grid=param_grid,
 cv=5, # 5-fold cross-validation
 scoring='accuracy',
 n_jobs=-1 # parallel processing
 )

STEP 3: FIT KARO (sab combinations try hongi)
 grid.fit(X, y)
 Har combination pe:
 1. Model train on 4 folds
 2. Validate on 1 fold
 3. Score record karo
 4. Best score update karo

STEP 4: RESULTS LO
 grid.best_params_ # best combination
 grid.best_score_ # best cross-val score
 grid.best_estimator_ # best model (ready to use)

Combinations:
 n=50, depth=3 -> score
 n=50, depth=5 -> score
 n=50, depth=10 -> score
 n=50, depth=None -> score
 n=100, depth=3 -> score
 ... and so on
 Best combination = winner!

HOW: RandomizedSearch kaise kaam karta hai?

RandomizedSearch sab combinations nahi, sirf random picks karta hai � n_iter iterations tak. Zyada parameters ya badi grid ho toh fast hai. Har iteration pe ek random combination pick hota hai aur cross-validation hoti hai.

RandomizedSearch ka process:

STEP 1: PARAMETER DISTRIBUTION BANAO
 RandomizedSearch distributions leta hai
 Example:
 param_dist = {
 'n_estimators': [50, 100, 200, 300],
 'max_depth': [3, 5, 7, 10, 15, None]
 }
 Total combos: 4 x 6 = 24

STEP 2: RANDOMIZEDSEARCHCV SETUP KARO
 random = RandomizedSearchCV(
 estimator=model,
 param_distributions=param_dist,
 n_iter=8, # sirf 8 random combos try karo
 cv=5,
 scoring='accuracy',
 random_state=42,
 n_jobs=-1
 )

STEP 3: FIT KARO (random picks hongi)
 random.fit(X, y)
 Sirf 8 combinations test honge (24 mein se)
 Har combination pe cross-validation

STEP 4: RESULTS LO
 random.best_params_
 random.random_score_

Grid vs Random comparison:
 Grid: 24 combos = 24 fits (slow but thorough)
 Random: 8 combos = 8 fits (fast, usually close)

Jab grid bahut badi ho (1000+ combos):
 GridSearch = time bahut lagega
 RandomizedSearch = smart choice!

WHAT: Python code � Grid Search aur RandomizedSearch

Ab code karte hain! Sklearn mein GridSearchCV aur RandomizedSearchCV dono available hain. Neeche dono ka example hai � dekho kaunsa fast hai aur kaunsa zyada combinations try karta hai.

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import numpy as np

X, y = make_classification(n_samples=200, n_features=5, random_state=42)

# Grid Search
param_grid = {
 'n_estimators': [50, 100, 200],
 'max_depth': [3, 5, 10, None]
}

grid = GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring='accuracy')
grid.fit(X, y)
print(f"Best params: {grid.best_params_}")
print(f"Best score: {grid.best_score_:.2%}")

# Random Search
random = RandomizedSearchCV(RandomForestClassifier(), param_grid, n_iter=6, cv=5)
random.fit(X, y)
print(f"Random best: {random.best_params_}")

Best params: {'max_depth': 10, 'n_estimators': 200}
Best score: 88.50%
Random best: {'n_estimators': 200, 'max_depth': 10}

HOW: Cross-Validation kaise kaam karta hai?

Cross-validation data ko K equal parts (folds) mein todta hai. Har baar ek fold validation ke liye aur baaki K-1 folds training ke liye. Isse model ka true performance pata chalta hai � sirf ek train-test split se zyada reliable hai kyunki har data point ek baar validation mein aata hai.

K-Fold Cross-Validation:

Data: 100 samples, K=5 folds

Fold 1: [Val] [Train] [Train] [Train] [Train] -> score1
Fold 2: [Train] [Val] [Train] [Train] [Train] -> score2
Fold 3: [Train] [Train] [Val] [Train] [Train] -> score3
Fold 4: [Train] [Train] [Train] [Val] [Train] -> score4
Fold 5: [Train] [Train] [Train] [Train] [Val] -> score5

Final Score = (score1 + score2 + score3 + score4 + score5) / 5

Benefits:
 - Har data point ek baar validation mein aata hai
 - Score zyada reliable hota hai
 - Overfitting ka risk kam hota hai
 - Model ki generalization pata chalti hai

Why CV > single train-test split:
 Single split: ek baar split, ek score
 (agar split unlucky hai toh score galat)
 CV: K baar split, K scores ka average
 (average reliable hota hai)

Common K values:
 K=5: most common, good balance
 K=10: zyada reliable, thoda slow
 K=3: fast, kam reliable
 K=N (Leave-One-Out): sabse slow, sabse reliable

✦

Pro tip: GridSearchCV aur RandomizedSearchCV mein cv=5 ka matlab 5-fold cross-validation. Har parameter combination pe 5 baar model train hota hai. Total fits = combinations x K. Isliye K zyada mat rakho � 5 ya 10 kaafi hai.

Try it: Python playground

Neeche ka editor Python jaisa hai. Yahan GridSearch ka code likho aur "Run Python" dabao. Screen par output dikhenge � yeh browser-based execution hai, real Python chalega.

Python playgroundGridSearchCV � Hyperparameter Tuning

main.py

# GridSearchCV � Hyperparameter Tuning
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score
import numpy as np

# Synthetic classification data
X, y = make_classification(
 n_samples=300, n_features=8, n_informative=5,
 random_state=42
)

print(f"Data: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Classes: {np.bincount(y)}")

# Parameter grid
param_grid = {
 'n_estimators': [50, 100, 200],
 'max_depth': [3, 5, 10, None],
 'min_samples_split': [2, 5, 10]
}
total_combos = 3 * 4 * 3
print(f"\nTotal combinations: {total_combos}")

# Grid Search
grid = GridSearchCV(
 RandomForestClassifier(random_state=42),
 param_grid, cv=5, scoring='accuracy', n_jobs=-1
)
grid.fit(X, y)

print(f"\n--- Grid Search Results ---")
print(f"Best params: {grid.best_params_}")
print(f"Best CV score: {grid.best_score_:.2%}")
print(f"Fits performed: {total_combos * 5}")

# Random Search (sirf 10 random combos)
random = RandomizedSearchCV(
 RandomForestClassifier(random_state=42),
 param_grid, n_iter=10, cv=5,
 scoring='accuracy', random_state=42, n_jobs=-1
)
random.fit(X, y)

print(f"\n--- Random Search Results ---")
print(f"Best params: {random.best_params_}")
print(f"Best CV score: {random.best_score_:.2%}")
print(f"Fits performed: {10 * 5}")

# Best model use karo
best_model = grid.best_estimator_
print(f"\nBest model: {best_model}")
print(f"-> Tuned model ready for deployment!")

Optional input

Code ko apni info se update karke run karein

Quick check

GridSearchCV kya karta hai?

Sochho: Grid Search ek dict leta hai jismein parameter ki values hain. Phir har parameter ki har value ke saath sab combinations try karta hai. Har combination pe cross-validation hoti hai. Jo combination sabse zyada score de � woh best_params_ mein milta hai.

Grid Search vs RandomizedSearch

Grid Search vs RandomizedSearch:

GRID SEARCH (GridSearchCV):
 Sab combinations try karta hai
 Har parameter ki har value
 Thorough lekin slow
 Best parameters guaranteed milte hain
 Jab parameter space chhota ho (10-20 combos)
 Example: 3 x 4 x 3 = 36 fits

RANDOMIZED SEARCH (RandomizedSearchCV):
 Randomly n_iter combinations pick karta hai
 Fixed iterations tak chalta hai
 Fast hai, time bachta hai
 Best nahi milta hamesha, lekin kaafi close
 Jab parameter space bada ho (100+ combos)
 Example: sirf 10 fits (36 mein se)

When to use what:
 Chhota parameter space -> GridSearch (thorough)
 Bada parameter space -> RandomizedSearch (fast)
 Time kam hai -> RandomizedSearch
 Accuracy critical hai -> GridSearch
 Quick experiment -> RandomizedSearch
 Production tuning -> GridSearch (best combo)

Cost comparison:
 Grid: O(K^d) combinations, K=folds, d=parameters
 Random: O(n_iter) combinations, fixed cost

Rule of thumb:
 Total combos < 50 -> GridSearch
 Total combos 50-500 -> RandomizedSearch (n_iter=20-50)
 Total combos > 500 -> RandomizedSearch (n_iter=50-100)

ADVANCED: Aur kya kya tune kar sakte ho?

Sirf Random Forest nahi � har ML model ke hyperparameters tune ho sakte hain. Neeche kuch common examples hain jo dikhaate hain tuning kitni flexible hai.

# Different models ke hyperparameters

# 1. Logistic Regression
from sklearn.linear_model import LogisticRegression
param_grid_lr = {
 'C': [0.01, 0.1, 1, 10, 100], # regularization
 'penalty': ['l1', 'l2'], # penalty type
 'solver': ['liblinear', 'saga'] # optimization algo
}

# 2. Support Vector Machine
from sklearn.svm import SVC
param_grid_svm = {
 'C': [0.1, 1, 10, 100],
 'kernel': ['linear', 'rbf', 'poly'],
 'gamma': ['scale', 'auto', 0.01, 0.1]
}

# 3. Gradient Boosting
from sklearn.ensemble import GradientBoostingClassifier
param_grid_gb = {
 'n_estimators': [50, 100, 200],
 'learning_rate': [0.01, 0.1, 0.2],
 'max_depth': [3, 5, 7],
 'subsample': [0.8, 1.0]
}

# 4. KNN
from sklearn.neighbors import KNeighborsClassifier
param_grid_knn = {
 'n_neighbors': [3, 5, 7, 11, 15],
 'weights': ['uniform', 'distance'],
 'metric': ['euclidean', 'manhattan', 'minkowski']
}

# Har model ka tuning process same hai:
# 1. Param grid banao
# 2. GridSearchCV / RandomizedSearchCV setup karo
# 3. Fit karo
# 4. Best params lo
# 5. Best model use karo

✦

Pro tip: param_distributions mein continuous values bhi de sakte ho � 'learning_rate': [0.001, 0.01, 0.1, 0.2] ya 'C': np.logspace(-3, 3, 7). RandomizedSearch sirf n_iter tak combos try karega � isliye badi grid mein bhi fast rahega.

Common beginner mistakes

Parameters aur hyperparameters confuse karna: coef_ aur intercept_ parameters hain jo model seekhta hai. n_estimators, max_depth hyperparameters hain jo hum set karte hain. Tuning sirf hyperparameters ki hoti hai.
Test data pe tune karna: GridSearchCV hamesha training data pe chalta hai (cross-validation se). Kabhi test data ko tune mein mat daalo � woh final evaluation ke liye hai. Data leakage ho jaayega.
Boht zyada combinations try karna: 1000+ combinations means 1000 x K fits. Time bahut lagega aur overfitting ka risk bhi. Chhoti grid se start karo, phir refine karo.
Cross-validation skip karna: cv=None mat rakho � default hota hai but explicit cv=5 ya cv=10 likho. CV ke bina sirf ek split se score milega jo unreliable hai.
Best score ko final score samajhna: best_score_ cross-validation ka score hai, test accuracy nahi. Final evaluation hamesha test set pe karo after tuning.

Hyperparameter Tuning clear?

Ab Ensemble Methods par chalo � Boosting, Stacking, Voting, XGBoost sab kuch. Multiple models ka power dekho!

HYPERPARAMETER TUNINGMODEL KE BEST SETTINGS DHOONDHO!