Lesson 05 · Intermediate

DECISION TREES
KAISE KAAM KARTE HAIN?

Decision Tree ek flowchart jaisa model hai — rules se decisions leta hai. Jaise aap Socho: agar temperature > 30 hai toh AC chalao, warna fan. Tree bhi yahi karta hai — data ko questions ke through split karta hai jab tak pure groups na ban jaayein. Easy to understand aur interpret karna.

◛ 20 min◉ Intermediate✓ Prerequisite: Logistic Regression

WHY: Decision Tree kyun important hai?

Logistic Regression acha hai lekin kabhi kabhi data itna complex hota hai ki straight line se separate nahi hota. Decision Tree non-linear boundaries bana sakta hai — jaise real life mein aap ek flowchart follow karte ho ("agar baarish hai toh umbrella lelo, nahi toh chashma pehno"). Tree bhi exactly yahi karta hai data pe. Aur sabse best baat: tree ko explain karna easy hai — koi bhi dekh ke samajh sakta hai model kyun decide kar raha hai.

SPLIT

Data ko do parts mein divide karna based on a condition. Jaise "Temperature > 25?" — haan ek taraf, naa doosri taraf. Har split ek question hai jo data ko cleaner groups mein baanta hai.

GINI

Impurity measure — kitna mixed hai ek group. Gini = 0 matlab pure (sab ek class), Gini = 0.5 matlab 50-50 split. Tree hamesha lowest Gini wala split dhundhta hai.

DEPTH

Tree kitna deep hai — kitne levels tak splits hain. Zyada depth = zyada complex tree = overfitting ka risk. Shallow tree = simple but underfitting ho sakta hai.

PRUNE

Tree ko simplify karna — unnecessary branches katna. Jaise garden mein extra branches kaat dete ho. Pruning se overfitting kam hota hai aur tree faster chalta hai.

HOW: Decision Tree kaise banta hai?

Tree step by step banta hai — sabse pehla split sabse important question hota hai. Phir har group ke liye aur splits lagte hain jab tak pure groups na ban jaayein ya stopping condition na aaye.

Decision Tree banana - step by step:

STEP 1: BEST SPLIT DHUNDHO
 Har feature pe har possible threshold try karo
 Jis split pe sabse zyada Gini improvement ho, woh select karo
 Example: "Temperature > 25?" ya "Humidity > 60?"

STEP 2: DATA SPLIT KARO
 Condition true wale left, false wale right
 Har node pe ek question hota hai

STEP 3: REPEAT KARO
 Har group ke liye wapas best split dhundho
 Tab tak karo jab tak:
 Max depth na pahunch jaye
 Ya pure group na ban jaaye (sab ek class)
 Ya minimum samples na bach jaayein

STEP 4: LEAF NODE BANAO
 Jab split band ho, leaf node pe prediction aata hai
 Classification: majority class
 Regression: average value

Example tree:
 [Outlook?]
 / | \
 Sunny Overcast Rain
 | | |
 [Humid?] Yes [Windy?]
 / \ / \
 High Normal Yes No
 | | | |
 No Yes No Yes

Har internal node ek question hai
Har leaf node ek prediction hai

✦

Mental model: Decision Tree jaise 20 Questions game hai — aap ek cheez ke baare mein sochte ho aur tree har question ke baad options kam karta hai jab tak jawab na mil jaaye. Har split ek naya question hai.

HOW: Gini Impurity kya hai?

Gini Impurity batata hai ki ek group kitna mixed hai. Agar ek group mein sirf ek class hai toh Gini = 0 (pure). Agar 50-50 split hai toh Gini = 0.5 (maximum impurity). Tree hamesha Gini kam karna chahta hai.

Gini Impurity formula:

Gini = 1 - Sum(pi^2)

Jahan pi = probability of class i

Example 1: Pure group (sab "Yes")
 5 "Yes", 0 "No"
 Gini = 1 - (1.0^2 + 0^2) = 1 - 1 = 0
 ZERO impurity - bilkul pure!

Example 2: 50-50 split
 3 "Yes", 3 "No"
 Gini = 1 - (0.5^2 + 0.5^2) = 1 - 0.5 = 0.5
 Maximum impurity - sabse zyada mixed

Example 3: 70-30 split
 7 "Yes", 3 "No"
 Gini = 1 - (0.7^2 + 0.3^2) = 1 - 0.58 = 0.42
 Moderate impurity

Weighted Gini for a split:
 Gini_parent - weighted_avg(Gini_children)
 Jitna zyada reduction, utna better split!

Alternative: Entropy (Information Gain)
 Entropy = -Sum(pi * log2(pi))
 Same concept, different formula
 Sklearn mein criterion='entropy' use karo

WHAT: Python code - Decision Tree build karo

Ab code karte hain! Sklearn mein DecisionTreeClassifier aur DecisionTreeRegressor dono available hain. Neeche example mein tennis dataset pe tree banate hain.

from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split
import numpy as np

# Play tennis dataset
# Features: [Temperature, Wind]
# 0 = No play, 1 = Play
X = np.array([[1,1],[1,2],[2,1],[2,2],[3,1],[3,2],[4,1],[4,2]])
y = np.array([0,0,1,1,1,0,1,1])

X_train, X_test, y_train, y_test = train_test_split(
 X, y, test_size=0.3, random_state=42
)

# Decision Tree with max_depth=3
model = DecisionTreeClassifier(max_depth=3, random_state=42)
model.fit(X_train, y_train)

# Accuracy check
print(f"Accuracy: {model.score(X_test, y_test):.2%}")

# Tree structure dekho (text format)
print("\nDecision Tree Rules:")
print(export_text(model, feature_names=['Temp', 'Wind']))

# Feature importance dekho
print("\nFeature Importance:")
for name, imp in zip(['Temp', 'Wind'], model.feature_importances_):
 print(f" {name}: {imp:.3f}")

Accuracy: 66.67%

Decision Tree Rules:
|--- Temp <= 1.50
| |--- class: 0
|--- Temp > 1.50
| |--- Wind <= 1.50
| | |--- class: 1
| |--- Wind > 1.50
| | |--- class: 0

Feature Importance:
 Temp: 0.625
 Wind: 0.375

HOW: Pruning - Tree ko simplify karo

Agar tree bahut deep hai toh woh training data ko yaad rakh leta hai (overfitting). Pruning se extra branches kat ke tree ko simple banate hain.

from sklearn.tree import DecisionTreeClassifier
import numpy as np

# Overfitting ka example
X = np.random.rand(100, 5)
y = (X[:, 0] + X[:, 1] > 1).astype(int)

# Bina pruning - tree overfit karega
tree_no_prune = DecisionTreeClassifier(random_state=42)
tree_no_prune.fit(X, y)
print(f"No pruning - Depth: {tree_no_prune.get_depth()}")
print(f"No pruning - Leaves: {tree_no_prune.get_n_leaves()}")
print(f"No pruning - Train Accuracy: {tree_no_prune.score(X, y):.2%}")

# Pre-pruning (max_depth, min_samples)
tree_pre = DecisionTreeClassifier(
 max_depth=3,
 min_samples_split=10,
 min_samples_leaf=5,
 random_state=42
)
tree_pre.fit(X, y)
print(f"\nPre-pruning - Depth: {tree_pre.get_depth()}")
print(f"Pre-pruning - Leaves: {tree_pre.get_n_leaves()}")
print(f"Pre-pruning - Train Accuracy: {tree_pre.score(X, y):.2%}")

# Post-pruning (cost complexity)
tree_post = DecisionTreeClassifier(ccp_alpha=0.01, random_state=42)
tree_post.fit(X, y)
print(f"\nPost-pruning - Depth: {tree_post.get_depth()}")
print(f"Post-pruning - Leaves: {tree_post.get_n_leaves()}")
print(f"Post-pruning - Train Accuracy: {tree_post.score(X, y):.2%}")

✦

Pre-pruning vs Post-pruning: Pre-pruning = pehle se rules lagao (max_depth set karo). Post-pruning = pehle pura tree banao, phir extra branches kaato. Pre-pruning faster hai, post-pruning zyada accurate hota hai.

Try it: Python playground

Neeche ka editor Python jaisa hai. Yahan Decision Tree ka code likho aur "Run Python" dabao. Screen par output dikhenge — yeh browser-based execution hai, real Python chalega.

Python playgroundDecision Tree - Iris dataset classification

main.py

# Decision Tree - Iris dataset classification
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import numpy as np

# Iris dataset load karo
iris = load_iris()
X = iris.data
y = iris.target

print(f"Dataset: {X.shape[0]} flowers, {X.shape[1]} features")
print(f"Classes: {list(iris.target_names)}")

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(
 X, y, test_size=0.3, random_state=42
)

# Decision Tree banao
model = DecisionTreeClassifier(max_depth=3, random_state=42)
model.fit(X_train, y_train)

# Accuracy check
train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
print(f"\nTrain Accuracy: {train_acc:.2%}")
print(f"Test Accuracy: {test_acc:.2%}")

# Tree rules dekho
print(f"\nTree Depth: {model.get_depth()}")
print(f"Leaf Nodes: {model.get_n_leaves()}")
print("\nDecision Tree Rules:")
print(export_text(model, feature_names=iris.feature_names))

# Feature importance
print("\nFeature Importance:")
for name, imp in sorted(zip(iris.feature_names, model.feature_importances_),
 key=lambda x: x[1], reverse=True):
 print(f" {name}: {imp:.3f}")

Optional input

Code ko apni info se update karke run karein

Quick check

Decision tree ka Gini impurity kya hai?

Sochho: agar ek group mein sab ek jaise hain toh Gini 0 hai (pure). Agar 50-50 split hai toh Gini 0.5 hai (maximum impurity). Yeh batata hai kitna mixed hai data.

Decision Trees vs Logistic Regression

Decision Trees vs Logistic Regression:

LOGISTIC REGRESSION:
 Linear boundaries
 Fast training
 Needs feature scaling
 Easy to interpret (coefficients)
 Good for linearly separable data
 Less prone to overfitting (with regularization)

DECISION TREES:
 Non-linear boundaries
 Medium training speed
 No feature scaling needed
 Easy to interpret (visual tree)
 Good for complex data with interactions
 Prone to overfitting (needs pruning)

When to use what:
 Linear data -> Logistic Regression
 Non-linear data -> Decision Trees
 Need interpretability -> Both (different ways)
 Want ensemble base -> Decision Trees (Random Forest = many trees)

Common beginner mistakes

Depth bina control kiye tree banana: Agar max_depth set nahi kiya toh tree 100% train accuracy dega lekin test pe flop hoga (overfitting). Hamesha max_depth limit karo.
Feature scaling karna: Decision Trees ko scaling ki zaroorat nahi hai — woh thresholds pe kaam karta hai, distances pe nahi. Logistic Regression mein scaling zaroori hai.
Pruning skip karna: Bina pruning ke tree overfit karta hai. Pre-pruning (max_depth, min_samples) ya post-pruning (ccp_alpha) use karo.
Single tree pe bharosa karna: Single tree unstable hota hai — thoda sa data change ho toh poora tree badal jaata hai. Isliye Random Forest use karte hain (bohot saare trees ka ensemble).
Interpretability ka fayda nahi lena: Tree ka biggest advantage hai ki usko explain karna easy hai. Agar model samjhana hai stakeholder ko, tree ka visualization dikhao.

Decision Trees clear?

Ab Random Forest par chalo — bohot saare decision trees ka ensemble jo overfitting solve karta hai. Single tree ki weakness yahan strength ban jaati hai.

DECISION TREESKAISE KAAM KARTE HAIN?