Intermediate 22 min

Model Evaluation

Prerequisite: Dimensionality Reduction

Model Evaluation kyun zaroori hai?

Model banane ke baad evaluate karna zaroori hai � kitna accurate hai, overfit toh nahi ho raha. Metrics se pata chalta hai ki model kitna acha kaam kar raha hai.

Socho tumne ek model banaya jo 99% accuracy de raha hai test data pe � but kya sach mein itna acha hai✓ ya sirf training data ko yaad kar raha hai✓ Evaluation se hum ye sab pata lagate hain.

Key Metrics � Concept Grid

ACCURACY

Correct predictions ka total percentage. Simple hai � kitne sahi guess kiye total mein se.

Accuracy = (TP + TN) / (TP + TN + FP + FN)
💡

PRECISION

Jab model positive predict karta hai, usme se kitne actually positive the. False positive kam hona chahiye.

Precision = TP / (TP + FP)

RECALL

Sari actual positives mein se kitne ko model ne find kiya. False negative kam hona chahiye.

Recall = TP / (TP + FN)

F1 SCORE

Precision aur Recall ka harmonic mean. Dono ka balance dikhata hai � ek high aur ek low ho toh F1 down hoga.

F1 = 2 � (Precision � Recall) / (Precision + Recall)

Confusion Matrix samjho

Confusion Matrix ek table hai jo dikhata hai kitne predictions sahi the aur kitne galat � category wise.

Predicted: Positive Predicted: Negative
Actual: Positive True Positive (TP) False Negative (FN)
Actual: Negative False Positive (FP) True Negative (TN)
Yaad rakho:
  • TP: Model ne positive bola aur actually positive tha
  • TN: Model ne negative bola aur actually negative tha
  • FP: Model ne positive bola but actually negative tha (Type I error)
  • FN: Model ne negative bola but actually positive tha (Type II error)

Code Example � Metrics calculate karo

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.model_selection import cross_val_score
import numpy as np

# Actual aur predicted values
y_true = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 1, 0, 0, 1, 1, 0, 1, 0]

# Basic metrics
print(f"Accuracy: {accuracy_score(y_true, y_pred):.2%}")
print(f"Precision: {precision_score(y_true, y_pred):.2%}")
print(f"Recall: {recall_score(y_true, y_pred):.2%}")
print(f"F1: {f1_score(y_true, y_pred):.2%}")

# Confusion Matrix
print(f"\nConfusion Matrix:\n{confusion_matrix(y_true, y_pred)}")

# Full classification report
print("\nClassification Report:")
print(classification_report(y_true, y_pred))
Expected Output: 
Accuracy: 80.00%
Precision: 80.00%
Recall: 80.00%
F1: 80.00%

Confusion Matrix:
[[4 1]
 [1 4]]

Classification Report:
 precision recall f1-score support

 0 0.80 0.80 0.80 5
 1 0.80 0.80 0.80 5

 accuracy 0.80 10
 macro avg 0.80 0.80 0.80 10
weighted avg 0.80 0.80 0.80 10

Cross Validation � Reliable Evaluation

Ek baar split karke evaluate karna risky hai � data ka luck factor ho sakta hai. Cross validation se data ko k folds mein baant ke har fold pe evaluate karte hain.

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# Sample data banao
X, y = make_classification(n_samples=100, n_features=20, random_state=42)

# Random Forest model
model = RandomForestClassifier(n_estimators=100, random_state=42)

# 5-fold cross validation
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')

print(f"Fold scores: {scores}")
print(f"Mean Accuracy: {scores.mean():.2%}")
print(f"Std Deviation: {scores.std():.2%}")
Pro Tip: CV score ka standard deviation batata hai model kitna stable hai. Agar zyada hai toh data pe dependent hai.

Kab kya use karein?

Scenario Best Metric Kyun?
Balanced classes Accuracy Sab classes equal hain, overall correct matter karta hai
Imbalanced data (fraud detection) Precision False positive kam chahiye � galat fraud flag nahi karna
Cancer detection Recall Ek bhi positive miss nahi hona chahiye � life threatening hai
Balance chahiye dono mein F1 Score Precision aur Recall dono important hain

Interactive Editor � Model Evaluate karo

Neeche diye gaye code ko run karo aur metrics dekho:

model_evaluation.py 
Accuracy: 85.00%
Precision: 83.33%
Recall: 83.33%
F1 Score: 83.33%

Confusion Matrix:
[[8 2]
 [2 8]]

 precision recall f1-score support

 0 0.80 0.80 0.80 10
 1 0.89 0.89 0.89 10

 accuracy 0.85 20
 macro avg 0.84 0.84 0.84 20
weighted avg 0.84 0.85 0.84 20

5-Fold CV Accuracy: 91.50% (+/- 2.12%)

Exercise

sawaal

Accuracy aur F1 score mein kya fark hai?

Accuracy = overall correct predictions ka percentage � total mein se kitne sahi the. Simple hai but imbalanced data pe misleading ho sakta hai.

F1 Score = Precision aur Recall ka harmonic mean � dono ka balance dikhata hai. Jab classes imbalanced ho ya dono metrics important ho tab F1 use karo.

Example: 100 mein se 90 negative aur 10 positive hain. Agar model sabko negative predict kare toh accuracy 90% hogi but F1 0 hoga kyunki ek bhi positive find nahi kiya!

Summary

  • Accuracy: Overall correctness � balanced data pe useful
  • Precision: Positive predictions ki quality � FP kam chahiye
  • Recall: Positives ko kitna catch kiya � FN kam chahiye
  • F1 Score: Precision + Recall ka balance
  • Confusion Matrix: Detailed breakdown of predictions
  • Cross Validation: Reliable evaluation � data ko folds mein baant ke test karo
  • Scenario ke hisaab sahi metric choose karo � one size fits all nahi hai