AI Engineering / MLOps / Monitoring & Observability

Monitoring & Observability

Apne AI models ko production mein real-time track karo

? 20 min Intermediate AI Engineering
Prerequisite: MLOps Fundamentals ka basic understanding zaroori hai.

Ye Lesson Kyu Important Hai?

Monitoring se pata chalta hai model sahi kaam kar raha hai ya nahi � drift, latency, errors sab track hote hain. Bina monitoring ke tumhe pata bhi nahi chalega ki model kharab ho raha hai!

LOGGING

Events record karna � kaunsa input aaya, kya output mila, kab hua. Har cheez ka trail rakhna.

METRICS

Numbers track karna � latency, throughput, accuracy rate, error rate. Quantitative data jo batata hai kya ho raha hai.

ALERTS

Auto notification jab kuch gadbad ho � threshold cross ho, error spike aaye, ya drift detect ho.

DRIFT

Training data aur production data ka difference. Agar drift zyada hai toh model ki performance gir jayegi.

Model Monitor Class

Ye code ek basic monitoring system banata hai jo predictions, latency aur errors track karta hai.

Python 
import time, logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("ai-model")

class ModelMonitor:
 def __init__(self):
 self.predictions = []
 self.latencies = []
 self.errors = 0
 
 def log_prediction(self, input_data, prediction, latency):
 self.predictions.append(prediction)
 self.latencies.append(latency)
 logger.info(f"Prediction: {prediction}, Latency: {latency:.2f}ms")
 
 def get_stats(self):
 return {
 "total": len(self.predictions),
 "avg_latency": sum(self.latencies)/len(self.latencies) if self.latencies else 0,
 "error_rate": self.errors/len(self.predictions) if self.predictions else 0
 }

monitor = ModelMonitor()
monitor.log_prediction("input", "positive", 45.2)
print(monitor.get_stats())
Expected Output: 
{'total': 1, 'avg_latency': 45.2, 'error_rate': 0}

Drift Detection Example

Data drift detect karne ka simple approach � statistical difference check karo training aur production data mein.

Python 
import numpy as np

class DriftDetector:
 def __init__(self, reference_data):
 self.ref_mean = np.mean(reference_data)
 self.ref_std = np.std(reference_data)
 self.threshold = 0.2 # 20% drift threshold
 
 def check_drift(self, new_data):
 new_mean = np.mean(new_data)
 drift = abs(new_mean - self.ref_mean) / self.ref_mean
 return {
 "drift_percentage": round(drift * 100, 2),
 "is_drifting": drift > self.threshold
 }

# Training data ka stats
train_data = np.random.normal(50, 10, 1000)
detector = DriftDetector(train_data)

# Production mein naya data aaya
prod_data = np.random.normal(55, 10, 100) # thoda shifted
print(detector.check_drift(prod_data))

Interactive Editor: Monitoring Setup

// Output yahan dikhega...

Key Takeaways

  • Logging se events ka trail rakhte hain � debugging mein help milti hai
  • Metrics quantitative data provide karte hain � latency, accuracy, throughput
  • Alerts automate karte hain problem detection � threshold based notifications
  • Drift detection zaruri hai � production data change hota rehta hai
  • Production mein har model ke liye monitoring setup karna mandatory hai

✓ Exercise

Model drift kya hai?

Next Lesson ?

Security & Governance

→