Lesson 7 � Intermediate

Message Queues & Pub-Sub

Message Queues systems ke beech async communication ka medium hai. Ye loosely coupled architecture banata hai jisme services independently kaam karti hain.

? 22 min✓ Intermediate✓ Microservices basics

Message Queue hoti kya hai?

WHAT

Message Queue ek intermediate storage hai jo producer aur consumer ke beech messages store karti hai. Producer message bhejta hai, queue store karti hai, consumer jab chahe le sakta hai. Sync nahi, async communication hai.

WHEN

Jab services loosely coupled honi chahiye. Jab heavy processing ko background mein karna ho (email bhejna, image resize). Jab traffic spikes ko handle karna ho � queue messages hold kar lega.

WHERE

Email sending, order processing, notification dispatch, log processing, event streaming � har jagah message queues use hoti hain.

# Without Message Queue (Synchronous)
Order Service ✓ Payment Service ✓ Email Service ✓ Notification Service
(4 services sequentially � slow, coupled)

# With Message Queue (Asynchronous)
Order Service ✓ Message Queue ✓ Payment Service
 ✓ Email Service 
 ✓ Notification Service
(3 services independently � fast, decoupled)

# Benefits:
✓ Decoupling: Services ko ek doosre ke baare mein nahi pata
✓ Scalability: Consumers badha sakte ho independently
✓ Fault Tolerance: Consumer down hai toh messages queue mein safe
✓ Load Leveling: Traffic spikes absorb ho jaate hain

Mental Model: Message Queue jaise post office. Tum letter (message) daalte ho mailbox mein (queue). Postman (consumer) jab chahe collect karega. Tumhe wait nahi karna ki postman abhi aayega ya nahi. Letter mailbox mein safe rahega.

Queue Types & Tools

# 1. Point-to-Point Queue
✓ Ek message, ek consumer
✓ Producer ✓ Queue ✓ Consumer

# 2. Pub-Sub (Publish-Subscribe)
✓ Ek message, multiple consumers
✓ Publisher ✓ Topic ✓ Subscriber 1, Subscriber 2, Subscriber 3

# Tools:

# RabbitMQ (Message Broker)
✓ Traditional message queue
✓ Routing, priority, acknowledgments
✓ Use: Order processing, task queues

# Apache Kafka (Event Streaming)
✓ Distributed event streaming platform
✓ Messages stored as logs (not deleted after consumption)
✓ Use: Real-time analytics, event sourcing, logging

# AWS SQS (Managed Queue)
✓ Fully managed by AWS
✓ Standard queue (at-least-once) + FIFO queue (exactly-once)
✓ Use: Decoupling AWS services

# Redis Pub/Sub
✓ Lightweight, in-memory
✓ Use: Real-time notifications, chat apps

Messaging Patterns

# 1. Work Queue (Task Queue)
✓ Heavy tasks distribute karna workers mein
✓ Producer: Email list (1000 emails)
✓ Workers: 5 email sending services
✓ Each worker picks 1 email, sends, picks next

# 2. Pub-Sub (Fanout)
✓ Same message multiple services ko jaata hai
✓ Order placed event:
 ✓ Inventory Service: Stock update
 ✓ Email Service: Confirmation email
 ✓ Analytics Service: Track order
 ✓ Notification Service: Push notification

# 3. Request-Reply
✓ Request bhejo, reply ka wait karo
✓ Correlation ID se request-reply match karo
✓ Use: RPC over message queue

# 4. Dead Letter Queue (DLQ)
✓ Failed messages yahan jaate hain
✓ Debugging ke liye � messages lost nahi hote
✓ Retry logic implement kar sakte ho DLQ se

Kafka Deep Dive

# Kafka Architecture

Producer ✓ Topic ✓ Partition 0 ✓ Consumer Group
 Partition 1 ✓ Consumer Group
 Partition 2 ✓ Consumer Group

# Key Concepts:
- Topic: Message category (e.g., "orders", "logs")
- Partition: Topic ka chunk (parallelism ke liye)
- Consumer Group: Multiple consumers ek topic consume karte hain
- Offset: Message position in partition

# Why Kafka is special:
✓ High throughput (millions of msgs/sec)
✓ Messages stored as logs (replayable)
✓ Horizontal scaling (add partitions)
✓ Exactly-once semantics

# Use Cases:
- Real-time streaming (user activity tracking)
- Log aggregation (all services ke logs ek jagah)
- Event sourcing (state changes track karna)
- CDC (Change Data Capture from databases)

Exercise

Question: Message Queue ka sabse bada benefit kya hai jo services ke beech dependency kam karta hai? (1-2 words)

Question: Messages ko store karne ka Kafka ka approach kya hai jo messages ko replayable banata hai? (1-2 words)

Common mistakes

No DLQ: Dead Letter Queue na rakhna � failed messages lost ho jaayenge. Hamesha DLQ setup karo.
Overusing queues: Har cheez mein message queue mat lagao. Simple synchronous calls sufficient hain jab low latency chahiye.
No idempotency: Consumer multiple baar message process kar sakta hai (at-least-once delivery). Idempotent operations likho.
Queue monitoring skip: Queue depth, consumer lag, throughput � ye sab monitor karo. Queue full ho jaaye toh system hang.

Lesson complete?

Message Queues samajh aa gayi✓ Ab Microservices Architecture seekhte hain � large systems ko chhote services mein kaise divide karte hain.