Lesson 118 — Advanced

Deployment on AWS

CI ne image bana di, ab usse cloud pe chalana hai — bina keys ko machines pe bikhraaye, aur bina streaming ko load balancer kaat de. Is lesson mein hum EC2 + IAM role, Secrets Manager, CodeDeploy releases, blue/green ka choice, ALB timeouts, aur CPU ki jagah concurrency par scaling dekhenge.

AWS pe deploy — structure

Pehla principle: machine ko kuch yaad nahi, key uske paas kuch nahi. Har instance apni zaroorat ka access role se leta hai, secrets startup pe Secret Manager se aate hain. Iska fayda — rotation aur revocation bina kisi machine ko chhuye mumkin hai.

EC2, IAM role aur Secrets Manager

instance startup (user data, conceptual):
  apt-get install docker
  aws ecr get-login | docker login     # IAM role se, key file nahi
  docker pull registry/app:${TAG}
  docker run -d --restart unless-stopped \
    -e PROVIDER_API_KEY="$(aws secretsmanager \
       get-secret-value --secret-id prod/llm/key --query SecretString)" \
    registry/app:${TAG}

# na ~/.aws/credentials file, na hardcoded key — role hi identity hai

• IAM role: instance ko ek role do jo sirf zaroori cheezein allow kare — pull image, read specific secret. Access keys rakhna hi nahi.
• Secrets Manager: provider keys ka ek hi ghar, rotation support ke saath. Container start pe fetch, process memory mein, disk pe nahi.
• User data: startup script instance nayi ho toh stack khud setup kare — "snowflake instance" koi na ho.
• Tags lagao (app, env, owner) — inventory aur cost dono baad mein isi se bante hain.
• Vector DB data volumes (EBS) alag rakho — instance replace hone pe index safe.

CodeDeploy — in-place vs blue/green

IN-PLACE:
old instance -> naya image pull -> restart -> health check
  + kharcha kam (same instances)
  - restart ke dauran gap, rollback = purana image wapas (slow)

BLUE/GREEN:
blue (old) live
green (new) stack boot -> health check -> shift traffic -> blue retire
  + zero-downtime, rollback = traffic wapas blue (seconds)
  - do stack ka kharcha chhoti der ke liye

LLM app default: blue/green — streaming app pe restart gap dikhta hai

• CodeDeploy tagged image pull karta hai, naya stack boot karta hai, health checks pass hone pe traffic shift karta hai.
• Health check fail = release automatically abort aur rollback — "deploy kar diya, dekh lenge" system mein nahi hota.
• Blue/green ka rollback itna sasta hai ki log sach mein rollback karte hain — in-place mein log "adjust kar lete hain" bolte hain.
• Smoke test deploy ke baad: ek real request maar ke dekho (streaming sahi, guardrail sahi) — healthz ke alawa.
• Release ke saath deploy marker log karo — monitoring mein "kya badla, kab" dikhta hai.

Load balancer aur streaming timeouts

user -> ALB (HTTPS) -> target group (instances) :8000

ALB settings jo LLM apps ke liye matter karte hain:
  idle timeout        : stream ke beech gap se zyada (default se upar)
  target group health : /readyz (healthz nahi)
  deregistration      : drain — purane requests poori hone do

# galati: idle timeout chhota -> 30s+ ke stream beech mein kat jaate hain
# aur: websocket/SSE ke liye ALB buffering off / compatible settings

• ALB ka idle timeout stream ke beech ka gap hai — LLM "thinking" gap kabhi kabhi seconds leta hai, default timeout aksar kam padta hai.
• Target health readiness endpoint pe hona chahiye — instance tabhi traffic le jab dependencies ready hon.
• Deregistration delay = graceful drain: deploy ke waqt existing streams poori ho jaayein, beech mein na katein.
• HTTPS termination ALB pe — certificates ka management ek jagah.
• 5xx rate aur target response time ko pehle se dashboard pe laao — deploy ke waqt wahi signals decision dete hain.

Auto Scaling — CPU nahi, concurrency par

LLM app ka CPU profile:
  request aayi -> 2s compute -> 25s PROVIDER ka intezaar -> reply
  CPU utilization: aksar 5-15%  ->  CPU scaling: "sab shaant hai"

sahi signal: in-flight requests per instance (concurrency)
  concurrency target: e.g. N requests per instance
  -> N se zyada -> instance add
  -> derat ko traffic low -> remove

# aur: provider ke rate limits bhi ek cap hain — scaling se pehle unhe dekho

• LLM request ka zyada waqt model provider ke intezaar mein jaata hai — CPU nearly idle rehta hai, isliye CPU scaling galat faisla degi.
• In-flight concurrency sabse sahi load proxy hai: kitne requests ek saath pending hain.
• Target tracking policy banao: desired instances = total concurrency / per-instance capacity.
• Provider ke rate limits throughput ka asli cap hain — instances badhane se limit nahi badhti; limit-first sizing karo.
• Scaling ke saath cache bhi dekho: sabse tez scaling wala instance bhi cold vector DB connection pe fasega.

Exercise

Common mistakes

• Instance pe access keys rakhna: Key leak = compromise, rotation mein bhi machine chhuni chahiye. Fix: IAM role + Secrets Manager, long-lived keys zero.
• ALB default idle timeout pe streaming chalana: Lambi stream ya soch-ne wali gap pe connection kat jaati hai. Fix: idle timeout sabse lambi stream se zyada, drain delay set.
• CPU-based autoscaling: CPU 8% par hai par requests queue mein — provider ka intezaar CPU nahi khata. Fix: in-flight concurrency pe scaling policy.
• Health check liveness se karna: Provider down bhi instance "healthy" — traffic aata rehta hai, errors bhi. Fix: target group health readiness endpoint pe.

gaadi     -> instance (role se access, key nahi)
order slip-> secret manager se utha (startup)
new route-> blue/green (nayi gaadi, purani ready)
board     -> in-flight count (scaling signal)
phone limit-> provider rate limit (pehle wala cap)