Lesson 116 — Advanced

Application Serving & Containerization

Notebook chal jaata hai, par user ko chahiye tez, stream hua jawab jo kisi bhi machine pe same chale. Is lesson mein hum FastAPI backend, server-sent events se streaming, timeouts aur disconnect handling, health/readiness endpoints, ek thin frontend, aur poore stack ko Docker Compose se ek command mein chalayenge.

Serving ka structure kyun matter karta hai

LLM apps ka serving ka khaas masla: request lambi hai. Model seconds leta hai, agent minutes tak. Isliye normal request-response ka template kaafi nahi — streaming chahiye, timeouts chahiye, aur disconnect pe token kharcha bhi band karna padta hai.

FastAPI backend — validated requests, typed responses

from fastapi import FastAPI
from pydantic import BaseModel, Field

app = FastAPI()

class ChatRequest(BaseModel):
    message: str = Field(min_length=1, max_length=4000)
    session_id: str

class ChatResponse(BaseModel):
    answer: str
    request_id: str
    prompt_version: str

@app.post("/chat", response_model=ChatResponse)
def chat(req: ChatRequest):
    rid = new_request_id()
    answer = rag_chain(req.message, request_id=rid)
    return ChatResponse(answer=answer, request_id=rid,
                        prompt_version=CFG.prompt_version)

• Pydantic request model se validation free — blank ya bahut lamba message pehle hi 422, model call waste nahi.
• Typed response se client ko contract pata hai — answer ke saath request_id aur prompt_version bhi wapas (debugging bond).
• Sync def wale endpoint ko threadpool mein chalana padta hai — blocking model call ke liye async wrapper dhyan se likho.
• App startup pe config load aur connectivity checks — galat config ke saath listen mat karo.

Streaming — server-sent events aur disconnects

from fastapi.responses import StreamingResponse
from fastapi import Request
import json, asyncio

@app.post("/chat/stream")
async def chat_stream(req: ChatRequest, request: Request):
    async def gen():
        try:
            async for tok in rag_chain.astream(req.message):
                if await request.is_disconnected():
                    log_disconnect(req.session_id)   # token kharcha roko
                    break
                yield "data: " + json.dumps({"tok": tok}) + "\n\n"
            yield "data: [DONE]\n\n"
        except asyncio.TimeoutError:
            yield "data: " + json.dumps({"error": "timeout"}) + "\n\n"

    return StreamingResponse(gen(), media_type="text/event-stream")

• SSE (server-sent events): server se ek-way stream — har token ek event, browser mein EventSource se seedha padha jaata hai.
• WebSocket se simple hai (koi two-way nahi chahiye) aur normal HTTP proxy se hokar chalta hai.
• Disconnect check zaroori: user ne tab band kar diya toh model call abort karo — warna tokens bekaar jalte rahein.
• Har stream ka total timeout rakho — agent infinite soch mein bhi request kabhi na kabhi khatam hogi.
• Client ko [DONE] ya error event bhejo — stream ke end ka explicit signal, guess nahi.

Health aur readiness endpoints

@app.get("/healthz")
def healthz():
    return {"ok": True}          # process zinda hai? bas

@app.get("/readyz")
def readyz():
    checks = {
        "config": CFG is not None,
        "vector_db": ping_vector_db(timeout=1.0),
        "provider": ping_provider(timeout=2.0),
        "prompt_registry": registry_ping(),
    }
    ready = all(checks.values())
    return JSONResponse(status_code=200 if ready else 503, content=checks)

• Health (liveness): process chal raha hai ya nahi — container ko restart karne ke liye.
• Readiness: dependencies (vector DB, model provider, registry) reachable hain ya nahi — traffic lene layak hai ya nahi.
• Readiness fail pe load balancer naya traffic rok deta hai, existing connections chalte rehte hain — draining isi se banti hai.
• Checks ke timeouts chhote rakho — health check hi lamba ho gaya toh orchestrator galat decide karega.
• Kabhi-kabhi deliberate "not ready" drill karo — monitoring kaam kar rahi hai ya nahi, isi se pata chalta hai.

Thin frontend — sirf render karo

import streamlit as st, requests

st.title("Ask the docs")
if msg := st.chat_input("Sawaal poochho"):
    with st.chat_message("user"):
        st.write(msg)
    with st.chat_message("assistant"):
        box = st.empty()
        text = ""
        with requests.post(BACKEND + "/chat/stream",
                           json={"message": msg, "session_id": SID},
                           stream=True) as r:
            for line in r.iter_lines():
                if line.startswith(b"data: ") and b"[DONE" not in line:
                    text += json.loads(line[6:])["tok"]
                    box.markdown(text)          # token-by-token render
        if "error" in text:
            st.error("Jawab nahi aaya — dobara koshish karo.")

• Frontend ka kaam sirf: input lo, stream render karo, errors dikhaao. Config, RAG, guardrails sab backend mein.
• Backend URL env se aaye — frontend image alag, backend alag, jagah badalne pe sirf env badlo.
• Timeouts aur guardrail blocks ko plain language mein dikhao: "ye sawaal scope se bahar hai" — stack trace kabhi nahi.
• Session ID frontend generate karta hai — conversation continuity ke liye, debugging ke liye bhi.

Docker images aur Compose stack

# Dockerfile (backend — frontend ka bhi same pattern)
# FROM python:slim
# WORKDIR /app
# COPY requirements.txt . && RUN pip install -r requirements.txt
# COPY . .
# CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0"]

# docker-compose.yaml
services:
  backend:
    build: ./backend
    environment:
      - PROVIDER_API_KEY   # runtime secret, image ke andar nahi
    ports: ["8000:8000"]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/healthz"]
      interval: 30s
  frontend:
    build: ./frontend
    environment:
      - BACKEND_URL=http://backend:8000
  vectordb:
    image: <tumhara vector store>
    volumes: ["vdb_data:/data"]
volumes:
  vdb_data:

• Ek command: docker compose up — poora stack (backend, frontend, vector DB) ek machine pe chal jaata hai.
• Service names hi DNS hain — frontend backend ko naam se dhoondhta hai, IP yaad rakhne ki zarurat nahi.
• Secrets runtime pe environment se aayein — image ke andar bake karna supply-chain risk hai; image public registry pe gayi toh key bhi gayi.
• Vector DB ka data volume mein — container restart pe index safe.
• Healthcheck compose mein hi define karo — unhealthy service pe stack ka baaki hissa bata paata hai.

Exercise

Common mistakes

• Frontend mein business logic daalna: UI mein bhi RAG code daal diya — har fix dobara deploy. Fix: frontend sirf render + backend calls, baaki sab backend mein.
• Streaming aur disconnect ka dhyan nahi: User tab band kar diya par backend token jalata raha. Fix: disconnect check + timeouts, per-request budget.
• Health endpoint ko readiness samajhna: Process zinda hai par provider down — deployment traffic bhejta raha. Fix: alag liveness aur readiness, readiness dependencies check kare.
• Secrets image mein bake karna: Image registry mein push, key sabke haath mein. Fix: secrets runtime env ya secret manager se — image ke andar kabhi nahi.

kitchen kit   -> container image (ek jaisi har jagah)
menu card     -> config.yaml
slip no.      -> request id
thali packing -> streaming (ek-ek item dikhta jaaye)
store room    -> volume (data restart pe safe)