Lesson 165 — Advanced

Monitored Production API

Demo aur production ka farq yahi hai: model timeout kare, vendor down ho, bill uchhal jaye — tab bhi API gracefully handle honi chahiye. Yeh project ops story banata hai: routing, fallback, budget aur tracing.

⏱ 20 min• Advanced• MCP Server for a Real Tool

Project kya hai

WHAT

Ek AI endpoint jo tasks ke hisaab se model route karta hai, failure par fallback karta hai, per-user budget enforce karta hai, aur har request ko trace karta hai. Dashboard par p95 latency, errors, cost dikhte hain.

WHEN

Jahi bhi AI feature real users ke saath jaaye — chat endpoint, document pipeline, batch job. "Model call" tab tak code hai, monitoring ke saath service banta hai.

WHERE

FastAPI + middleware layer (ratelimit, tracing, budget) + ek metrics store (Prometheus/OTel) + Grafana/kloud dashboards. L10 pipeline ka mini version.

Deliverable sirf code nahi: ek SLO document (latency, error, budget targets) + dashboards + alert rules + ek runbook (alarm aane par kya karna hai).

SLO pehle: "99% requests <3s, fallback hit <10%, daily cost <$X" — targets likhe bina alerts ke kya judge karoge nahi pata chalega.

Routing aur fallback chain

def route(task: str) -> str:
    if task == "classify":   return "small-cheap-model"
    if task == "summarize":  return "mid-model"
    return "capable-model"          # default: complex reasoning

FALLBACKS = {
    "capable-model": ["mid-model", "small-cheap-model"],
    "mid-model": ["small-cheap-model"],
}

def generate(task, prompt, timeout=8):
    model = route(task)
    for m in [model] + FALLBACKS.get(model, []):
        try:
            return llm[m].generate(prompt, timeout=timeout)
        except (Timeout, RateLimit, UpstreamError) as e:
            metrics.increment("fallback.hit", tags={"from": m, "err": type(e).__name__})
    raise ServiceUnavailable("all models failed")   # dhoomila nahi, structured error

Fallback order decide: Kis order mein girenge (capable -> mid -> small)? Har level par quality gir jaati hai — response header se client ko batao ki kaunsa model use hua, quality expectations transparent rakho.

Cost caps

Production mein sabse tez shock bill hota hai. Do layer rakho: per-request (tokens in/out cap + output max_tokens) aur per-user/per-day budget (running total se upar = reject/queue).

def check_budget(user_id: str, est_cost: float) -> bool:
    used = redis.incrbyfloat(f"cost:{user_id}:{today()}", est_cost)
    if used > DAILY_USER_CAP:
        metrics.increment("budget.exceeded")
        return False                 # fail closed: mehnga ho toh roko
    return True

# Estimation: tokens * rate — exact billing aaye toh adjust karo (cron se reconcile)

Fail closed ya open: Critical user-facing feature = budget ke baad bhi degraded reply (chhota model); internal batch = fail closed. Yeh decision likh do README mein — interview mein poochha jaata hai.

Tracing

Har request ek trace: uske andar spans — prompt build, model call, post-process, DB. Yahi batata hai slow kahan hai aur error kis stage par aaya.

Span attributes: model name, tokens in/out, temperature, retry count, fallback depth, cache hit.
Trace ID response header mein — support ticket aaye toh seedha us request ka waterfall.
Sampling:100% nahi, pehle errors + slow +1% sampling — storage sane rakho.

AI golden signals: Latency, errors, traffic, saturation + ek naya: cost/day aur fallback hit rate. Fallback hit rate hi vendor health ka pehla alarm hai.

Dashboards aur alerts

Dashboard panels: p95 latency (per model), error rate, tokens/day, cost/day, fallback hits, budget rejections.
Alerts user pain par: p95 breach, error spike, budget exhaustion — CPU1% upar par nahi.
Runbook: har alert ke saath next5 minutes ka action (kya dekhna, kaise rollback — L10.10).
Weekly review: cost trend vs usage trend — dono badhein toh normal, cost badhe usage na badhe toh leak.

Alert hygiene: Jo alert har hafte silence hota hai use karo ya hatao — warna asli alarm bhi khoya reh jaata hai (alert fatigue, L10.9).

Exercise

Question: Primary model fail hone par automatically doosre model par jaana — ek shabd, English?

Question: Request ke har step ka timing-wise record karne wala term? (ek shabd, English)

Question: Monthly/daily spend limit jo dashboard par set karte ho — ek shabd?

Common mistakes

Sirf happy path deploy karna: Timeout aur vendor outage production mein guaranteed hain. Fallback + structured error bina ke pehla hi din incident banega.
Cost monitoring ke bina: Ek runaway loop ya bot traffic din bhaag gaya toh bill us raat hi aayega. Budget layer + cost/day graph mandatory hain.
Alert floods: Har warning par Slack ping toh team alerts ko mute kar deti hai. Sirf user pain wale alerts rakho, baaki dashboards mein.

Desi example: Generator aur inverter

Bijli jaate hi inverter apne aap lag jaata hai — ghar ki light band nahi hoti, aur tumhe pata bhi nahi chalta. Fallback wahi karta hai: primary model down toh secondary automatically pick hota hai:

chain = [primary, secondary_cheap]
for m in chain:
    try:
        return m.generate(prompt, timeout=8)
    except Timeout:
        metric.increment("fallback.hit")
raise ServiceUnavailable("chain down")

Interview angle — yahan se sawaal aate hain

Q: AI endpoint ka SLO kaise define karoge? (latency p95, error budget, fallback rate, cost/day — aur kyun wahi)
Q: Fallback chain kaise design ki jaati hai? (quality tiers + headers se model disclosure + retry policy)
Q: Cost spike hua toh pehle kya dekhoge? (cost vs traffic graph, per-user breakdown, token counts, cache misses — L10.9 ka debugging flow)
Q: Tracing kaise implement karoge? (OTel spans, attributes = model/tokens/retries, trace-id header, sampling policy)

Lesson complete?

Routing, budget aur tracing — teenon production ki real zaroorat hain. Ab ek fake outage inject karke fallback + alert ka flow khud test karo.