Lesson 35 — Beginner

Robustness — Retries, Timeouts & Fallbacks

API hamesha nahi chalti — 429 aati hai, 500 aati hai, network kat jaata hai, kabhi-kabhi jawab hi tootta hai. Is lesson mein hum kaunse errors retry karne hain, kaunse fail-fast, exponential backoff + jitter, timeouts, fallback models aur reliability logic kahan likhni hai — sab Python code ke saath dekhenge.

Aam API failures aur pehla rule

401 / 403  auth        : KEY GALAT — retry MAT karo, fauran fail + alert
400 / 422  bad request : payload hi galat hai — retry se fix nahi hoga
429        rate limit  : transient — wait + backoff + retry
500/502/503 server     : transient — limited retry
timeout / network      : transient — retry (par naya timeout de kar)
context length error   : prompt hi bada hai — history trim karo, retry nahi
malformed output       : parse/repair step (M2.4), retry with fix instruction

Retry policy — backoff, jitter, max tries

import random, time

RETRYABLE = {429, 500, 502, 503, 504}
MAX_TRIES = 4

def call_with_retry(fn, timeout=20):
    for attempt in range(MAX_TRIES):
        try:
            return fn(timeout=timeout)
        except Exception as e:
            status = getattr(e, "status_code", None)
            if status not in RETRYABLE or attempt == MAX_TRIES - 1:
                raise                              # fail fast ya aakhri try
            wait = min(2 ** attempt, 8) + random.uniform(0, 1)   # backoff + jitter
            print(f"retry {attempt + 1} in {wait:.1f}s ({status})")
            time.sleep(wait)

• Exponential backoff — 1s, 2s, 4s… har baar doogna; server ko breathing space milta hai.
• Jitter — thoda random wait add karo; warna tumhare jaise saare clients ek hi second par dobara ping karenge (thundering herd).
• Max tries — hadd lagao. Infinite retry = user beinteha wait karta hai aur bill badhta rehta hai.
• Idempotency — refund/send jaise side-effect calls dobara chalana khatarnak ho sakta hai; retry sirf read/pure calls par ya unique key ke saath.

Timeouts aur fallback chain

def ask_with_fallback(messages):
    chain = [
        ("primary",  PRIMARY_MODEL, 15),
        ("fallback", FALLBACK_MODEL, 30),   # alag model/provider bhi ho sakta hai
    ]
    last_err = None
    for name, model, tmo in chain:
        try:
            return client.chat.completions.create(
                model=model, messages=messages, timeout=tmo, max_retries=0)
        except Exception as e:
            last_err = e
            print(f"{name} down: {type(e).__name__} — agla try")
    raise RuntimeError(f"poora chain fail: {last_err}")   # degradation last

• Timeout har call par — bina timeout ke hanging request tumhara worker packet kar ke baithi rehti hai. LLM calls ke liye thoda generous rakho, generation lamba ho sakta hai.
• Graceful degradation — fallback bhi fail toh: cached result, chhota canned response, ya "try again later" — app crash kabhi nahi.
• Fallback model alag quality ka ho sakta hai — style aur behaviour badal sakta hai; isliye prompt aur output contract dono tolerant rakho.
• Timeout budget socho: user 5s wait karega toh primary 3s + fallback 2s — poora chain usi mein sametna hai.

Reliability logic kahan likhni hai

Layer 1 — Provider SDK   : built-in max_retries, HTTP-level retry (basic)
Layer 2 — Orchestration  : chain-level retries, callbacks, framework fallback
Layer 3 — YOUR APP       : business rules — kaunsa error retry, idempotency,
                           degradation, alerts, user-facing message

Rule: har layer apne level ka error handle kare — ek jagah sab mat rakho.

• SDK retry aksar sirf connection/5xx level karta hai — tumhare business rules (401 fail-fast, side-effect guard) app layer mein hi honge.
• Tool-level failure ≠ LLM-level failure: tool ka error tool result banao (model adjust kare); API ka error poore request ka failure hai (degradation dikhao).
• Retries ko metrics se jodo — kitne retries, kitni baar fallback laga (agle lesson ke observability se).
• Retry log mein attempt number, status code aur wait time zaroor likho — warna "chalo baad mein chal gaya" wali mystery reh jaati hai.

Exercise

Common mistakes

• Har error par retry: — 400/401 retry se fix nahi hote; bas load aur bill badhta hai. Retryable status ka set banao, baaki fail-fast.
• Bina jitter ke fixed wait: — saare clients ek saath laut aate hain aur dobara stampede hoti hai. Exponential wait + random jitter dono chahiye.
• Timeout na lagana: — hanging request tumhare worker ko block rakhti hai, thread pool bhar jaata hai. Har call par explicit timeout do.
• Side-effect tool ko blindly retry karna: — refund dobara chal sakta hai, duplicate email ja sakta hai. Idempotency key ya confirmation check lagao.

try 1 : wait 1s (+0.4s jitter)   busy
try 2 : wait 2s (+0.7s jitter)   busy
try 3 : wait 4s (+0.2s jitter)   LAG GAYI

fallback: nahi lage -> SMS chhod do (canned message)
App kabhi crash nahi honi chahiye — user ko ek saaf line mile.