Lesson 38 — Beginner

Capstone — Streaming Tool-Using Assistant

Poore L2 ka kaam yahin judta hai: ek assistant jo stream karta hai, tools chalata hai, failures handle karta hai aur har run ko trace karta hai. Is capstone mein hum architecture, core loop ka skeleton, aur ship karne se pehle ka checklist dekhenge — code apne project mein paste kar ke shuru karo.

Capstone mein kya banega

Scope deliberately chhota rakho: do tools, ek provider, ek conversation state. Quality bar features ki sankhya nahi, integration ki saaf-suthri hai — har lesson ka concept iske andar dikhna chahiye.

Architecture — kya kahan baithta hai

user input
   |
   v
[conversation state] ---- messages[] (stateless API ke liye)
   |
   v
[assistant loop]
   |-- no tool -> [stream reply] ----> tokens -> UI (SSE/terminal)
   |-- tool?   -> [validate args] -> [execute tools] -> role "tool" result
   |                 |                     |
   |                 +-- error -> structured error message (loop zinda)
   |
[reliability layer] : timeout + backoff retry (wrap har API call)
[trace]             : per round -> model, tokens, latency, tool, ok/error

• State: messages list tumhara session store hai — app state (DB) alag; history trim policy bhi yahin decide hoti hai.
• Loop: tool_calls aayein toh execute + wapas bhejo; content aaye toh stream kar ke khatam. Chakkar tab tak jab tak final na mile (max rounds ka cap!).
• Reliability: har provider call call_with_retry se guzre; tool execution safe_run se — dono alag layers (M2.9).
• Trace: har round ka record — round number, model, tokens, tool names, latency, status — ek JSONL file mein.

Core loop ka skeleton

import json, time, logging

log = logging.getLogger("assistant")
TOOLS = [order_status_schema, faq_search_schema]     # M2.5/M2.6 wali schemas
MAX_ROUNDS = 5

def chat(user_text, history):
    history.append({"role": "user", "content": user_text})
    for round_no in range(MAX_ROUNDS):
        t0 = time.perf_counter()
        resp = call_with_retry(lambda **k: client.chat.completions.create(  # M2.9
            model=MODEL, messages=history, tools=TOOLS, **k))
        msg = resp.choices[0].message
        history.append(msg)

        if not msg.tool_calls:
            return msg.content          # agla step: isi ko stream karo (M2.2)

        for call in msg.tool_calls:     # safe_run -> errors bhi tool result
            history.append({"role": "tool", "tool_call_id": call.id,
                            "content": safe_run(call)})

        log.info("round", extra={"round": round_no, "tool": msg.tool_calls[0].function.name,
                 "tokens": resp.usage.total_tokens,
                 "ms": round((time.perf_counter() - t0) * 1000), "ok": True})
    return "Bahut zyada steps lag gaye — shuru se ek hi sawaal chhota karke batao."

def stream_final(history):              # M2.2 — final answer stream
    parts = []
    for chunk in client.chat.completions.create(model=MODEL, messages=history, stream=True):
        d = chunk.choices[0].delta.content
        if d:
            parts.append(d)
            yield d
    history.append({"role": "assistant", "content": "".join(parts)})

Ship checklist

• Streaming: final answer token-by-token aaye, partial on interrupt mark ho, flush sahi ho.
• Tool loop: dono tools chalte hain, tool_call_id match, malformed args par structured error (loop zinda), max rounds cap.
• Robustness: 429/5xx par backoff retry, auth error fail-fast, timeout set, fallback/canned message wired.
• State: nayi session history banti hai, history trim policy lagu hai, keys env/secret manager se aa rahi hain.
• Observability: har run JSONL trace — request_id, rounds, tokens, latency, tools, errors. Ek test run se failing step dikhe.
• Manual test script: 10 sample conversations (simple, tool wali, tool failure wali, overload wali) — har lesson ke exercises se inspire hoke.

Exercise

Common mistakes

• Sab kuch ek hi function mein likh dena: — state, loop, retry aur logging alag functions hain. Ek giant function mein bug fix karna nightmare hoga; lessons ke code ko modules mein paste karo.
• Loop cap na lagana: — model tool_calls ka cycle bana de toh assistant ghoomta hi rahega. MAX_ROUNDS cap + user ko saaf message do.
• Tracing ko last mein chhod dena: — pehle likhoge toh har test run debuggable hoga. Bahut der baad jodoge toh debugging ke purane runs ka data hi nahi hoga.
• Capstone mein naye features add karte jaana: — do tools se 6, RAG bhi daal diya. Scope chhota rakho — integration ki saaf-suthri pehle, features baad mein.

Order suno (user msg) -> menu check (tool choose) -> kitchen (execute)
Khana plate mein aate hi bolna (stream)
Kitchen busy? thoda ruk kar dobara (retry + backoff)
Shift end: khata — kya bikta raha, kya atka (trace)

Yehi tumhara capstone hai — bas waiter AI hai.