Lesson 129 — Advanced

Managed Building Blocks: Build or Buy

Har managed service ko "ya toh sab kharido ya sab khud banao" wali nazar se dekhna sabse bada mistake hai. Is lesson mein hum har layer alag se decide karna seekhenge — retrieval, agent runtime, guardrails, evaluation, fine-tuning — aur har candidate ko control, transparency, eval depth, cost aur portability pe judge karna.

⏱ 30 min• Advanced• Model Routers & LLM Gateways

Build vs buy — layer by layer

Kya hai?

Managed building block matlab platform ka ready component: knowledge base (ingestion + retrieval), agent/workflow runtime, guardrails, evaluation service, fine-tuning job — configuration se lag jaata hai, code se nahi.

Kab kaam aata hai?

Har architecture review mein — jab tum decide karo ki is layer ka code tum likhoge ya platform se loge. Ek layer ka decision doosri layer ko bind nahi karta.

Kahan milta hai?

Cloud AI consoles mein (knowledge bases, guardrail configs, eval dashboards, fine-tune jobs) aur unke SDK/APIs mein. Tumhara deployment in sab ka mix ho sakta hai.

Funda: buy the plumbing you cannot differentiate on, build the layer your product is judged by. Chunking strategy, prompt tone, evaluation criteria — ye tumhare product ki shaan hain. Ingestion ka scaling, GPU scheduling, rate limiting — inpe difference bana kar kuch milta nahi.

Decision unit = layer: Managed retrieval lena tumhe managed agents par commit nahi karta. Har layer ka apna decision, apna evaluation. Wholesale adoption ya wholesale rejection dono extreme hain — audit mein sabse mushkil defend bhi yahi dono hote hain.

Managed retrieval — knowledge base vs apna stack

managed knowledge base (common shape):
  data source connect -> ingestion (chunk/embed auto)
  retrieval config     -> top-k, filters, hybrid toggle
  integration          -> platform ka RAG endpoint
  billing              -> per retrieval / usage unit

apna stack (L4 wala):
  loaders -> splitter -> embedder -> vector DB -> retriever -> reranker
  (har knob tumhare haath mein)

Buy jab: corpus standard hai, chhoti team hai, chunking ki deep tuning ki zarurat nahi aur deadline paas hai. Config se hafton ka kaam bachta hai.
Build jab: custom chunking (structural/semantic), apna reranker aur evaluation depth chahiye — L4 mein jo banaya uska control managed service aksar nahi deti.
Cost at your corpus size: chhote corpus pe managed retrieval sasta padta hai; bade corpus pe per-unit charges vs apna vector DB ka fixed cost compare karo.
Pehla test: apne chhote golden set se shuru — managed aur apna stack dono ko usi pe chala kar recall compare karo, demo data pe nahi.

Control mapping: Poochho: chunk boundaries main set kar sakta hoon? Retriever tuning params hain? Apna eval set import ho jaayega? Teen jawab "haan" ho toh managed retrieval bhi apne stack jaisa chalega; "nahi" toh trade-off decision doc mein likho.

Managed agent aur workflow runtimes

# Managed agent runtime ka typical configuration surface
agent:
  name: doc-qa-agent
  model: "tumhara chosen model"
  tools:
    - type: platform-search   # platform ke built-in tools
    - type: custom-webhook    # tumhara endpoint
  memory: platform
  orchestration: platform     # steps, loops, handoffs platform ke
# tumhare paas rehte hain: prompts, tool definitions,
# guardrail rules aur evals

Assumptions dekho: runtime kya maan raha hai — tools ka format, memory ka scope, orchestration ka control. L5 wale agents mein tum loop ke har step pe the; managed runtime wahi le jaata hai.
Control question: kya main apna planner/loop override kar sakta hoon, ya platform ka orchestration hi contract hai?
Debug depth: failure pe har step ka trace milta hai ya sirf final answer? Production debugging isi par tikti hai.
Portability angle: agent definition platform ke config format mein baith gaya toh migration mushkil — prompts toh chalte hain, orchestration graph nahi.

Standard use case ho toh: Common workflow (support triage, doc QA) ho toh managed runtime kaafi — scaling platform ka kaam hai. Custom planner ya complex state chahiye toh apna orchestration rakho, runtime ke andar fit hone ki koshish mat karo.

Managed guardrails aur managed evaluation

# Managed guardrails ka config shape
filters:
  content_safety: on          # category-wise block/warn
  topics: [allowed list]      # bahar ka topic reject
  sensitive_data: redact      # PII pattern redaction
# position: request path mein, gateway/platform level pe
# apne business checks: app pe bhi chahiye (dono layer)

eval service ka shape:
  dataset upload -> run against model -> metrics report
  central question: ye metrics tumhare eval suite (L7) ke
  sawaal ka jawab dete hain?

Guardrails ka position: platform filter turant lagta hai (request path), par tumhare business rules (kya data bahar ja sakta hai, kaunsa action allow) app pe hi rehte hain. Dono layer chahiye.
Managed evaluation: platform ke ready metrics (relevance, safety scores) useful hain — par sawaal yahi hai ki kya ye tumhare L7 eval suite ke questions jawab dete hain?
Eval depth ke teen sawaal: kya apna custom dataset import hota hai? Kya thresholds set kar sakte ho? CI mein chal sakta hai? Teenon haan ho toh managed eval kaafi.
Redaction ka trade-off: managed redaction sasta aur fast, par domain-specific identifiers miss ho sakte hain — apne regex bhi chalate raho.

Eval rule: Platform dashboard "accuracy dikha raha hai" — wo unke metric hain. Tumhare apne golden set pe run karke dekho, tabhi confidence. Managed eval service ko apne eval suite ka runner maano, replacement nahi.

Managed fine-tuning aur distillation

Managed fine-tuning mein tum dataset upload karte ho, platform training job chalata hai, aur ek naya model deployment milta hai. Distillation mein bade teacher se chhote student ko sikhaya jaata hai. Do sawaal sabse pehle: kaunse base models adapt ho sakte hain, aur resulting weights kahan rehte hain — kya tum unhe le kar aa sakte ho?

Base model options: platform sirf apne supported base models par fine-tune allow karta hai — tumhara pasandida open model us list mein ho zaroori nahi.
Weights kahan rehte hain: kahin fine-tuned model platform ke account mein hi ek deployment hai, kahin export possible hai. Ye pehle poochh lo — baad mein migration mushkil hoti hai.
Training data terms: dataset platform pe ja raha hai — retention aur training-terms wahi document padho jo residency lesson mein seekha tha.
Kab socho: prompt engineering aur retrieval se kaam chal raha ho toh fine-tune mat karo; latency ya per-call cost pe consistent gain chahiye ho tabhi.

Portability test: Fine-tuned weights export nahi ho rahe toh tumne ek deployment lock-in liya hai jo app-level wrapper se bach nahi sakta. Decision doc mein ye line likho — "weights portable hain ya nahi" — baad mein yahi line matter karegi.

Decision framework — judge kis param pe karo

har candidate ke liye ye columns bharo (layer-wise):
  control        -> knobs jo chahiye wo milte hain?
  transparency   -> andar kya ho raha hai, dekh/padh sakte ho?
  eval depth     -> apna dataset, custom metrics, CI run?
  cost @ volume  -> apne corpus/traffic pe monthly number
  auditability   -> auditor ko explain kar paoge?
  portability    -> exit kitna mehnga hai? (portability cost)

portability cost ka rule:
  prompts            -> minutes mein move
  eval suite         -> mostly portable (tumhara data hai)
  knowledge base     -> re-ingest + re-tune (hafton)
  agent/guardrail config -> re-implement (sabse mehnga)

Layer by layer: retrieval buy kiya, agents apne, guardrails mixed, eval apna suite + platform runner — ye combination bilkul valid hai.
Control aur transparency: regulated domain mein "platform ne kyun block kiya" ka jawab chahiye — opaque filter ko defend karna mushkil hota hai.
Cost at your volume: pricing page ka example nahi — apna corpus size, apna traffic. Chhote scale pe sab sasta dikhta hai.
Portability cost pehle naapi: adoption se pehle likho ki exit par kya-kya bachega aur kya dobara banega.

Doc format: Ek table, har layer ka row: buy/build, judge parameters, exit cost, owner. Is table ko architecture review mein le jao — "humne sab managed le liya" ya "sab khud likha" se ye table kahin zyada defendable hai.

Jab buy karna honest call hota hai

Chhoti team: do-teen logon ki team ke liye managed retrieval + managed guardrails sahi hain — differentiation yahan nahi hai, shipping mein hai.
Standard use case: platform ka component already tumhara quality bar clear kar raha ho (apne eval se verify kiya ho) — phir rebuild ka justification nahi.
Compliance deadline: audit certification jo managed service ke paas pehle se hai, tumhe mahine chahiye hote — buying time bhi buying hai.
Non-core layer: tumhara product judged hota hai answer quality par, ingestion scaling par nahi — wahin buy karo.

Aur honest opposite bhi: jab tumhara competitive edge usi layer mein hai (custom ranking, proprietary corpus handling), ya managed component tumhara eval bar nahi chhoo raha — tab build karo, budget kam ho ya nahi.

Dil se sawaal: Ye decision capability ka nahi, priority ka hai. "Hum khud banayenge kyunki maza aata hai" — architecture review mein nahi chalega; "is layer pe differentiation hai, ispe nahi" — ye chalega.

Exercise

Question: Managed knowledge base mein ingestion, chunking aur embedding platform karta hai ya client ko khud karna padta hai?

Question: Build vs buy ka decision unit kya hona chahiye — poora system ya alag alag layer?

Question: Fine-tuned weights export nahi ho rahe — exit ke waqt isse kya kehte hain?

Question: Managed guardrails ke content filters kahan lagte hain — request path par ya sirf dashboard par?

Common mistakes

Wholesale adoption — ek liya toh sab le liye: Managed retrieval se managed agents tak sab le liya, control kahin bhi nahi. Fix: har layer ka alag decision, alag judge parameters, alag exit cost.
Managed eval ko apna eval samajhna: Platform dashboard ke metrics unke questions ke hain, tumhare golden set ke nahi. Fix: apna dataset import ya apna runner — dono na ho toh managed eval ko supplementary rakho.
Portability cost adoption ke baad poochna: Knowledge base le chuke hain, tab pata chala exit par sab re-ingest hoga. Fix: adoption se pehle exit column bharo — prompt minutes, config hafton mein move hota hai.
Jaldi mein fine-tune kar lena: Prompt + RAG abhi theek chal rahe the, fine-tune ne timeline aur complexity dono badha di. Fix: baseline pehle, fine-tune sirf tab jab per-call gain measured ho.

Desi example: Readymade suit vs tailor — layer by layer

Office ke liye suit chahiye: counter se ready size le lo (buy — standard), par jahan fitting matter kare wahi tailor ke paas jao (build — custom). Poora suit hi custom karwana har jagah zaroori nahi:

shirt (standard fit)  -> counter se buy (managed retrieval)
pant ki length        -> tailor se alter (custom guard rules)
blazer ki fitting     -> tailor (custom ranking/eval — core)
billing sab ek jagah  -> ek bill + tags (attribution)
rule: jo off-the-shelf fit ho wo buy, jo fit hi ho nahi wo build

Interview angle — yahan se sawaal aate hain

Q: Build vs buy kaise decide karoge? — Layer-wise: control, transparency, eval depth, cost at volume, auditability, portability cost.
Q: Managed knowledge base vs apna stack ka farq? — Control over chunking, retrieval tuning aur evaluation depth; cost corpus size pe depend karti hai.
Q: Managed eval service ko kaise judge karoge? — Kya apna dataset import hota hai, thresholds set ho sakte hain, CI mein run hota hai — apne golden set se compare.
Q: Weights export na ho toh? — Deployment lock-in; wrapper se bach nahi sakta, isliye adoption se pehle portability column bharna zaroori.
Q: Buy kab honest hai? — Chhoti team, standard use case, compliance deadline, ya component jo apna eval bar already clear kar raha hai.

Lesson complete?

Build-vs-buy ka framework clear — ab har layer ko alag se judge kar sakte ho aur exit cost pehle naap sakte ho. Agla lesson — Portability, Lock-in & Migration — mein dekhenge adopt karke bhi trapped kaise na bane jaaye aur workload ko kaise move kiya jaata hai.