Lesson 157 — Intermediate

Latent Diffusion & Stable Diffusion Internals

Stable Diffusion-style pipeline ek assembly line hai: Text → conditioning vectors → noise in latent space → UNet denoises conditioned on text → VAE decodes to pixels. Is lesson mein har component ka kaam, uska failure mode aur har API knob uske upar map hoga. L12 wale multimodal level mein image-generation API se product banaya tha — yahan uska blueprint khulega, taaki tum "image blur kyun" jaise sawaalon ka jawab component ke naam se de sako.

⏱ 20 min• Intermediate• Diffusion Fundamentals: Learning to Denoise

Pipeline ka mental model — teen models, ek assembly line

Kya hai?

Poora flow: text encoder prompt ko conditioning vectors mein badalta hai, VAE image ko chhote latent grid mein (aur aakhir mein wapas pixels mein) jaata hai, UNet har step pe noise hatata hai — text ke kehne par — aur scheduler raasta tay karta hai. Har knob (steps, guidance, seed) isi kisi component pe jaata hai.

Kab kaam aata hai?

Jab tumhe API ke peeche ka mechanism chahiye — debugging ke liye: text image mein kyun nahi dikh raha, same result kyun nahi aata, slow kyun hai. Jitni acchi API usage aati hai, utna hi zaroori hai ki har param kaunsa part hilata hai.

Kahan milta hai?

diffusers ke pipeline components mein (vae, text_encoder, unet, scheduler), provider dashboards ke advanced params mein, aur design interviews mein jo poochhte hain — "guidance scale karta kya hai?"

Pehle teen naam clear karo: Stable Diffusion ek family/brand hai; pipeline uske components ka assembly hai; latent diffusion uska mechanism hai — pixels ki jagah compressed latent grid pe denoise karna. Inko mix kiya toh interview mein confusion turant dikhega.

Har knob ek component pe baithta hai: steps → scheduler ke timesteps; guidance → CFG mix; seed → initial latent noise; prompt → text encoder. Random guess band, mapping yaad rakho — debugging isi mapping se chalti hai.

Why latent space — denoising 48 guna chhote grid pe

# pixels vs latent — kitna chhota hota hai (hand-checkable arithmetic)
h, w, c = 512, 512, 3
pixels  = h * w * c                  # 786,432 values per image

lh, lw, lc = 64, 64, 4              # latent grid: 8x downscale, 4 channels
latent  = lh * lw * lc              # 16,384 values

print(pixels // latent)             # 48 -> denoise 48 guna chhote grid pe

VAE ka naya role: pehle lesson mein VAE generator tha; yahan uska kaam hai jagah banana — image → latent (encode), latent → image (decode). Denoising ka poora drama isi chhote grid pe chalta hai.
Kyun fayda: UNet har step chalta hai; grid chhota toh har step sasta. Ye koi magic nahi, seedha arithmetic hai — isi se latent diffusion mainstream bana.
Price — lossy compression: VAE fine texture aur chhote text jaisi cheezein encode/decode mein kho deta hai. Aksar dikhi wali "chhota text garbled" wali shikayat yahin se aati hai.
Resolution ka asar: latent grid badla toh denoising budget badla — 512 pe train model ko achanak 1024 dena layout tootne jaisa de sakta hai. Steps/resolution ek doosre ke saath chalte hain.

Latent space VAE ka hi hai: Yahi woh VAE hai jo lesson 1 mein seekha tha — ab bas uska kaam badla: generate karna nahi, denoising ke liye jagah banana. Lineage interview mein bolna: autoencoder → VAE → latent diffusion.

UNet denoiser — U-shape, time embedding aur cross-attention

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained("<koi bhi SD-style checkpoint>")
print(pipe.components)
# {..., "vae": ..., "text_encoder": ..., "tokenizer": ..., "unet": ...,
#  "scheduler": ..., ...}

# components ka ek-line kaam:
#   vae          -> image <-> latent (start mein encode, end mein decode)
#   text_encoder -> prompt -> conditioning vectors
#   unet         -> (latent_t, t, text) -> noise prediction, HAR step
#   scheduler    -> timesteps + coefficients + sampler rule

U-shape: encoder side latent ko chhota karke broad context pakdati hai, decoder side wapas — aur skip connections dono ko jodti hain kyunki fine detail beech mein dab jaati hai.
Time embedding: model ko t batana zaroori hai — step 5 pe layout set ho raha hai, step 45 pe texture. Same weights, alag behaviour isi se badalta hai.
Cross-attention: text conditioning yahin ghushti hai — latent queries text ke keys/values se har layer mein milti hain. Isliye prompt ki phrasing composition badalti hai.
Output kya hai: latent grid ke size ka ek noise map — pixels nahi. Isliye decode sirf aakhir mein hota hai, beech mein kabhi nahi.

Har step wahi UNet hai: 50 steps ka matlab 50 baar ek hi network forward (CFG ke saath do pass har step). Model alag nahi hota — sirf t aur latent badalte hain. Isliye steps, latency ka sabse bada determinant hain.

Text conditioning — CLIP-style vectors aur phrasing ka asar

Text encoder prompt ko tokens se embedding mein badalta hai — ye poori prompt ki ek "meaning vector" hai, keyword lookup nahi. Isliye shabd ka order, extra modifiers (style tokens) aur language dono composition ko badalte hain — model ne training mein isi tarah ke caption-image pairs dekhe hain.

Conditioning kya karti hai: cross-attention ke through har denoise step pe "kya banana hai" ka signal deti hai — ek baar nahi, poore raaste.
Negative prompt: guidance ke bina-prompt wale pass mein use hota hai — jise tum nahi chahte uska idea wahan daal dete ho (clutter, extra limbs, blur).
Guidance weak prompt ko nahi bachata: scale sirf conditioning ke influence ko badhata hai, text encoder ko nahi. Prompt hi kamzor ho toh words badlo, dial nahi.
Long prompts: kuch checkpoints truncate karte hain — aage ke shabd zyada weight le jaate hain. Ye checkpoint pe depend hai, verify karo docs se, yaad mat rakho.

Alignment training se aata hai: Encoder khud "beautiful image" nahi jaanta — wo caption-image pairs se aligned hua hai. Jab caption out-of-distribution ho (jargon, nayi language), conditioning weak ho jaati hai — bug nahi, coverage ki baat.

Guidance, samplers aur seed — teen dials jo output therte hain

# classifier-free guidance (CFG) — do predictions ka mix
eps_uncond = 0.20   # bina prompt ke (negative prompt ka idea yahin jodte ho)
eps_cond   = 0.50   # prompt ke saath
w          = 7.5    # guidance_scale — API knob

eps = eps_uncond + w * (eps_cond - eps_uncond)
# 0.20 + 7.5 * 0.30 = 2.45 -> prompt ki taraf kheencha, aur tez
print(round(eps, 2))   # 2.45

Guidance scale: kam = zyada freedom, variation zyada, prompt thoda ignore; zyada = prompt literal, par overshoot aur artifacts. Ye prompt-fidelity dial hai.
Seed: initial latent noise fix karta hai. Same seed + same everything = same image (deterministic sampler par).
Steps: scheduler ke timesteps set karta hai — refinement ka budget, UNet ka structure nahi.
Scheduler swap: alag timesteps pattern = alag refinement behaviour. Kam steps mein behtar detail ya alg aesthetic — scheduler hi wo knob hai.
Negative prompt: uncond pass ka content set karta hai — "kya nahi chahiye" wala half CFG.

Guidance zyada = stubbornness, pyaar nahi: Bahut zyada guidance par model prompt ke shabdon pe clingy ho jaata hai — composition tight, variation kam, artifacts. Pehle prompt aur negative prompt theek karo, phir dial ko last 10% adjustment rakho.

Har API knob ka component map — debugging list

API knob          component kya badalta hai             dikhta kya
----------------  -------------------------------------  --------------------
prompt            text_encoder ki conditioning          composition ka matlab
negative_prompt   uncond pass ka content                kya hatana chahiye
num_inference_... scheduler ke timesteps (UNet calls)   refinement + latency
guidance_scale    CFG mix w (do predictions)            prompt-fidelity dial
seed              initial latent noise                  reproducibility
scheduler name    schedule + reverse algorithm          steps ka behaviour
height/width      latent grid size                      detail + per-step cost

Quality badhani ho: pehle scheduler + steps (behtar wala combo), phir prompt/negative align, phir guidance — resolution sabse aakhir.
Latency ghatani ho: steps kam aur chhota resolution — CFG ko batch mein rehne do, guidance ko kam karne se do passes ka kharcha nahi bachta (mix waise bhi hota hai).
Repro chahiye: seed + steps + scheduler + checkpoint version — charon log, warna same seed pe bhi image badalti rahegi.
L12 wali API yaad karo: wahan params isi liye maange jaate the — ab har naam ke peeche ye component hai. Product lesson aur mechanism lesson isi jagah judte hain.

Debug checklist: Image blur? → VAE/decode ya guidance kam. Text nahi dikh raha? → conditioning ya phrasing. Same result nahi? → seed/scheduler/version. Slow? → steps/resolution. Har shikayat ek component pe jaati hai — andhere mein mat ghumo.

Exercise

Question: Image ko latent grid mein encode karne wala pipeline component ka naam? (ek shabd, chhota)

Question: UNet ko har step pe kis cheez ka pata hona chahiye — noise level ka embedding kya kehlata hai?

Question: eps_uncond 0.2, eps_cond 0.5, guidance 7.5 — CFG formula se eps kitna? (do number)

Question: 512x512x3 pixels aur 64x64x4 latent — kitne guna ka farq? Sirf number.

Question: Text conditioning UNet ke andar kis mechanism se judti hai? (do shabd)

Common mistakes

Latent diffusion ko chhota model samajhna: UNet wahi bada hai — bas grid chhota hai (48 guna chhota arithmetic). Fix: model size aur compute dono alag naapne wali baat hain; latent ne grid ghataaya, weights nahi.
Guidance scale ko prompt ka substitute banana: Conditioning weak hai toh dial badhane se overshoot aur artifacts aate hain, meaning nahi aati. Fix: pehle prompt/negative prompt, guidance ko fine-tune wala last dial rakho.
Seed pin karke scheduler/model version badal dena: Repro incomplete reh jaati hai — dobara image same nahi aayegi. Fix: seed + steps + scheduler + checkpoint, charon ek saath log karo.
Har blur ki shikayat UNet pe daal dena: Blur decode (VAE) ya guidance se bhi aata hai — sirf denoiser ko blame karne se fix galat jagah milega. Fix: encode/decode aur guidance dono ko blame list mein lao.

Desi example: Sketchbook aur canvas — pehle chhota, phir bada

Ek artist pehle rough sketch chhote notebook mein banata hai — sasta, tez, baar baar theek karta hai (latent grid + UNet). Jab sketch final ho jaata hai toh use bada canvas pe paint karta hai (VAE decode). Palette order deta hai (text conditioning), aur "aur dhyan se" wali instruction hi guidance scale hai:

notebook = start_from_noise()          # latent 64x64 — 48 guna chhota
for t in steps:                          # har step: sketch thodi aur saaf
    notebook = unet(notebook, t, palette) # palette = text conditioning
canvas = vae.decode(notebook)            # aakhir mein pixels (512x512)
# guidance = "palette ki baat mano" ka level — 7.5 pe artist literally le raha

Interview angle — yahan se sawaal aate hain

Q: Pipeline ke components aur ek-ek line ka kaam? — VAE encode/decode, text encoder conditioning, UNet har step noise predict, scheduler timesteps/coefficients, CFG do predictions ka mix.
Q: Guidance scale kya badalta hai? — Uncond aur cond predictions ka mix weight; structure (UNet) nahi, text encoder nahi — sirf prompt-fidelity dial.
Q: Latent space mein kyun denoise karte hain? — Grid lagbhag 48 guna chhota, har UNet step sasta; price VAE ki lossy compression hai (fine texture, chhota text).
Q: Reproducible image ke kitne factors chahiye? — Seed, steps, scheduler, checkpoint version, prompt — sab fixed toh same output (deterministic sampler par).
Q: "Prompt likha par text image mein nahi" — kahan dekhoge? — Conditioning path: tokenization/truncation, phrasing, guidance kam; decode ko aakhri blame do.

Lesson complete?

Har API knob ab tumhare liye component ban gaya — steps scheduler, guidance CFG mix, seed latent init. Agla lesson — Reinforcement Learning Foundations for LLMs — mein generation se language ki taraf: bandit se shuru karke RLHF loop tak.