Lesson 148 — Advanced

LoRA & QLoRA

LoRA ne fine-tuning ko ek chhoti team ke liye possible banaya: base frozen rehta hai, chhote adapter matrices train hote hain. Is lesson mein hum adapter ka mechanism, QLoRA mein quantized base kaise madad karta hai, rank/alpha/target modules ke knobs, GPU memory ka hisaab, tooling aur run tracking, aur DIY vs managed ka decision dekhenge.

⏱ 20 min• Advanced• Supervised Fine-Tuning (SFT)

LoRA — base frozen, chhota adapter

Kya hai?

LoRA mein base model ke saare weights freeze kar diye jaate hain aur har targeted layer mein do chhoti low-rank matrices (A aur B) train hoti hain, jo original weights ke saath add hoti hain. Update chhota hai, behaviour poora badal sakta hai.

Kab kaam aata hai?

Jab memory/storage chhoti ho, experiment chal rahe hon, ya ek base pe kai behaviour versions chahiye. Narrow behaviour task ke liye yahi default option hai.

Kahan milta hai?

PEFT libraries (jaise peft) ke through training scripts mein, Axolotl/Unsloth jaisi tooling ke config files mein, aur serving engines jo adapters load kar sakte hain.

Mechanism: har weight update ko do chhote factors ke product mein represent kiya gaya — d×r aur r×d ki matrices, jahan r (rank) bahut chhota hai original dimension se. Isliye trainable parameters laakhon mein (millions ke bajaye), optimizer states bhi utne hi, aur storage sirf adapter jitni.

# LoRA ka math (mechanism level)
# original:  h = W x          (W frozen)
# LoRA:      h = W x + (B @ A) x * (alpha / r)
#            A: r x d_in,  B: d_out x r,  r << d

def adapter_params(r, d_in, d_out, layers):
    return layers * (r * d_in + r * d_out)   # sirf A aur B

print(adapter_params(16, 4096, 4096, 32) / 1e6, "M trainable (example math)")

Rank ka matlab capacity: r badhao = adapter zyada cheezein seekh sakta hai, par params aur memory badhte hain. Chhota task chhota rank; agar loss floor pe aa jaaye par quality adhuri ho, rank badhao — LR se zyada useful aksar yahi hota hai.

QLoRA — quantized base par wahi kaam

QLoRA ka idea: base model ko quantized (kam bytes per weight) form mein load karo — wahi frozen base, par memory kaafi kam. Adapter phir bhi normal precision mein train hota hai.
Kyun yaad rahta hai: training ke waqt base weights computation ke liye chahiye, gradient unke liye nahi. Base chhota ho toh GPU pe jagah milti hai.
Quantization type ek knob hai: kum bits = zyada saving par thoda precision loss — tune ke baad quality check isi liye zaroori hai.
Trade-off: QLoRA se training toh chhoti GPU pe ho jaati hai, par har backward pass mein de-quantize overhead hai — run thoda slow ho sakta hai.
Memory bacha nahi? Pehle sequence length aur batch dekho, phir precision, tabhi base quantization karo.

FULL TUNE          LORA              QLoRA
base: trainable    base: frozen      base: frozen + quantized
memory: sabse zyada beech mein       base pe sabse kam
storage: har run   adapter chhota    adapter chhota
quality ceiling:   high (task pe)    lagbhag LoRA, thoda quant cost

kab kaunsa:
  budget + chhota task   -> LoRA / QLoRA (default)
  base ko hi badalna ho -> full (tabhi, jab zarurat ho)

Ek sawaal interview mein aksar: Q: "LoRA mein gradients kiske liye bante hain?" A: base frozen hai, sirf adapter matrices ke liye — isliye optimizer states bhi sirf unke paas hain. Yahi memory saving ki asli wajah hai.

Rank, alpha, target modules aur baaki knobs

# config jo dobara chalao toh same result de (illustrative values)
base_model: <open-weights-model>
method: lora
r: 16               # rank — adapter ki capacity
alpha: 32           # scaling: effective strength = alpha / r
target_modules: [q_proj, v_proj]   # kis layers mein adapter lage
dropout: 0.05       # adapter par regularization
precision: bf16
quant: none         # QLoRA = quantized base (e.g. 4-bit)
epochs: 3
lr: 2e-5

r (rank): capacity ka dial. Chhota = sasta aur generalise; bada = zyada seekh sakta hai par overfit ka darr.
alpha: adapter ka scale factor — alpha/r se effective strength milta hai. Rakh kar r badhana ek classic trick hai (strength same, capacity zyada).
Target modules: attention projections (q/k/v/o) default; feed-forward bhi jodne se capacity badhti hai. Sab modules par lagana hamesha zaroori nahi.
Dropout aur precision: chhote dataset pe dropout thoda regularise karta hai; precision (bf16/fp16) stability se juda hai.
Ye saare knobs ek hi config file mein rakho — wandb/tensorboard jaisi logging tools mein run ke saath save hote hain (ya apna YAML manifest).

Tuning ka order: Pehle data quality (agle-agle lesson wali), phir LR/epochs, phir rank/target modules. Ganda data ko rank badha kar theek nahi karoge — knobs sirf usi cheez ko improve karte hain jo data mein hai.

GPU memory ka hisaab — pehle, rent se pehle

def tune_memory_gb(params, mode, bytes_per_param=2, seq=1024, batch=4):
    weights = params * bytes_per_param
    if mode == "full":
        opt = params * 8          # Adam: 2 moment states (fp32) + master copy
        grads = params * bytes_per_param
        core = weights + opt + grads
    else:                          # lora / qlora
        core = weights             # base frozen -> optimizer states nahi
    activations = batch * seq * 4096 * 24   # rough activation footprint
    return (core + activations) / 1e9

# params = 7e9 -> pehle ye number nikalo, phir GPU decide karo
# QLoRA: base 4-bit -> weights ka hissa aadha (approx, mechanism)

Weights: params × bytes per param. Precision kam = memory kam.
Full tune mein: gradients aur optimizer states (momentum, variance, aur aksar fp32 master copy) — ye weights se bhi bhaari ho sakte hain.
Adapter mode mein: optimizer states sirf adapter ke liye — isliye memory drop itna dramatic hota hai.
Activations: batch × sequence length se badhte hain — chhoti GPU pe pehle ye kam karo (batch 1, grad accum).
Formula se rough number nikalo, phir small dry-run chala ke actual GPU use dekho — estimate 20% galat ho toh chalega, 2x galat nahi.

Rent se pehle 10-minute ka math: Params × precision, phir mode (full/adapter), phir batch × seq. Ye teen line likh kar hi decide ho jaata hai ki 1 GPU chalega ya nahi — mehngi instance pe failed OOM run se sasta hai.

Runs track karna aur DIY vs managed

# har run ka manifest — tabhi experiments compare honge, yaad nahi
run-012:
  dataset: support-v3 (hash abcd1234)
  method: lora r=16 alpha=32 q,v
  epochs: 3, lr: 2e-5
  val: <score from your eval>
  outputs: samples/run-012/        # 10 blind prompts
  verdict: better-format, tone-same

# compare: run-012 vs run-009 -> kaunsa rakhna hai, ye faisla aasaan

Tooling ke options: Python libraries (TRL + PEFT) se direct control, ya config-driven tools (Axolotl, Unsloth) jahan ek YAML se poora run chalta hai — dobara chalana aasaan.
DIY: base model choice unlimited, weights tumhare paas, har knob tunable — par infra, debugging, serving khud sambhalna padta hai.
Managed service: kuch click mein training, ops vendor ke paas — par base model ki list limited, aur aksar weights bahar le jaana restricted ho sakta hai.
Decision rule: experiment chal rahe hain ya custom data flow → DIY; ek baar ka production tune, team mein GPU experience nahi → managed se shuru.
Jo bhi chuno — dataset version, config, eval results har run ke saath log karo. Do hafte baad yaad rakhne pe koi experiment nahi chalta.

Reproducibility = config + data: Ek doosra engineer tumhara run dobara chala kar wahi val score paaye — tabhi experiment asli hai. Config file, dataset hash aur command teeno repo mein hon; "mere laptop pe chala tha" experiment nahi kehlata.

Exercise

Question: LoRA mein base model ke weights kya rehte hain — frozen ya trainable?

Question: Adapter ki capacity aur params decide karne wala rank wala knob kya kehlata hai?

Question: QLoRA mein base model kis form mein load hota hai jisse memory milti hai?

Question: Full fine-tune mein optimizer states kiske liye bante hain — poore model ke liye ya sirf adapter ke liye?

Question: Adapter strength ka scaling knob jo rank ke saath mil kar effective strength deta hai — kaunsa hai?

Common mistakes

Bina memory estimate kiye GPU rent karna: OOM run se hi pehla paisa gaya. Fix: params × precision, mode (full/adapter), batch × seq — teen line ka estimate pehle.
Rank/alpha sab jagah badha dete rahna: Data kharab ho toh knobs bekaar — loss floor pe aayega par quality nahi. Fix: pehle data + LR, phir capacity ke knobs.
Har run ko yaad rakhne par chhod dena: Do hafte baad pata nahi kaunsi config kis data pe chali. Fix: har run ka YAML manifest — dataset hash, config, val score, samples.
Managed service ke baad weights bahar maangna end mein: Provider ka license tune ke weights rakhne deta hi nahi — late pata chalta hai. Fix: shuru mein hi terms padho, weights le jaane ka raasta ho tabhi depend karo.

Desi example: Purane kurte pe sirf pocket aur patti badalna

Tailor ko poora kurta nahi silna — base waise ka waise rehta hai, sirf nayi patti, pocket aur silai (adapter) badalni hai. Isse time, kapda aur paise bachte hain, aur chaaho toh doosre kurte pe wahi patti alag laga do:

base kurta (frozen weights)  -> wahi rehta hai
patti/pocket (A, B matrices) -> yahi train hoti hai
r = kitni chaudi patti       -> capacity
dry clean (QLoRA)            -> kapda chhota karke silai fit

Interview angle — yahan se sawaal aate hain

Q: LoRA kaise kaam karta hai? — Base freeze, har targeted layer mein do low-rank matrices (A, B) train hoti hain jo output mein add hoti hain.
Q: LoRA aur QLoRA ka farq? — QLoRA frozen base ko quantized form mein load karta hai, isliye wahi tune chhoti GPU pe baith jaata hai.
Q: Adapter size kaise estimate karoge? — Layers × r × (d_in + d_out) — sirf A aur B params, base ke muqable chhote se chhota.
Q: Alpha aur rank ka rishta? — Effective strength alpha/r hai — r badhao par alpha proportionally badha do toh strength same, capacity zyada.
Q: DIY aur managed fine-tuning mein kya dekhoge? — Base model choice, weights le jaane ki permission, knobs par control, aur team ke ops bandwidth.

Lesson complete?

LoRA/QLoRA clear — ab tum memory estimate karke adapter train kar sakte ho aur run ko reproducible rakh sakte ho. Agla lesson — Preference Tuning & DPO — mein dekhenge kaise "A behtar hai B" wala feedback training mein jaata hai.