Lesson 147 — Advanced

Supervised Fine-Tuning (SFT)

SFT wo step hai jahan tumhara dataset actual weight updates mein badalta hai. Is lesson mein hum SFT kya change karta hai, kaunse hyperparameters pehle matter karte hain, full vs parameter-efficient training, ek pehla training run chalana, loss curves padhna, catastrophic forgetting pehchanna, aur validation pe checkpoint chunna dekhenge.

SFT kya change karta hai

SFT ka mechanism simple hai: example aaya → model ne jawab banaya → uske tokens ka loss nikaala → weights thoda adjust. Har example ke baad adjustment bahut chhota hota hai — isliye kuch sau examples ka asar dikhta hai par base model ki aadatein nahi udti. Par "bahut chhota" ka matlab "zero side-effect" nahi — agla section isi liye hai.

Hyperparameters jo pehle matter karte hain

# pehle in 5 knobs ko samjho, baaki defaults pe chhod do
cfg = {
    "learning_rate": 2e-5,   # chhota LR — weights ko nudge karna hai, rewrite nahi
    "epochs": 3,             # zyada epochs = overfit ka darr
    "batch_size": 8,         # per device
    "grad_accum": 4,         # effective batch = 8 * 4 = 32 (GPU chhoti ho toh)
    "max_seq_len": 1024,     # isse lambi example yahan truncate ho jaayegi
}

# decision: loss unstable hai -> LR chhota karo, LR badhane se nahi

• Learning rate: sabse sensitive knob. Curve tez-tez chhatka raha ya NaN aa gaya — LR kam karo. Chhota LR = slow par stable, zyada chhota = kuch nahi ho raha.
• Epochs: data ko kitni baar dobara dekha. Ek baar kam, teen-se-zyada overfit ka zone — val curve hi batayega.
• Batch size + gradient accumulation: chhoti GPU pe effective batch gradient accumulation se banao — stability bhi milti hai, memory bhi nahi badhti.
• Sequence length: cutoff se lambi examples kat jaati hain — truncated training data ka matlab hai aadhe jawab seekhna.
• Warmup, weight decay aur scheduler aate hain — par pehle in paanch pe confident ho jao, tabhi aage badho.

Full fine-tuning vs parameter-efficient

• Full fine-tuning: har param update hota hai. Poori capacity, par memory itni ki optimizer states alag hi bhaari ho jaate hain, aur har run ka apna poora weight copy chahiye.
• Parameter-efficient (adapters/LoRA): base frozen, chhoti trainable parts attach — memory aur storage dono kam, aur base ke saath swap karna aasaan.
• Kab full: bada dataset, bada budget, base ko hi nayi aadat daalni hai. Kabhi kabhi adapters ke bina target metrics nahi chhupte.
• Kab adapter: narrow behaviour task (hamara default), experiments chalane hain, ya ek base pe kai variations rakhni hain.
• Decision metric wahi hai: held-out score. "Adapter ne kaafi nahi kiya" ek valid experiment ka result hai, gunah nahi — tabhi full try karo.

FULL TUNE                     ADAPTER (e.g. LoRA)
har param update               chhote trainable matrices
memory: weights+grad+optimizer  base frozen -> memory kaafi kam
storage: poora model per run    adapter chhota, base share
swap: naya run = naya model     kai adapters ek base pe

rule: adapter se shuru, full tabhi jab adapter ki ceiling aa jaaye

Pehla SFT run — aur curves padhna

from trl import SFTTrainer, SFTConfig   # example training library

trainer = SFTTrainer(
    model="base/model-name",
    train_dataset=train_ds,
    eval_dataset=val_ds,
    args=SFTConfig(
        output_dir="runs/sft-01",
        learning_rate=2e-5,
        num_train_epochs=3,
        per_device_train_batch_size=8,
        eval_strategy="steps",
        save_strategy="steps",
    ),
)
trainer.train()
print(trainer.evaluate())    # validation loss — yahi asli number hai

reading the curves (illustrative shape, numbers tumhare honge):

epoch   train_loss   val_loss    reading
1       girta hai    girta hai   dono seekh rahe — aage badho
ditto   girta hai    girta hai   stable, aur improvement — theek
last    girta hai    chadh gaya  train neeche, val upar = OVERFIT
zero    bada hai     bada hai    dono nahi gir rahe = UNDERFIT
zigzag  unstable     unstable     LR zyada hai — chhota karo

• Underfit: dono curves flat aur high — ya data kam hai, ya model ko sikhne hi nahi diya (LR chhota, epochs kam).
• Overfit: train girta rahe, val upar chadh jaaye — model ne training data rat liya. Ruko, checkpoints compare karo.
• Unstable: loss chhatka raha hai — LR kam karo, batch stability do. Ghuma-phira kar wahi fix.
• Catastrophic forgetting: target task seekh gaya par general abilities khoyi — instruction-following, safety, normal formats. Ek chhote tune mein ye chhup jaata hai, isliye off-task prompts bhi test karo.

Outputs padho, sirf loss nahi

# har checkpoint pe blind sample lo — numbers se jhooth bhi milta hai
CHECKS = [
    "format parse hota hai?",        # JSON/schema strictly
    "tone consistent hai?",          # spec ke against
    "length theek hai?",             # over-verbose ya truncated
    "hallucination toh nahi?",       # facts se guess kar raha
    "base wali cheezein bhi hain?",  # general ability intact
]

def read_checkpoint(path, prompts, checks=CHECKS):
    for p in prompts:
        out = generate(path, p)
        log(path, p, out, [c for c in checks if fail(c, out)])

• Loss kam ho sakta hai par jawab worst — format ki aadat ban gayi ho toh loss low, quality mid. Isliye outputs ka manual review hi final judge hai.
• Har checkpoint pe wahi 10-15 fixed prompts chalao — same prompts, same order, taaki checkpoints comparable rahen.
• Ek chhota "regression pack" rakho: wo 10 general prompts jo base model achhe karta tha. Tune ke baad bhi waise hi chalne chahiye.
• Jo failure dikhe use label karo (format/tone/fact/general) — agli training round mein wahi dataset mein jaayega.

Exercise

Common mistakes

• Sirf training loss dekh kar khush hona: Loss low, jawab bekaar — format rakh liya par quality nahi. Fix: har checkpoint pe val metrics + sample outputs padho, blind 10 prompts ke saath.
• Epochs ko quality proxy samajhna: Zyada epochs = zyada seekha? Overfit mein train loss hi chhota rehta hai. Fix: val curve ka divergence dekho, checkpoint val se chuno.
• Ek saath bahut saare hyperparameters badalna: Kaunsa change kaam kar gaya pata hi nahi chalta. Fix: ek waqt ek knob, har run ki config save, small split pe compare.
• Off-task prompts test na karna: Target task behtar, par model ab normal multi-turn ya safety behavior bhool gaya — production mein surprise. Fix: regression pack har run pe chalao.

approved report (example) -> pattern dekho
meinat (forward pass)     -> jawab banao
correction (loss)         -> weights thoda adjust
test (validation)         -> ab khud likh ke dikhao
best copy (checkpoint)    -> jo sabse achha likhe — wahi rakhna