Lesson 21 — Beginner

Statistics & Probability for AI

Eval chalate waqt sabse aam galti ye hai — ek number dekha aur faisa suna diya. Is lesson mein tum Data → sample → summary stats → interval → decision wala raasta chaloge jismein har kadam pe error bars nazar mein hain: descriptive statistics jo latency, cost aur eval scores ko naapte hain, sampling jo batata hai ki test set sirf ek sample hai, confidence interval jo run-to-run variance ko dikhaata hai, hypothesis testing ka p-value intuition, aur wo multiple-comparisons jeb jo 20 variants chunne mein chhup jaati hai.

Statistics — gut feel ka alternative

Funda ye hai: har number jo tum decision mein use karte ho, wo ek sample se aaya hai aur usmein ghalat-fahmi (uncertainty) hai. Statistics ka kaam us ghalat-fahmi ko chhupana nahi, usko saath le kar chalna hai. Chain hamesha wahi rahegi: data → sample → summary stats → interval → decision, aur har step pe spread nazar mein.

Descriptive statistics jo actually matter karte hain

# latency data — 9 requests normal, 1 request slow (tail)
lat = [118, 121, 119, 124, 120, 122, 119, 123, 121, 900]   # ms

mean   = sum(lat) / len(lat)        # 1987 / 10 = 198.7  -> outlier ne duba diya
srt    = sorted(lat)
median = (srt[4] + srt[5]) / 2       # (121 + 121) / 2 = 121.0 -> roz ka asli feel
p90    = srt[8]                     # 124 se upar sirf 1 request -> tail dikh gayi
slow   = [x for x in lat if x > 300]  # [900] -> tail count, 10%% of traffic

• Mean: sab ka average. Total/capacity planning ke liye sahi (poore din ke requests), par ek bada outlier ise poora hila deta hai.
• Median: beech wali value. "Typical" dikhata hai — skewed data (latency, cost per request, income jaise) pe mean se zyada sach bolta hai.
• Variance / standard deviation: values kitni bikhri hui hain. SD chhoti = consistent service; SD badi = same average, par har request pe darr.
• Percentiles (p50, p95, p99): tail dikhate hain. p95 ka matlab 5% requests itni slow hain — SLA, alerts aur user pain isi se naapa jaata hai.
• Eval scores pe bhi wahi: accuracy ka mean alag-alag categories mein alag ho sakta hai — ek average se category-level failure chhup jaata hai.

Distributions & sampling — test set ek sample hai

Tumhara test set population ka chhota tukda hai — 500 examples jinhe tumne "world" maan liya. Isliye uska score estimate hai, fact nahi. Sampling error us estimate ki ghalat-fahmi hai, aur ye n badhne par 1/sqrt(n) ke rate se gharta hai — matlab samples 4 guna karne se error sirf aadhi hoti hai, dugni nahi.

from math import sqrt

# sampling error (standard error) ka basic form
def se(sd, n):
    return sd / sqrt(n)

print(se(10, 100))   # 10 / 10 = 1.00
print(se(10, 400))   # 10 / 20 = 0.50   -> 4x samples, aadhi uncertainty

# distribution types aur kahan:
# normal    : averages ke liye (daily mean latency across days)
# binomial  : success/fail outcomes (spam ya not-spam, click ya nahi)
# long-tail : latency, cost, traffic — mean/SD bekaar, percentile chahiye

• Normal distribution: chhoti-chhoti random effects ka jhund jab milta hai toh beech mein ghoomta hai (bell shape). Means aur intervals yahin se aate hain.
• Binomial distribution: har trial ya toh success ya fail — spam classification isi tarah ke 500 independent trials hai. Iska mean = p, spread p(1-p) se juda hai.
• Long-tailed distributions: latency, cost per request, tweet viral hona — yahan average aur SD jhooth bolte hain. Percentiles aur max hi asli picture dete hain.
• Sampling error chhota hota hai, bias nahi: galat test set (biased examples) chahe 10,000 ka ho, tab bhi galat hi rahega. n se precision milti hai, sahi-sahi nahi.

Confidence intervals aur run-to-run variance

# same eval, same prompt — phir bhi 5 runs alag kyon? sampling params
scores = [0.71, 0.74, 0.72, 0.73, 0.70]     # temperature/seed/example sampling

m  = sum(scores) / 5                        # 0.72
sd = 0.0158                                 # runs ki spread (variance = 0.00025)
se = sd / (5 ** 0.5)                        # 0.0158 / 2.236 = 0.0071
lo, hi = m - 1.96 * se, m + 1.96 * se       # 0.72 +/- 0.014
print(round(lo, 3), round(hi, 3))           # (0.706, 0.734)

# ab do runs compare: 0.72 vs 0.73 dono isi interval mein -> SIGNAL/NOISE?

• Confidence interval kya hai: ek line jismein true value ke rehne ka zyada-tar jata hai (95% interval = 1.96 × SE dono taraf). Iska matlab hai "score 0.72 hai" nahi, "score 0.706 se 0.734 ke aas paas hai".
• Run-to-run variance kyun: eval set ka sample kaunsa aaya, temperature/seed, aur kabhi-kabhi GPU kernels bhi — identical config pe bhi numbers thoda hilte hain.
• Overlapping intervals: do runs ke intervals overlap karein toh farq conclude nahi karoge. Interval chhota chahiye toh runs badhao, examples badhao.
• Reporting pattern: "mean 0.72 ± 0.015 over 5 runs" — isi tarah se tumhara number defendable banta hai, ek screenshot nahi.

Hypothesis testing — p-value ka seedha intuition

A/B test ka frame (naya prompt vs purana):

null hypothesis H0 : dono barabar hain (naya prompt kuch nahi badla)
observation        : 100 items pe A ne 68 sahi, B ne 64 sahi
p-value            : agar H0 sach hai, itna gap kitni baar milti?
p chhota (< 0.05)  : kam milta -> H0 ke against evidence hai
p bada             : fark ka evidence nahi -> "no evidence", proof of no effect nahi

multiple-comparisons trap (20 variants try kiye):
  har variant pe 5%% false-positive rate -> 20 * 0.05 = ~1 chandi ka variant
  fix: variants pehle define karo, best chun kar ALAG confirmation set pe ek baar test

• Null hypothesis: pehle se assume karo ki kuch nahi badla hai. Phir dekho data us assumption ke against kitna bhaari hai.
• p-value: null sahi maankar aisa data milne ki probability. Chhota p matlab "ye hona normal ke against tha" — ye galat hone ki probability nahi hai.
• Threshold ek convention hai: 0.05 ek mashhoor line hai, koi kanoon nahi. Effect ka size aur interval bhi saath dekho — sirf p se faisla nahi.
• A/B tests mein paired comparison: same examples pe do versions chalao (A/B), phir per-example farq naapo — examples ki difficulty ka shor khud cancel ho jaata hai.
• Multiple comparisons: jitne zyada variants, utni zyada chance ki koi toh luck se jeet jaaye. Twenty variants mein se best chunna ek statistical trap hai, skill nahi.

Real AI work mein statistics kahan ghusta hai

• Evals: har score ke saath interval aur runs ki spread. Regression tab mana jaata hai jab baseline ke interval se bahar gir jaaye.
• Prompt A/Bs: fixed example set, paired comparison, significance check — "2 prompt chalaye, dono theek lage" wali judgement nahi.
• Capacity planning: mean traffic pe capacity nahi, p95/p99 peaks pe. Andaza percentile se lo, average se nahi.
• Monitoring thresholds: alert sirf mean pe toh kabhi trigger hi nahi hoga. Spread-aware rules likho — rolling mean + 3 × SD jaisa pattern, taaki ek badhla hua tail turant dikhe.
• Drift checks: kal ka score aaj ke mean ± interval se bahar gaya = data ya model badla. Yahi concept drift ka sabse sasta signal hai.

alert rules — spread aware (style, tumhare numbers honge)

p95_latency > SLO  for 3 consecutive windows   -> tail alerterror_rate  > rolling_mean + 3 * rolling_sd    -> z-score styleeval_score(run) < baseline_mean - 1.96 * baseline_se -> regression, shor nahi
category_p95_shift > baseline_p95 + 2 * baseline_sd  -> category level drift

yaad rakho: mean alert kabhi trigger nahi karta jabki p95 pe dikkat ho chuki hoti hai

Exercise

Common mistakes

• Sirf mean dekh kar faisa lena: Ek slow request ya ek mehnga outlier average ko poora badal deta hai, aur tail chhup jaati hai. Fix: median aur p95/p99 saath rakho, mean sirf total planning ke liye.
• Ek eval run ka number final maanna: Sampling params ki wajah se same config ke runs bhi hilti hain — ek number pe over-read karna bias hai. Fix: 3-5 runs chalao, mean ± sd likho, phir compare karo.
• p-value ko "galat hone ki probability" samajhna: p = 0.03 ka matlab null ke under data rare tha — model ka accuracy proof nahi. Fix: p ke saath effect size aur confidence interval bhi report karo.
• Ek hi test set pe 20 variants chun lena: Best variant chhunkar test set pe hi tune karoge toh luck bhi skill dikhega. Fix: variants pehle se likho, selection validation pe, final ek baar held-out test pe.

lat = [118, 121, 119, 124, 120, 122, 119, 123, 121, 900]
mean = sum(lat) / len(lat)        # 198.7 — party wale din ne duba diya
srt = sorted(lat)
median = (srt[4] + srt[5]) / 2   # 121.0 — asli roz ka bill
if any(x > 300 for x in lat):    # tail alert
    warn("aaj kuch alag hai!")   # mean alert kabhi trigger nahi hota