Lesson 5 � Intermediate

GPT ARCHITECTURE:
TEXT GENERATION KA KING.

GPT sabse popular LLM hai � GPT-3, GPT-4, ChatGPT sab GPT architecture pe based hain. Ye decoder-only transformer hai jo text generation ke liye best hai. Samajhna zaroori hai ki ye kaise kaam karta hai.

WHY: GPT kyun important hai?

GPT ne modern AI ko define kiya hai. ChatGPT, GitHub Copilot, DALL-E � sab GPT family se aaye hain. Agar tumhe LLMs samajhne hain toh GPT architecture samajhna zaroori hai. Ye decoder-only approach hai jo text generation ke liye sabse efficient hai.

WHAT: GPT kya hai?

GPT ek decoder-only transformer architecture hai jo auto-regressive language modeling se kaam karta hai. Iska matlab hai � ye left-to-right direction mein text generate karta hai. Har token ko previous tokens ke context mein dekhke predict karta hai.

# GPT ka kaam samjho
# 1. Input: "I love"
# 2. Model dekhta hai: ["I", "love"]
# 3. Predicts next token probabilities
# 4. "learning" ko sabse zyada weight milta hai
# 5. Output: "I love learning"

# Simple tokenization example
tokens = ["I", "love", "learning", "AI"]
token_ids = [15496, 2831, 4023, 15340] # GPT-2 token IDs
print(f"Tokens: {tokens}")
print(f"IDs: {token_ids}")

Architecture ka breakdown

GPT ka architecture transformer ka decoder hai with some modifications. Ye masked self-attention use karta hai taaki future tokens pe na dekh sake.

Pre-training: Kaise seekhta hai GPT

GPT ko sabse pehle bahut bade text corpus pe train kiya jaata hai using next-token prediction. Ye unsupervised learning hai � koi labels nahi chahiye, bas text hai.

# Pre-training concept (simplified)
# Training data: "The cat sat on the mat"

# Step 1: Input sequence
input_text = "The cat sat on"
target = "the" # next token

# Step 2: Model predicts probability distribution
# P(the | The cat sat on) = 0.35
# P(a | The cat sat on) = 0.25
# P(floor | The cat sat on) = 0.15
# ... (vocabulary mein baaki tokens)

# Step 3: Loss calculate karo (cross-entropy)
# Step 4: Backpropagation se weights update karo
# Step 5: Millions of examples pe repeat karo

Fine-tuning: Task adaptation

Pre-training ke baad GPT ko specific tasks ke liye fine-tune kar sakte ho. Ye supervised learning hai � labelled data use karke model ko task-specific banate hain.

• Zero-shot: Bina example ke seedha task solve karo. GPT-3 ne ye popular kiya � sirf instruction likho aur model kaam karega.
• Few-shot: 2-3 examples do prompt mein. Model pattern seekh lega aur baaki cases handle karega.
• Fine-tuning: Labeled data pe additional training karo. Model weights update hote hain task ke according.

# Fine-tuning example (conceptual)
# Pre-trained GPT-2 model lo
# Apne custom data pe train karo

# Example: Sentiment classification fine-tune
# Input: "This movie is great"
# Label: POSITIVE

# Pre-trained model already samajhta hai:
# - "great" ka matlab positive hai
# - "movie" ek film hai
# Fine-tuning sirf classification head add karti hai

# HuggingFace se fine-tune karna:
# from transformers import GPT2LMHeadModel
# model = GPT2LMHeadModel.from_pretrained("gpt2")
# ... training loop ...

GPT Evolution: Version by version

Temperature: Creativity control

Temperature parameter se control hota hai ki model kitna random ya deterministic hona chahiye. Ye logits pe apply hota hai before softmax.

# Temperature effect on output
# Low temperature (0.1-0.3): Deterministic
# - Same input pe same output
# - Factual tasks ke liye best
# - "The capital of France is Paris"

# Medium temperature (0.5-0.7): Balanced
# - Thoda variation hai
# - Creative writing ke liye good
# - "The old man sat on the bench, watching..."

# High temperature (0.8-1.2): Creative/Random
# - Zyada variety
# - Artistic tasks ke liye
# - "Quantum elephants dance on moonbeams..."

# API call mein temperature set karna:
# response = client.chat.completions.create(
# model="gpt-4",
# temperature=0.7, # yahan set karo
# messages=[...]
# )

OpenAI API: GPT use karna

OpenAI ka API use karke tum GPT models ko directly access kar sakte ho. Chat completions endpoint sabse popular hai.

# OpenAI API example
import openai

# client = openai.OpenAI(api_key="your-key")

# Basic chat completion
# response = client.chat.completions.create(
# model="gpt-3.5-turbo",
# messages=[
# {"role": "system", "content": "You are helpful."},
# {"role": "user", "content": "Explain GPT in simple words"}
# ]
# )
# print(response.choices[0].message.content)

# GPT models ka comparison
models = {
 "GPT-3.5-turbo": {"params": "~175B", "speed": "Fast"},
 "GPT-4": {"params": "Unknown", "speed": "Medium"},
 "GPT-4-turbo": {"params": "Unknown", "speed": "Fast"},
 "GPT-4o": {"params": "Unknown", "speed": "Fastest"},
}
for name, info in models.items():
 print(f"{name}: {info['speed']}")

Try it: Tokenization simulator

Neeche apna text likho aur dekho ki GPT kaise tokenize karega. Real GPT tokenizer complex hai, but ye basic idea deta hai.

Key Takeaways

• Decoder-Only: GPT sirf transformer ka decoder use karta hai � encoder nahi. Isliye ye text generation ke liye best hai.
• Next Token Prediction: GPT ka core kaam hai � har token ko previous context se predict karna. Left-to-right direction.
• Pre-training + Fine-tuning: Pehle bade corpus pe train, phir specific tasks ke liye adapt. Ye two-stage approach powerful hai.
• Scale: GPT-1 (117M) se GPT-4 (unknown but huge) tak � parameters aur data dono badhe hain. Scale hi capability badhata hai.
• Temperature: Output ki randomness control karne ka tool. Low = deterministic, High = creative.