Lesson 05 � Intermediate
TRANSFORMERS
SE DHYAN DEO.
Attention mechanism se long sequences handle karna seekho � GPT, BERT, Claude sab ka foundation!
WHY: Transformers kyun padhna hai?
Transformers GPT, BERT, Claude � sab ka foundation hai! Ye attention mechanism use karke long sequences efficiently handle karte hain. Modern NLP ka sabse important architecture hai ye.
Transformers ek neural network architecture hai jo 2017 mein "Attention Is All You Need" paper mein introduce hua. Ye sequence-to-sequence tasks ke liye use hota hai.
Jab text generation, translation, summarization, ya koi bhi NLP task karna ho � transformers sabse best hain.
GPT, BERT, T5, Claude, LLaMA � sab modern LLMs transformers pe based hain.
Attention Mechanism
Attention mechanism ek soft search technique hai jo input sequence ke alag-alag parts ko different importance weights deta hai. Jab tum "The cat sat on the mat" padhte ho, toh attention mechanism "cat" ko zyada weight dega kyunki "sat" uspe depend karta hai.
Model ka focus mechanism � kis word pe zyada dhyan dena hai
Ek word dusre words ke saath kaise relate hota hai
Architecture jo input ko process karke output deta hai
Transformers ke popular variants jo different tasks ke liye use hote hain
Key Innovations
- Parallel Processing: RNNs ki tarah sequential nahi, parallel mein process hota hai
- Attention Mechanism: Har word ka poori sequence pe focus rehta hai
- Positional Encoding: Words ki order ko track karta hai
- Scalability: Bahut bade datasets pe efficiently train ho sakta hai
Self-Attention Deep Dive
Self-attention mein har word apne aap ko dusre words ke context mein dekhta hai. Ye three matrices use karta hai: Query (Q), Key (K), aur Value (V).
Current word kya dhundh raha hai
Har word ka identity/position
Har word ka actual content/information
Multi-Head Attention: Ek hi attention se kaam nahi chalega. Multiple attention heads use karte hain taaki model alag-alag relationships seekh sake.
Attention Code Example
# Simplified transformer attention
import numpy as np
def softmax(x):
e_x = np.exp(x - np.max(x))
return e_x / e_x.sum()
# Self-attention example
query = np.array([1, 0, 1])
key = np.array([1, 0, 0])
value = np.array([0, 1, 0])
# Attention score calculate karo
score = np.dot(query, key)
attention = softmax(np.array([score, score*0.5, score*0.8]))
output = attention * value
print(f"Attention output: {output}")Encoder-Decoder Architecture
Transformers ka architecture do main parts mein divide hota hai: Encoder aur Decoder.
Input sequence ko process karta hai, context vectors create karta hai. BERT sirf encoder use karta hai.
Output sequence generate karta hai, encoder ke output ka use karta hai. GPT sirf decoder use karta hai.
Dono encoder aur decoder use hota hai. Sequence-to-sequence tasks ke liye best.
BERT vs GPT
Dono transformers ke variants hain lekin alag-alag use cases ke liye design kiye gaye hain.
Bi-directional context, classification tasks ke liye best. Named Entity Recognition, Sentiment Analysis. Masked Language Modeling se train hota hai.
Uni-directional (left-to-right), text generation ke liye best. Question Answering, Summarization. Next word prediction se train hota hai.
# Using transformers library
from transformers import pipeline
# Sentiment analysis pipeline
classifier = pipeline("sentiment-analysis")
result = classifier("I love this product!")
print(result)
# Output: [{'label': 'POSITIVE', 'score': 0.9998}]
# Text generation pipeline
generator = pipeline("text-generation", model="gpt2")
output = generator("Transformers are", max_length=50)
print(output[0]['generated_text'])Quick check
Exercise
Self-attention kya hai✓ Apne words mein explain karo.
Sochlo: ek word baaki sabhi words ke saath apna relationship calculate karta hai.
Key Takeaways
- Transformers attention mechanism pe based hain jo parallel processing enable karta hai
- Self-attention se har word baaki sabhi words ke context mein dekhta hai
- Q, K, V matrices attention calculate karne mein use hoti hain
- BERT (encoder) classification ke liye, GPT (decoder) generation ke liye use hota hai
- Modern LLMs transformers ki modifications hain
Ab GPT Architecture par chalo � jahan aap GPT ka detailed structure seekhoge.