Transformer Attention Visualizer
Visualize self-attention weights, step through attention heads, and explore Q/K/V matrices
Attention Heatmap
Head 1/4Transformer Block Diagram
Input Embedding
Multi-Head Attention
Add & Norm
Feed Forward
Add & Norm
Output
Input
0: The1: cat2: sat3: on4: the5: mat
Attention Heads
Q / K / V Matrices
Query (Q)
The:[0.0, 0.7, 0.4, 0.1]
cat:[0.3, 0.0, 0.7, 0.4]
sat:[0.6, 0.3, 0.0, 0.7]
on:[0.9, 0.6, 0.3, 0.0]
the:[0.2, 0.9, 0.6, 0.3]
mat:[0.5, 0.2, 0.9, 0.6]
Key (K)
The:[0.0, 0.3, 0.6, 0.9]
cat:[0.5, 0.8, 0.1, 0.4]
sat:[0.0, 0.3, 0.6, 0.9]
on:[0.5, 0.8, 0.1, 0.4]
the:[0.0, 0.3, 0.6, 0.9]
mat:[0.5, 0.8, 0.1, 0.4]
Value (V)
The:[0.0, 0.5, 0.0, 0.5]
cat:[0.7, 0.2, 0.7, 0.2]
sat:[0.4, 0.9, 0.4, 0.9]
on:[0.1, 0.6, 0.1, 0.6]
the:[0.8, 0.3, 0.8, 0.3]
mat:[0.5, 0.0, 0.5, 0.0]
How it works
Self-Attention: Each token computes a weighted sum over all other tokens based on relevance.
Q/K/V: Queries, Keys, and Values are linear projections used to compute attention scores.
Heads: Multiple attention heads capture different types of relationships in parallel.
Heatmap: Darker orange = higher attention weight between token pairs.