Definition
Attention computes weighted combinations of values, where weights depend on how well queries match keys. Self-attention lets every token look at every other token in the sequence.
Multi-head attention runs several attention patterns in parallel to capture different relationships.
In simple terms
When reading a sentence, you glance back at the subject to interpret a pronoun. Attention is that selective looking, learned automatically.
Where you see it
- Transformers for translation, chat, and code.
- Vision transformers attending to image patches.
- Cross-attention in encoder-decoder and multimodal models.
How it works
1.Form Q, K, V
Project inputs into queries, keys, and values.
2.Score matches
Similarity between queries and keys becomes weights.
3.Weighted sum
Combine values using those weights.
Why it matters
- Attention unlocked transformers, and with them modern LLMs and multimodal AI.
Often confused
Attention means the model "understands" like a human.
It is a mathematical weighting scheme, powerful pattern matching, not conscious focus.