> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Understanding Attention Mechanism in Transformers
> Explains attention in transformers, showing how queries keys and values compute learned weights so models focus on relevant tokens for each prediction.
Question 9.
Which of the following best describes the concept of attention as used in transformer-based LLMs?
* A mechanism that randomly selects important words in a sequence?
* A mechanism that weighs the importance of different parts of the input sequence?
* A filter that removes stop words from the input sequence?
* Attention, a technique that extends the context window of the model.
Correct answer: A mechanism that weighs the importance of different parts of the input sequence.
In transformer-based LLMs, attention assigns learned weights to different tokens in the input so the model can focus on the most relevant pieces of context when producing each output token. This is a dynamic, per-token weighting process — not a random selection, stop-word filter, or a context-window extender.
Why attention matters (concise)
* Attention enables each output token to “look at” the entire input sequence and assign higher influence to relevant tokens for that particular prediction.
* This allows transformers to capture long-range dependencies and contextual relationships without relying on recurrence.
Illustrative example
Imagine the sentence:
"The cat was chased by the dog that ran up the tree."
When predicting the next token after "chased", the model gives more weight to tokens such as "cat", "dog", "chased", and "ran" than to function words like "the" or "by". Those higher weights indicate stronger relevance for the specific prediction.
How attention is computed (high-level)
1. Each token position has Query (Q), Key (K), and Value (V) vectors (produced by learned linear projections).
2. For a given query, compute similarity scores with all keys (dot products).
3. Scale the scores (typically by sqrt(d\_k)), then apply softmax to obtain attention weights.
4. Use the weights to compute a weighted sum of the values — that becomes the attention output for the query position.
Simplified pseudocode for scaled dot-product attention:
```python theme={null}
# simplified attention weight computation
# query, key, value are matrices of shape (seq_len, d_k)
scores = query @ key.T # (seq_len, seq_len)
scores = scores / (d_k ** 0.5) # scale by sqrt(d_k)
weights = softmax(scores, axis=-1) # attention weights (softmax over keys for each query)
output = weights @ value # weighted sum of values
```
Quick reference table
| Component | Purpose | Example / Note |
| -------------------- | -------------------------------------------------------------------------- | --------------------------------------------------- |
| Query (Q) | Represents current position (what we are looking for) | Derived from token embedding via learned projection |
| Key (K) | Represents each candidate context position (what each position offers) | Dot product with Q yields relevance score |
| Value (V) | The content used when combining information across positions | Weighted by attention weights to produce output |
| Multi-head attention | Runs several attention projections in parallel to capture varied relations | Concatenate heads then linearly project |
Additional notes
* Multi-head attention: multiple attention "heads" use different projections so the model can capture several relationship types simultaneously (syntax, coreference, positional signals, etc.).
* Attention is learned during training — the model optimizes the projections that produce Q, K, V to maximize downstream performance.
* Attention is not a stop-word remover nor a random selector: it’s a differentiable, data-driven weighting mechanism.
Attention is a learned, dynamic weighting mechanism that lets transformers focus on the most relevant tokens for each prediction — effectively a configurable spotlight over the input sequence.
Further reading and references
* Attention Is All You Need (original Transformer paper): [https://arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762)
* Illustrated Transformer (visual explanation): [https://jalammar.github.io/illustrated-transformer/](https://jalammar.github.io/illustrated-transformer/)
* Transformer model family and docs: [https://huggingface.co/docs/transformers/](https://huggingface.co/docs/transformers/)