Understanding Attention Mechanism in Transformers

Question 9. Which of the following best describes the concept of attention as used in transformer-based LLMs?

A mechanism that randomly selects important words in a sequence?
A mechanism that weighs the importance of different parts of the input sequence?
A filter that removes stop words from the input sequence?
Attention, a technique that extends the context window of the model.

Correct answer: A mechanism that weighs the importance of different parts of the input sequence. In transformer-based LLMs, attention assigns learned weights to different tokens in the input so the model can focus on the most relevant pieces of context when producing each output token. This is a dynamic, per-token weighting process — not a random selection, stop-word filter, or a context-window extender.

The image displays a question about the concept of "attention" in transformer-based LLMs, alongside an explanation describing it as a mechanism that weights the importance of different parts of the input sequence.

Why attention matters (concise)

Attention enables each output token to “look at” the entire input sequence and assign higher influence to relevant tokens for that particular prediction.
This allows transformers to capture long-range dependencies and contextual relationships without relying on recurrence.

Illustrative example Imagine the sentence: “The cat was chased by the dog that ran up the tree.” When predicting the next token after “chased”, the model gives more weight to tokens such as “cat”, “dog”, “chased”, and “ran” than to function words like “the” or “by”. Those higher weights indicate stronger relevance for the specific prediction. How attention is computed (high-level)

Each token position has Query (Q), Key (K), and Value (V) vectors (produced by learned linear projections).
For a given query, compute similarity scores with all keys (dot products).
Scale the scores (typically by sqrt(d_k)), then apply softmax to obtain attention weights.
Use the weights to compute a weighted sum of the values — that becomes the attention output for the query position.

Simplified pseudocode for scaled dot-product attention:

# simplified attention weight computation
# query, key, value are matrices of shape (seq_len, d_k)
scores = query @ key.T                          # (seq_len, seq_len)
scores = scores / (d_k ** 0.5)                  # scale by sqrt(d_k)
weights = softmax(scores, axis=-1)              # attention weights (softmax over keys for each query)
output = weights @ value                        # weighted sum of values

Quick reference table

Component	Purpose	Example / Note
Query (Q)	Represents current position (what we are looking for)	Derived from token embedding via learned projection
Key (K)	Represents each candidate context position (what each position offers)	Dot product with Q yields relevance score
Value (V)	The content used when combining information across positions	Weighted by attention weights to produce output
Multi-head attention	Runs several attention projections in parallel to capture varied relations	Concatenate heads then linearly project

Additional notes

Multi-head attention: multiple attention “heads” use different projections so the model can capture several relationship types simultaneously (syntax, coreference, positional signals, etc.).
Attention is learned during training — the model optimizes the projections that produce Q, K, V to maximize downstream performance.
Attention is not a stop-word remover nor a random selector: it’s a differentiable, data-driven weighting mechanism.

Attention is a learned, dynamic weighting mechanism that lets transformers focus on the most relevant tokens for each prediction — effectively a configurable spotlight over the input sequence.

Watch Video

Preparing Datasets for RAG Systems

Prompt Engineering for Text Summarization