Attention Score Calculation | Linear Algebra for AI Engineers
Read the Portuguese translation →
Author: The Augmented · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Attention Score Calculation | Linear Algebra for AI Engineers · Channel: The Augmented · 00:32 · Watch on YouTube
Topics: AI models
Summary
Section titled “Summary”An attention score is calculated as the dot product of one token’s query and another token’s key. Learned matrices WQ and WK project token vectors into queries and keys, and higher dot products indicate stronger alignment. Computing Q times K transpose produces scores for every token pair; dividing by the square root of the head size and applying softmax turns each row into weights that sum to one. Attention lets tokens determine which other positions to draw information from.
Key points
Section titled “Key points”- Each token starts as a vector.
- The learned matrix WQ projects token vectors into queries.
- The learned matrix WK projects token vectors into keys.
- An individual attention score is the dot product of a query and a key.
- A high score indicates that the query and key are aligned.
- Q times K transpose computes scores for every pair of tokens at once.
- For three tokens, the scores form a 3×3 grid.
- Scores are divided by the square root of the head size before softmax.
- Softmax turns each row into weights that add up to one.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | Queries, keys, and pairwise scores | Token vectors are projected into queries and keys, whose dot product gives the attention score for a pair. |
| 00:32 | Score grid and softmax | The pairwise scores form a grid, are scaled by the square root of the head size, and are converted into row-wise weights. |
| 00:32 | What attention does | Attention lets tokens identify aligned positions from which to pull information. |
Quotes
Section titled “Quotes”An attention score is just a dot product between one token’s query and another token’s key. — 00:00
High value here means they’re aligned. — 00:00
softmax turns each row into weights that add up to one. — 00:32
Attention is how tokens decide who to pull information from. — 00:32
Concepts and entities
Section titled “Concepts and entities”attention score, query, key, dot product, WQ, WK, softmax, Transformer
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 An attention score is just a dot product between one token’s query and another token’s key. Every token starts as a vector. A learned matrix WQ moves each one into a query, which is the question this token asks. Another matrix WK moves each one into a key. What question can this token help answer? To score one pair, you take one token’s query and another token’s key and do the dot product. High value here means they’re aligned. This token has what the other one is looking for. Q at K transpose does every pair at once. Three tokens, a
00:32 3x3 grid of scores. Then divide by the square root of the head size and softmax turns each row into weights that add up to one. Transformers process every position in parallel. Attention is how tokens decide who to pull information from. An attention scores measure which tokens are aligned under these learned projections.