What is Sparse Attention?

Author: Data Science in your pocket · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: What is Sparse Attention? · Channel: Data Science in your pocket · 05:19 · Watch on YouTube

Topics: Model inference and optimization, AI models

Summary

Sparse attention reduces the quadratic cost of standard self-attention by having each token attend only to a selected subset of tokens. A mask excludes other connections, so their scores become zero after softmax and need not be computed. The video describes local, strided, global-token, and Big Bird attention patterns, along with examples of models that use them. Sparse attention is useful for long inputs and tight memory budgets, while dense attention remains suitable for short sequences.

Key points

Standard self-attention compares every query with every key, producing an N by N score matrix and quadratic compute and memory costs.
Sparse attention allows each token to attend only to a chosen subset; a mask excludes disallowed connections.
With a sparse pattern of size K, the stated cost falls from order N squared to order N times K.
Local or sliding-window attention focuses on nearby tokens; the video says Mistral uses this pattern.
Strided or dilated attention connects tokens at fixed intervals; the video describes early GPT-3 as alternating strided and local patterns across layers.
Global tokens act as information hubs; Longformer combines them with a local sliding window.
Big Bird combines block-local windows, global tokens, and random connections.
Sparse attention can benefit long documents, long-context models, high-resolution images, genomics, and tight memory budgets.
Dense attention can still make sense for short sequences, where its full detail is affordable.

Chapters

Start	Chapter	Summary
00:00	The cost of dense attention	The video recaps self-attention and explains how its N by N score matrix makes compute and memory grow quadratically.
01:05	How sparse attention works	Sparse attention masks most connections and uses local, strided, global-token, and Big Bird patterns to reduce computation.
02:41	Patterns and model examples	The video describes global tokens and Big Bird, then names models that use or borrow sparse attention techniques.
04:15	Example and when to use it	A sentence illustrates the reduction in attention pairs, followed by guidance on when sparse or dense attention is suitable.

Quotes

Double the tokens and you quadruple the work. — 00:00

each token attends only to a chosen subset of tokens. — 01:36

Cost drops from order N squared to order N times K. — 02:09

the longer your input, the more sparse attention pays off. — 04:48

Concepts and entities

Sparse attention, Self-attention, Local attention, Strided attention, Global tokens, Mistral, Longformer, Big Bird, Gemma, GPT-3

Full transcript (original language)

00:00 Welcome to data science in your [music] pocket. Transformers power almost every modern AI model, but their attention layer has a hidden weakness. >> [music] >> Its cost grows quadratically with sequence length. Double the tokens and you quadruple the work. Today, we will see how sparse attention fixes this by letting every token look only at what actually matters. Quick recap. [music] In self-attention, every token creates three vectors, a query, a key, and a

00:32 value. [music] We compare each query with all keys using dot products, scale the scores by root D, and pass them through softmax. The result is a weighted [music] sum of the values. In short, every token looks at every other token. Here >> [music] >> is the catch. Within tokens, attention builds an N by [music] N matrix of scores. 1,000 tokens means 1 million pairs. 100,000 tokens means 10 billion pairs. Memory and compute both explode.

01:05 Long documents, [music] high-resolution images, entire code bases, plain attention simply cannot afford [music] them. But look at a real attention matrix. Most of it is almost zero. Nearby words carry most of the signal, plus a few important tokens like the subject of the sentence. It is like reading a book. You focus on the paragraph in front of you, not every page at once. Sparse attention makes this idea official. Instead of attending to

01:36 everything, >> [music] >> each token attends only to a chosen subset of tokens. All other connections are masked out before the softmax. The attention matrix becomes mostly empty, and we never even compute those entries. Here’s the maths. We compute scores S = [music] Q * K transpose over root DK. Then we add a mask matrix M. Allowed positions get zero. Disallowed positions get minus infinity. After softmax, minus

02:09 infinity [music] becomes exactly zero. So those tokens contribute nothing. Cost drops from order N [music] squared to order N times K. Pattern one, local or sliding window attention. Each token attends only [music] to its W nearest neighbors on each side. The attention matrix becomes a narrow band around the diagonal. This is exactly what Mistral uses. It is cheap, simple, and captures short-range structure beautifully. [music]

02:41 Pattern two, strided or dilated attention. Instead of neighbors, each token attends to [music] every STH token, jumping across the sequence in fixed steps. This gives long-range reach at a low cost. >> [music] >> The sparse transformer, used in early GPT-3, alternated strided and local patterns across layers. Pattern three, global tokens. A few special tokens, like the CLS token,

03:11 attend to everything and every other token attends back to them. They act as information hubs, gathering [music] and broadcasting context across the whole sequence. Longformer combines these global tokens with a local [music] sliding window. Pattern four, Big Bird. It mixes block local windows, a few global tokens, and a sprinkle of random connections. The [snorts] random links work like shortcuts in a road network, keeping every token within a few hops of every

03:43 other. Remarkably, Big Bird is provably a universal approximator >> [music] >> and even Turing complete with enough layers. Sparse attention [music] is everywhere. GPT-3 alternated dense >> [music] >> and strided sparse layers. Longformer and Big Bird read documents [music] thousands of tokens long. Mistral and Gemma use sliding window attention. And many modern long context models borrow these tricks [music] to stretch their context windows without

04:15 exploding their budgets. Let’s make it concrete. Take this sentence. The cat sat on the warm window sill enjoying the sunlight. 10 words. >> [music] >> Dense attention computes 100 pairs. With a window of two on each side, the word window sill attends only to its four closest neighbors, like the in warm plus one global token. That is about 34 pairs in total, [music] not 100. So, when should you go sparse? Sparse attention wins for long documents, long

04:48 context language models, high-resolution images, genomics, [music] and tight memory budgets. Dense attention still makes sense for short sequences, >> [music] >> where its full detail costs little and buys maximum accuracy. Rule of thumb, [music] the longer your input, the more sparse attention pays off. To recap, [music] attention is powerful but quadratic, and sparse attention fixes [music] this by letting each token attend only to a carefully chosen

05:19 subset. Same idea, a fraction of the cost. Thanks for watching and see you in the next video.