How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song · Channel: AI Engineer · 14:35 · Watch on YouTube

Topics: Model inference and optimization, AI models, Multimodal models

Summary

Olive Song describes how Minimax M3 was trained from scratch with native multimodality and Minimax Sparse Attention (MSA). MSA combines a lightweight index branch with block-level retrieval, preserving different token selections across GQA heads while improving efficiency for GPU memory access. The team found that introducing multimodal data from the start largely preserved text capability and produced more integrated visual-text attention than later-stage multimodal training. The talk also covers the model’s vision architecture, including the use of 3D attention across video frames and a four-frame trade-off between performance and efficiency.

Key points

Minimax M3 combines coding and agent capabilities, a one-million-token context window, and native multimodal training.
MSA uses a lightweight index branch to select relevant tokens for the main attention branch.
The index branch is trained with a KL loss to match the main branch’s attention scores.
Unlike a shared selection across heads, MSA keeps token selection different across KV heads to preserve GQA diversity.
Block-level retrieval reduces candidate counts and top-k overhead while supporting more hardware-friendly memory access.
The team designed efficient training and inference kernels to make sparse attention practical.
Training with multimodal data from scratch was found to largely preserve text capability under the same text-token budget.
Attention maps suggested that native multimodal training integrated text and visual information more naturally than CPT-style training.
The vision pipeline uses a ViT to produce visual tokens that are inserted into the language model input.
3D attention lets visual patches from different video frames interact, and four frames gave the best trade-off explored between performance and efficiency.

Chapters

Start	Chapter	Summary
00:01	Introducing Minimax M3	Olive Song introduces M3 and outlines its coding and agent capabilities, million-token context window, sparse attention, and native multimodal training.
02:13	How Minimax Sparse Attention works	The talk explains MSA’s index and main branches and why the team approaches sparse attention through algorithm and infrastructure co-design.
04:19	Adapting sparse attention for GQA and GPUs	Song describes issues with applying an existing sparse attention design to GQA and explains MSA’s per-head selection, block retrieval, and efficient kernels.
08:39	Native multimodal training	The presentation compares later-stage multimodal training strategies with training from scratch and discusses how native training affects text-vision integration.
11:52	Vision architecture and 3D attention	Song describes how visual tokens enter the language model and how 3D attention across frames improves visual modeling, with four frames as the explored efficiency trade-off.

Quotes

a simpler architecture can always lead to better scaling properties. — 02:44

we took an algorithm algorithm and infrastructure co-design approach. — 03:48

we also show that the design does not introduce a significant performance drop. — 08:08

native multimodal training from scratch. — 10:47

using four frames gave us the best best trade-off between performance and efficiency. — 14:03

Concepts and entities

MiniMax M3, MiniMax, Olive Song, Minimax Sparse Attention (MSA), DSA, GQA, ViT, atenção 3D

Full transcript (original language)

00:01 [music] >> Um all right. Um so, hi everyone. I My name is Olive. I work at Minimax on reinforcement learning research. And today, I’m going to talk a little bit about um our new model, Minimax M3, and how we trained it natively multimodal from step zero, and how we trained it with MSA, which is called uh

00:34 which is Minimax sparse attention. Um a little bit introduction of our team. Uh we are one of the few labs over the world that uh works on multimodalities, which means that we have published our um of course LLM models before. Um we worked on our speech models. We have our own video generation models, and we have our coding agent models. And lately, we just released our Minimax M3, which is open weight

01:08 uh with frontier performance. So, it is actually the first open weight model to combine the following three um frontier capabilities. The first one is what we call soda uh coding and agent capabilities. The model is able to do a lot of the uh agentic tasks that you imagine. Um it it can do automatic task decomposition. It can work in your coding agents. It can drop in for 90% of the coding works that you are currently doing. And the second pillar is that the model

01:42 has 1 million context window. Um so, which is very long that enable like unlocks pretty much base all basic user long user interactions um of like it handles full code bases. And we trained it with MSA Minimax sparse attention. So, it is nine times faster pre-fill and 15 times faster decode versus full attention. And the third one is that the the model was trained native multi-model from step zero.

02:13 Which means that from step from scratch, we combined visual understanding with um language generation. So, the model would understand both capabilities very well from the start. And uh so that we can build many more applications on that, say computer use agents, right? So, first I’ll briefly explain what sparse attention does. Our core design principle is that a simpler architecture can always lead to

02:44 better scaling properties. This is the principle we have followed throughout the entire design. So, our goal is to always pursue a cleaner and simpler architecture. The sparse attention consists of two branches. The first one is what we call the index branch. Which performs a lightweight full attention computation. It uses four query heads and one key head. And it is trained with a KL loss to match the attention scores of the main branch. The index branch is used to identify the

03:18 tokens with the highest attention scores. These selected tokens are then attended by by the main branch. In other words, the sparse branch does not attend to all of the tokens. But only to a subset of them. This subset is selected through the lightweight computation of the index branch. Then, the sparse branch would use the blocks selected by the index branch to

03:48 to determine which keys and values are needed for attention computation. All right. And what I want to emphasize today is that our design is not purely algorithm like algorithm driven. Instead, we took an algorithm algorithm and infrastructure co-design approach.

04:19 Um a good good example to start with is DSA. DSA was a very strong design, but it is also highly optimized for Deep Seek’s own architecture, especially the combination of MQA and the large head beam. However, when we move to a more commonly used GQA architecture, directly applying the same design becomes less straightforward. It introduces several practical issues, especially from the infrastructure and implementation perspective. So, this motivates our design choice.

04:53 Instead of simply borrowing an existing sparse attention design, we re-think the architecture under the constraints of GQA and real system efficiency. So, what are the issues if we directly apply the existing sparse attention architecture? First, from the algorithm perspective, GQA has multiple KV heads, which should ideally provide more diversity across different KV groups. However, if we directly reuse the same

05:27 token selection strategy, multiple KV heads will end up attending to exactly the same selected tokens. This essentially cancels out the diversity that GQA is is supposed to provide. Second, from the infrastructure perspective, this design is not very friendly to GPU memory access. GPUs are much more efficient when reading contiguous data. In the MQA setting, we can think of the KV 8 layout as one KV head with a large head dimension, for example, 1 by 512.

06:01 This is relatively contiguous and efficient to read. However, in GQA, the same total dimension may be split across multiple KV heads. For example, it’s 4 by 128. These are no longer one single contiguous read. Instead, the GPU needs to perform multiple independent reads across different KV heads. This increases memory access overhead and reduces the compute-to-memory access ratio.

06:32 And another issue here is the top-k computation itself. To select the most important tokens, the indexer needs to compute top-k scores over a large number of token-level candidates. However, top-k has inherent data dependencies. So, it is difficult to fully parallel on GPU. As a result, token-level selection introduces non-trivial computation and merging overhead. So, how did we address these issues in

07:02 MSA? From the algo- From the algorithm side, we remove the aggregation of the index branch outputs. In the original design, the results from all index branch In index branch heads are merged into a single set of selected tokens. But, as we discussed earlier, this would remove the diversity brought by GQA. Instead, we keep the selection different different across KV heads. Under our sparse attention design, when using GQA, different KV heads can attend

07:35 to different sets of tokens. This helps preserve the diversity of GQA rather than forcing all heads to share the same sparse tension. For the infrastructure side, we move from token level selection to block level retrieval. Compared with token level selection, block block level retrieval significantly reduces the number of candidates and lowers the overhead of top key computation and result merging. At the same time, block level retrieval is more hardware friendly friendly.

08:08 It improves memory access efficiency and increases the compute to memory access ratio. Importantly, we also show that the design does not introduce a significant performance drop. Finally, we design efficient kernels for both training and inference. I I I won’t go into the kernel details here today, but this is the system level component that makes the sparse attention design practical and efficient. So, with this design, we were able to

08:39 extend the training of our M3 model to 1 million context very efficiently. Um and as we talked about earlier, the third pillar is the native multi-modality. Our model, Minimax M3, was trained from scratch with multi-modal data. What does that mean? Our model was

09:10 trained with both text and vision understand standing from step zero, which is very different approach from current open weight models. Our starting point is very clear. At the current stage, a standalone VL model has relatively limited application scenarios. But combining it with with, you know, text understanding during the training can be challenging. So, the key question and challenge there was how can we introduce multimodal capability without hurting the core text performance?

09:45 To answer this, we actually explored several training strategies at scale. The first and most common approach is to add multimodal training after the pre-training decay stage, which is essentially a CPT style approach. However, at that point, the model has already largely converged. Adding multimodal data at this stage tends to interfere with the learned text capability and causes degradation.

10:16 Another approach is to enter multimodal training before pre-training the decay stage. For example, after the model has already seen a number of amount of text tokens. We also tried this direction, but found that the results were highly sensitive to the data mixture and training hyper parameters, such as learning rate, for example. In one setting, a smaller learning rate might reduce the performance drop, but this conclusion does not necessarily generalize to

10:47 another run. More importantly, during large-scale pre-training, these parameters cannot be arbitrarily changed just for multimodal training. And this led us to our final strategy, native multimodal training from scratch. In this setting, multimodal capability is introduced from the very beginning rather than being added after the text model has already formed its structure. Under the same text token budget, we find that the

11:19 this approach almost does not hurt text capability. We actually did many fun analysis during these experiments. Um one of them is this interesting observation uh from the attention maps. You can see from this picture that for CPT-style multimodal models, the text tokens often pay very little attention to visual tokens. The visual token regions are mostly dark in the attention map.

11:52 This suggests that the multimodalities are not fully integrated. Well, in contrast with native multimodal training, the attention map looks much more integrated. We do not see here a clear separation between text tokens and visual tokens. This suggests that the model learns to fuse visual and textual information more naturally from the beginning. So, overall, our training strategy is not just about adding vision capability.

12:25 It is about integrating multimodality into the model while preserving its strongest text capabilities. And of course, uh we did many experiments on multimodal architecture choices. Our overall architecture is relatively standard. The image or video is being processed by a ViT, which produces visual tokens. These visual tokens are then inserted

12:55 into the language model input by re- replacing the original image placeholders in the text sequence. After that, the visual embeddings and text embeddings are fed together into the language model. One important finding is that applying 3D attention inside the ViT is very helpful for vision understanding. In a standard ViT setting, attention is is applied within each individual image or frame. For video inputs, this means that visual patches from different frames do not

13:28 directly interact at the ViT stage. But with 3D attention, patches across different frames can attend to each other directly. This gives the model stronger temporal and cross-frame visual modeling capability. And we found that this improves visual understanding, and the benefit also transfers to some image benchmarks. Suggesting that it strengthens the overall visual representation rather than only improving video-specific tasks.

14:03 And one practical detail is that we cannot simply extend 3D attention to an unlimited number of frames because that would introduce additional system and load balancing challenges. In our exploration, using four frames gave us the best best trade-off between performance and efficiency. So, overall, with our architecture design, with our training strategies that we explored throughout the stages, and with our analysis and ViT

14:35 explorations, we were able to train the model from scratch with multimodality understanding. And that would be the end of the presentation today. Thank you. >> [music]