Everything you need to know about Muse Spark Open: A conversation with Meta and Together AI

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Everything you need to know about Muse Spark Open: A conversation with Meta and Together AI · Channel: Together AI · 58:56 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, Local and open-source models, Multimodal models

Summary

The webinar introduces Muse Spark Open, Meta’s open-weight multimodal model designed for long-horizon agentic workflows such as coding and customer service. The speakers describe its mixture-of-experts architecture, million-token context window, reasoning controls, and chat format for handling tool calls. Together’s inference team explains serving challenges and techniques involving KV-cache management, speculative decoding, and prefill/decode disaggregation. The discussion also covers fine-tuning, audio use cases, tool-call recovery, workload-specific deployment, and the rationale for making models openly available.

Key points

Muse Spark Open is described as an 816-billion-parameter mixture-of-experts model, activating 42 billion parameters per token.
It accepts text, image, video, and audio inputs and produces text output, with support for a context window of up to one million tokens.
The model is intended for coding, multi-step reasoning, multimodal understanding, and extended agent workflows involving tools.
Its chat format separates reasoning, user-facing commentary, and tool calls, allowing inference systems and agent harnesses to handle each differently.
Serving the model requires careful management of GPU memory and KV cache; the sliding-window attention design helps reduce cache needs for long contexts.
The inference discussion covers speculative decoding, prefill/decode disaggregation, and multi-tier KV caching as ways to improve throughput and accommodate more requests.
Meta’s representative says the model can be fine-tuned, while noting that useful fine-tuning depends on clear goals, suitable data, and strong evaluations.
Native audio input may enable businesses to reason over recorded calls and other audio without first converting everything to text.
The speakers highlight the model’s ability to retry incorrect tool calls and use feedback from failed calls or retrieval attempts.
Meta’s representative frames open-weight releases as a way to broaden access and enable more innovation and use cases.

Chapters

Start	Chapter	Summary
00:01	Welcome and introductions	The hosts introduce the webinar, its speakers from Meta and Together AI, and the plan for presentations followed by discussion.
04:13	Introducing Muse Spark Open	Meta’s Corey Featherly presents the model’s architecture, open-weight release, intended agentic workflows, and multimodal inputs.
06:17	Use cases and evaluations	The presentation outlines reasoning, coding, tool use, and multimodal applications, then highlights evaluation results and user feedback.
09:24	Model architecture and serving implications	Together’s Joey Huang explains the model’s sparsity, attention design, reasoning-effort options, and the serving considerations they create.
14:08	Attention and long-context memory	Huang discusses how sliding-window attention reduces KV-cache requirements and describes the added complexity of cache management.
17:13	Reasoning format and inference techniques	The talk explains reasoning levels and message channels before covering memory constraints, speculative decoding, and prefill/decode disaggregation.
27:14	Caching for agent workloads	Huang describes strategies for improving prefix-cache hits and using host or disk tiers to extend cache capacity.
30:21	Fine-tuning and agent-focused chat format	The panel discusses fine-tuning for specialized use cases and how separate reasoning, commentary, and tool-call channels support agent systems.
41:19	Multimodal and audio applications	The speakers explore why native audio input matters and how models might reason over recordings for enterprise and personal-agent tasks.
44:53	Optimizing serving for different workloads	Huang explains how cache-hit rates, prefill and decode demand, and autoscaling affect deployment choices for coding agents and chat.
48:31	Tool-call recovery and model evaluation	The panel discusses the model’s ability to check and retry tool calls, as well as how developers can evaluate it for their own tasks.
52:41	Serving priorities and open release	The speakers identify global KV caching and prefill/decode disaggregation as important serving considerations, then close with Meta’s case for open releases.

Quotes

It is an 816 billion mixture of experts for a parameter model. — 04:44

So, first the model is super sparse. — 11:30

the model is absolutely fine-tunable. — 33:32

one of the ways that we can make that better is by sort of like getting the model to check its own homework and retry if it doesn’t get it right. — 50:03

if we’re training based on everyone’s data, then take the model and put it out there and give everyone access to it, right? — 58:25

Concepts and entities

Muse Spark Open, Meta Superintelligence Labs, Together AI, Corey Featherly, Joey Huang, Dan, Mixture of Experts (MoE), Sliding Window Attention (SWA), speculative decoding, PD disaggregation, KV cache, Muse model license, Together platform

Full transcript (original language)

00:01 [music] [music] [music] [music]

00:33 >> Hey everybody, welcome. We are going to give maybe a couple of minutes here for people to start trickling in and then we can get started. Hello. Hey Dan. >> Hey, how’s it going? >> Hey. We’re just waiting for super long. Awesome. We’re just waiting for people to

01:03 trickle in and then we can we can go around with intros and then we can get started with presentations and then we’ll have a discussion at the end as well. Right, awesome. So we’re streaming on YouTube. We’re live on X Twitter.

01:37 All right. Okay, cool. I think I’ve got the green light. I think we should be good to go. Um So I maybe maybe what I’ll do is I’ll set the stage and then I’ll throw it over to everybody for intros and then we can we can get into it. So Uh, in this webinar maybe for the next hour we’re going to dive into M Spark open. Um, and so when I joined Together about 2 years ago, that was the week that Llama 3 had launched. And that was a different

02:10 era. That was before the era of agents. That was a a legendary model. And now we’ve got this model that’s built for agents. So I’m super excited to kind of learn more about it, both the the model behavior side of things, and then also the serving stack and how to make it um run uh in in in prod. Uh, so yeah, I’ll start off with intros uh for myself, and then I’ll throw it over to Corey from Meta and then the Together team. Uh, I’m an engineer on the DevRel team. Um, super excited to put these webinars together. And uh if

02:41 if the community has other uh other things that they’re interested in, other questions, feel free to do feel free to let me know. What we’ll do is we’ll collect questions throughout the webinar. So feel free to throw them into um into uh YouTube. Throw them in the chat. Uh, and then we can pick them up. Afterwards, I’ve also got some questions on my end that uh I want to dive into. So we can do that as well. So uh maybe let me stop there and throw it over to Corey for intros. >> All right, nice to meet everybody. I’m Corey Featherly from Meta. I’ve been

03:11 working at Meta for about 8 and 1/2 years now out of the Austin office, and I’m on the GenAI partner engineering team. And looking forward to chatting with you all about uh M Spark open today. >> Awesome. Uh, Joey, maybe maybe you go next. >> Yeah, sure, sure. Hi everyone. Uh, my name is Joey Huang, and uh I’m part of the inference team at Together, where I’m mainly working on like identifying if there’s any uh like optimization uh opportunity for basically let the inference run faster

03:42 and uh higher throughput. So yeah, today is very super excited to to learn and uh to share anything about the the new like latest new uh M Spark. Yeah. >> Awesome. Thank you. Looking forward to your uh forward to your talk. Uh, and last but not least, then >> Hey, I I’m Dan. I I lead research at Together. >> Awesome. Um okay, cool. So, like I said, if folks have questions, feel free to throw them in the chat. We’ll get to it

04:13 after our presentations. Um but without further ado, let me throw it over to Corey, uh who will take us through Muse Spark Open and and get us started off here. So, I’ll take us off stage and I’ll bring your slides on on stage, Corey. >> All right. Okay, great. Um yeah, like I said, I’m excited to talk to everyone about Muse Spark Open today. So, what is Muse Spark Open? Uh it’s an open-weight model from the Meta Superintelligence Labs. And today, what I’ll cover in a quick intro

04:44 are the capabilities developers can build with and how it supports coding, agentic work, and multimodal reasoning. Okay, so Muse Spark Open, what is it? It is an 816 billion mixture of experts uh for a parameter model. It’s an open-weight model from MSL, and it’s trained for a long-horizon agentic workflows such as coding and customer service agents. Uh like I mentioned, it’s an MoE model, so

05:14 each token um activates 42 billion of the of that total parameter set. Uh the model is natively multimodal. It takes in um text, image, video, and audio inputs, and it produces text output. Um its use cases include things like complex software engineering, multi-step reasoning, and long-horizon task execution. And it’s released under the Muse model license. So, with that, um let’s talk about some of the model capabilities here. So, like

05:45 I mentioned, enhanced coding capabilities is one of the the primary things that this model is trained for. Uh it’s also excels at multimodal understanding, um agentic workflows, and um it’s uh developer choice is number four. The weights are available for download through hugging face in both BF16 and and BFP4 versions. Uh and it’s also available via API through inference partners. Um

06:17 Okay. So, let’s dig a little bit deeper into what some of the use cases for this model are. Uh it performs well across a variety of use cases. Um And those include things like multi-step reasoning. Sustaining coherent plans across complex extended workflows, encoding agents that write, debug, and resolve real-world software engineering tasks. Uh for end-to-end agentic tasks tasks, that includes multi-step planning,

06:47 sequential tool invocation, failure recovery, and long-horizon task executions. Tool use also extends across multi-turn workflows, including diagnosing errors and retrying when tool calls fail. Um like I mentioned earlier, multimodal reasoning is one of the areas where this model excels, and it has some really fun use cases across things like interpreting screenshots, charts, documents, images, uh and even audio and video as well.

07:18 Okay, so show me the evals. How did the model perform in evaluations? Um here we’re comparing Mixtral Spark Open with other leading open-weight models of similar size and architecture. The evaluations span agentic capabilities, knowledge and reasoning, coding, and multimodal performance with tools. Um a couple things to highlight, Mixtral Spark Open scores 61.8 on Job Bench and 76.4 on Terminal Bench. So, I’ll let you digest those more fully

07:50 um on your own time. Uh and that to pull it all together here, this is the model card for Mixtral Spark Open. Um like I said, 816 billion total parameters, 42 active with the MoE architecture. Um And supports uh 1 million token context window at inference. Takes in multimodal input, text, image, video, and audio, and produces text output. Uh also, the model is a multilingual, trained on data from more than 100

08:21 languages, and again, licensed under the Mongoose model license. So, what are some of the people that have used the model saying about it? Um from Factory, [clears throat] um we have Zimu, uh who said Miqu Spark Open consistently selected the right next tool and kept its responses compact, exactly the decisiveness production agents need. And from Ollama, we have Dong, who said, “This model performs particularly well

08:52 on tool calling in long context, including multi-step and parallel calls with clean structured outputs.” Um So, with that, I’ll hand it over to the team to continue um the presentation. >> Awesome. Thank you so much. Um so, I we’ve got some questions here, but maybe what we’ll do is uh we’ll carry on with the conversation, and then we can have the discussion uh afterwards. Um

09:24 so, you just went through kind of model details, uh how developers are using it it um how different companies a bunch of companies were trying the model out. Uh awesome things to say. Um this model is if you think back maybe a year, 800 billion would be kind of uh a huge model. So, if you think about serving this at trillion token scale, I I wanted you all to dive into some of the details uh on the serving stack. What are the complexities? How um how the Together team optimized the serving

09:56 aspect of it? So, maybe what we’ll do um any comments on the presentation first of all? If there’s no comments, then we can uh we can go into the next presentation. All right, cool. So, maybe what we’ll do, Joel, I’ll add your slides here. You can you can go through the presentation Uh and then we can come back and we can get into the discussion session. >> Yeah, sure. Sounds good. Okay, yeah, let’s start. So, as I said, I I do I do inference in together. So, so in

10:29 this presentation I will like maybe start from a perspective of serving and but my presentation will be two part. The first part will will will give a deep dive into the model architecture like and the second part will be like how do we how did that model architecture mean for serving especially for inference provider like together. Okay, yeah. So, first for Muse part, so some big number we already covered like

11:00 model itself has 816 billion parameters, but for each token it will only activate like 42 billion per token. So, it’s roughly about 5% of the weights super sparse and for intermediate context length it can support up to 1 million tokens. And uh Yeah, and and the four modality text, image, video, audio four levels of effort levels. Um yeah, that’s And uh

11:30 so by looking at those numbers, uh we can like from from my perspective like there’s a couple of bets we can see from this uh numbers. So, first the model is super sparse. So, each token only use eight out of out of 256 expert and then it’s later uh MoE. So, in end-to-end is each token only use the compute for like only 5% of the uh full model. Um which means like we

12:02 we Yeah, we can we can talk a little details in a little bit, but the model is super sparse. So, it that bets on the sparsity. The second uh when we look at the attention layers, it so Uh, you know, high level they uh, the most back you Muse back use a GQ8 group cure attention. Um, but the most of their layers like a 46 out of the 62 layers actually only like take a sliding window of uh, 2K tokens. So no matter no matter how long the

12:34 prompt is, it will only look at up to 2K tokens for those layers. And uh, so like so this kind of design also make 1 million context lens uh, affordable and like like economical to serve. And and then it’s like a it’s of course a recent model and uh, the user can specify different effort level to trade like the the the how much you will spend on the sinking and then if you if it’s a if it’s a like

13:04 accuracy matters most you might want to set a um, extra high. And if you want like low latency use case maybe you can set low. Um, so for for the Sparse Expert part it uh, implies uh, a latent MoE uh, structure which means it will like so so the model’s hidden size itself is 8,192. And and then before entering the like uh, MoE layer MoE the core MoE it will

13:36 be down projected to a 4K dimension and then the router will select like the like a router will like in parallel will select uh, eight out of 256 expert and then we’ll only use those expert to calculate for that particular expert. And then after doing the MoE it will like expand like uh, expand back to the 8K uh, hidden states and and and and the merge back to the like mainstream. Uh, in total it’s like a end to end only

14:08 5% of the weight got activated for each token. Um, so for like so for high throughput uh, scenario uh, that will be super efficient and like And the four for the like look uh SWA, you can here here’s a visual visualization of the distribution of dense layer versus SWA [clears throat] layers. Um the choice is simple like uh if we

14:38 assume it’s like FP8 precision, which is already uh super uh like compact. And uh If all the layers are like vanilla dense uh GQA, if we want to support 1 million context length, you will need to uh have like 124 GB uh just to serve this one single like 1 million context length request, uh which is basically not that yes, not

15:09 feasible for uh the current hardware. Uh but because of the GQA, um the GQA compared to the dense layer like especially at the 1 million context length this this this scale, um it’s uh it’s a cost of GQA is basically negligible because GQA only need to make keep the KV cache of 2K tokens. So, most majority of the KV cache can be evicted immediately after doing the preview.

15:39 Uh so, although it is still take some like but you can roughly uh calculate that it only need to uh it immediately save like a 75% of the uh KV cache, which means four times more KV cache room for for for for for for serving more traffic. Uh so so with with GQA, uh I was sorry with SWA, it only need to uh cost like reserve 32 GB uh for each

16:11 group for for for 1 million context length request. And if you do TP to tensor parallelism, the 32 can also be like divided by if it’s TP2 TP4, the each GPU only need to keep 8 GB for each request. Uh if we do TP8, then we’ll be 4 GB for each GPU, then it’s much uh much more affordable. Um also like because of this hybrid attention TV, so

16:42 itself is super uh efficient and um cost efficient. Um what else uh we can talk about details uh in in the serving part, but it is make it slightly uh a little bit more complicated to manage the key make the lifetime of uh the key make cache and also add some little bit complexity uh for previous caching. Cuz nowadays for coding agent or like general agent, the

17:13 this company has super long so the previous cache request is pretty important. Okay, then is the reasoning. So for reasoning uh the this model supports four reasoning efforts, and then it will be mapped to like four different levels. Uh you can see it’s actually we present it at the log scale, but each like when we increase the reasoning effort, it actually increase uh couple of times uh larger. And uh higher

17:44 reasoning effort means it will the model will spend so more tokens at the reasoning uh thinking stage. So that So that So that if you look at the template, then it will be separate channels. Model will do thinking and then then make two calls and then go back to the user. Uh so a general tips is to if you want to use high uh reasoning effort, then at least you need to set the max token large enough, otherwise uh if there is a chance that the model

18:16 keeps doing a lot of thinking and then they got truncated in the middle, then you might sometimes see empty output. Uh then it could be this case. So, to take a closer look at the chat format, so I I’m for my opinion the chat template is for a beautiful agent. You can see that there’s a different separated channels. The user channels is basically input. The assistant can talk to itself.

18:46 Uh this part and they all harness it can show or not present to the user. And after doing the reasoning, the assistant the model can do like commentary output just to show the progress for the user. Um and immediately the the assistant can also make two calls and harness does the actual two call and get back to the two call to the model. The model uh generate the actual output to the user.

19:19 So, this provides a very clean channels and and the template for for for agent use. And um yeah, it’s like to me makes a lot of sense. So, in in in actual for for serving, you can see that it also distinguished EOT and EO end of message. So, for EOT, uh the model generation will stop. That this basically separation of the

19:51 model generation, but EO message from the serving perspective it is is basically just that you tell this channel has stopped, but then the model generation still keeps going on. So, in practice then what the model Sorry, the user give a prompt and the model will keep generating until the first the two call being made and it hits the first EOT. And then it will make a two call and it get back the two call result to the model. Then eventually the model we to

20:21 make another uh generation and uh get back to the user. Uh for multimodal inputs uh is uh it has a separate encoder for different modality and then my map to the embedding and and and the fill in the placeholder tokens in the uh text input. So, uh how does that mean for serving? So, so here I I might just

20:53 mostly cover the techniques details. It does not mean necessarily mean our production use uh the the the strategy or or the technique I mentioned in the in this presentation. But like to to give a example uh so this model itself has 800 816 billion parameters even in uh MVIP 4 is still like a at least just to serve just to have the model itself you need to at least

21:26 uh 400 a little more than 400 GB. Um that means uh the minimum number of GPU to serve is like at least a four B200 and to visualize the memory footprint um the most of memory already been taken by the model weights and uh the remaining you still need to do other stuff like to to to book the TV cache and CUDA graph um a lot of like workspace for kernels. Uh header room for for

21:59 preview. So, uh you can see it’s actually pretty compact. And uh so actual server if you use this uh if as an inference provider if we use this kind of config the TV cache is actually the biggest the bottleneck. So, yeah. This when we talk about the inference we immediately the first first thought will be okay, how do we do do the preview? How do we efficiently do the decoding and the preview is more compute bound decoding is more memory bandwidth bound but that before

22:32 talking about that I think the the biggest immediate problem is how do we handle the how do we do how’s the key key we cash how do we be able to batch more requests as much as possible. And so internally compared to the initial implementation we did some uh tricks to share the the the replicated the layers and also to share the

23:03 um the the tensors uh like as much as possible. There are also other tricks we haven’t putting onto the table. So it’s really the a lot of the engineering details to cut down the memory footprint so we have more space for the key we cash. And uh and then and then uh the after the talk about after tackling the key we cash another important thing is is a speculative decoding. As we said

23:36 like if we don’t have enough memory we’re always fighting with the key we cash memory footprint and uh because if we don’t have enough key we cash we cannot batch enough requests to push the decoding like to from memory bandwidth bound towards compute bound. Then it also means if we don’t have enough key we cash like it always be um the the GPUs compute is uh the the utilization of GPU is not high

24:08 enough. Uh one way to like to to cope with it it is to use a speculative decoding basically to use more compute. We use a smaller draft to provide the candidate tokens and then one forward pass the the target model check all these tokens of draft tokens in one forward pass. And then uh to like >> [clears throat] >> or and and then we can generate a multiple tokens in one forward pass.

24:38 So, in this case we basically trade more compute to to like make generation faster like um So, So, when [snorts] we talk about speculative coding is sometimes always a hard thing to do the evaluation because it’s uh in the in the in the in the public post you might see there’s a very like uh crazy headline numbers like 1,000 TPS. Uh because speculative decoding is

25:10 really uh is easy to overfit and then it’s also domain dependent. So, if you really want to get a super super high number, that’s that’s definitely possible. But, here we also want to be realistic and uh um we measure on the different data set and uh in the coding agent this can achieve at at least the 2.7 uh maybe roughly three acceptance lens. And which is actually already pretty good to me.

25:41 Uh but we was but yeah, of course there is a big room to further improve. Um another case is uh PD dis- PD disaggregation. Um so, I I mentioned the the the the the prior comparison I made is basically on aggregated serving. So, we when we do uh preview and decoding in the same path. Uh another optimization we can make is to do the PD disaggregation because

26:12 preview and decode having such a different um like feature like previous uh need to do a lot of computing in one forward pass so it’s uh preview bound sorry, it’s it’s compute bound and uh and the decoding is like likely to be uh either key we cash bound or memory bound response if key cash is not a problem. And uh and then by separate reason it’s also provides much higher

26:43 room for optimization because we can for preview we can do a different like parallel strategy for preview. Maybe we can do more like uh pipeline parallelism. So we don’t we we don’t need to do a very expensive communication for for each rank. And uh for for decode maybe we can do uh do higher TP rank. So basically shot the model in a fine grain and then we have more headroom for key we cash.

27:14 Uh this is why we do pre- PD disaggregation we always target for high throughput, higher concurrency and then it would be uh like it would through uh like a bigger challenge for uh the decode node on the key we cash side. And another thing we can do is is as we talk about the how to how to optimize the throughput for agent equal work workload

27:45 is always key we cash and how do we hit more previous cash? Uh for for for for stand alone for each part it itself has a device level previous cash. Uh so for dense layer we just need to hit uh everything we can hit for the sliding window attention maybe there’s something else we can do. Um because we only in practice we only need to keep the last window. We don’t need to keep the full

28:16 key we cash. Uh which means uh if the counter length change like if we do some little a little change in the middle to do some edit in the middle uh it it might not hit the previous cache. Uh because uh in practice we will only keep the last window window and we cannot guarantee the prefix to still keep in the the the

28:47 the the the device memory. Um as the new request coming, it’s very easy to evict the the prefix. Uh but as long as the the the prompt is we append the new tokens after the initial prompt, then it is likely to hit. So, in practice we will hit the previous cache in our production and our testing results shows we can achieve at least a 92 and even higher So, 92 is actually a very conservative estimation.

29:19 Um [clears throat] Yeah. And but again, the device memory is pretty small. So, in practice we will likely to do uh couple other tiers of key cache. Uh maybe hostile and a disk disk level. Each has its own uh specification like disk is is really relatively slow. So, we might not be able to flash all the key cache into the disk, but but

29:50 uh instead like keeping a checkpoint or couple of something of tokens for the host memory, we will likely to do a global level uh allocated the key cache to Um so, we do not need [clears throat] to always attach one session to one pass and that can actually flow to other pass because of the global key cache.

30:21 Yeah. So, yeah, that’s pretty much, I think. Uh yeah, feel free to try out the new Spark open uh together with this endpoint. Um Awesome. Thank you so much. Um we can go into uh just from these couple of presentations, I’ve got a bunch of questions. Um I there’s some questions from the from the chat as well.

30:53 So I may [snorts] I think I’m getting feedback. Drew, that might be you. >> [clears throat] >> Um let’s see. Okay, so maybe maybe uh what I’ll do is I’ll set the stage for kind of this model, and then uh I wanted to get into both what people are using this model for, uh and then I wanted to ask both Corey and Drew a questions around like how the model was designed, how we’re serving the model to address uh a lot of those

31:25 use cases. Um so uh like the Llama series of models that Meta launched where uh even people are people are even using them now. And so one question that I had for Corey was around uh fine-tune-ability. One of the kind of the biggest use cases that we saw on Together for the Llama series of models, Llama 3 8B, Llama 3 70B, was that people would fine-tune the model for their own use case, and then they would deploy kind of dedicated models or Lora serverless uh fine-tunes of it. Now

31:57 we’re seeing kind of um the and they would kind of uh fine-tune it on their own data. Now that this model is more optimized for kind of always-on agent use cases, coding agents, um RL fine-tuning. Uh uh have you guys kind of looked at the fine-tune-ability of this model, and are you guys seeing uh people fine-tune this model and put it into production? Um I feel like that could be a very powerful use case, uh especially for this model.

32:29 >> Yeah, for sure. And I you know, I think fine-tuning is a really one of the things that is uh unique value proposition for open-weight models. Uh and it allows you to hit an area of like performance and cost for a specific use case that can be really difficult to achieve otherwise. Um, so yeah, I mean I think it’s one of the really important um use cases for the model is the ability for uh anyone to be able to take that model and fine-tune it for whatever it is that they’re doing specifically.

33:00 Um, so yeah, you asked like a multi-part question there. Let me remind me what are what are the parts that you wanted >> Yeah, I guess one one one thing that’s really interesting is uh whether um before you guys release the model, do you guys study how fine-tunable this model is for if I wanted to fine-tune it on my own uh code base, uh would I be able to do that? And then another use case that a lot of companies are building around are these always-on async personal agents. So I I know that Meta has the Muse

33:32 product, which is quite successful. Uh was a motivation behind this model to kind of power a lot of these uh personal agent use cases as well. >> Yeah, um yeah, the the model is absolutely fine-tunable. Um, so and that’s one of the things that you know you know I think is behind the intention of Meta releasing the model. Uh and it’s explicitly, you know, called out in the license that like, you know, fine-tuning is is um allowable and one of the things that’s uh supported use case for the model.

34:03 Um, and in terms of how it’s fine-tuned, I think that really depends on the end user uh and what their what their use case is. I think obviously, given the way the model was trained, it’s probably going to be an easier lift to fine-tune for things that are adjacent to those primary use cases, right? So if you if you stick with those primary use cases, and then you wanted to eke out a little bit of additional performance for your specific code base or help the model become more familiar with what’s going on there, um that is something that you can do. It’s also worth keeping in mind with a

34:33 million, you know, token context window, you can also just put some of that stuff directly in the context window, right? You and you could an attractive option for you to go that route as opposed to doing the fine-tune. Like if if there’s needed context, do you have a tool or something that can inject that into the context so that in in each specific round of inference, you’re getting the information you need that way. Um but in some cases, people do want to do a fine-tune and that’s supported as well and I think it’s a reasonable approach. I think when it

35:04 comes to fine-tuning, one of the things that’s really critical is you have to have a good set of eval stick to guide the fine-tune otherwise you you don’t you don’t really know what you’re doing or you can’t get really get a useful direction out of it, right? So, if you have a very specific goal and a good set of evals that lets you know like this is what good looks like and this is what I need from the model and then large enough data set where you can train it toward that, then yeah, there’s definitely an opportunity to use the model for that. It does come with a little bit more work in terms of like you then have to go do the fine-tuning process, right? It’s not as

35:34 accessible as just, you know, hitting the API and and getting your response immediately. >> [snorts] >> Yeah, you you touched on the the evals and if we start talking about evals, I feel like we’ll be here for another another couple of hours. I’ve spoken to so many companies that are kind of building their evals out and and kind of identifying what data and what good evals look like. And I feel like especially now that companies are are eling the model, the end task needs to be verifiable.

36:05 This is kind of leads to even underscoring the importance of evals that are quantifiable and less more less vibes. There’s one question in the chat around whether fine-tuning for this model is supported. Fine-tuning for this model is actually supported right now on the Together platform. I’ll drop a link for in in the in the thread here. You can go over here. This is where you’ll see all the models that are supported. So, Mews park open is supported. You can RL, SFT, DPO

36:37 fine-tune this model on the on the platform. Okay, awesome. So, so the next question I wanted to go into maybe what I’ll try to do is I’ll ask a question about the model then I’ll ask about the serving stack and then I’ll we can ping-pong back and forth and we can also take some some questions from the from the chat here. So the next question maybe to actually this one can go to Ju a or Corey because this is more about the chat format. Ju a you mentioned that the chat format is specifically for agents and you mentioned that you can have this kind of multiple stream where the agent

37:08 can stop one aspect of it but it can keep um you can have a end of EOT versus an EOM special token that’s in the chat template. Can you talk can you talk a little bit more about practically what this means for developers in terms of use cases and where what use cases would would this be relevant for and how is this different from all the other models that developers might be used to see? >> Uh yeah, I can maybe I can share a little bit like comments from from inference guy but

37:39 like I will let the Corey say more like later. So to to me that different model has their own chat template for sure and for for this one I think it makes a lot of sense because the it first of course like as as other model it has its own channel to do the reasoning so it uh that that that part is like a kind of like a regular because everyone like right now every model is always more or less that do some like reasoning and then but it’s

38:11 that the interesting part is like a separate channel for reasoning and a separate channel for get some like brief feedback to the user before doing anything more. Like you want to do reasoning after doing the reasoning and then get back to the user to say what what’s what’s I’m going to do is to make some tool calls for example for for tool calls and then there will also be a tool call channel um just to to describing how the model going to use specific tool

38:42 um and then we hand over to the to to the harness that you do at making actually the two calls and get back to the model. So, uh So, yeah. So, I think that the separation that the different the channel is very clear for first. And then also uh when when when when designing the these channels is an interesting select there they have different special token to mark the the end for each channel. Um and then from again like a from the

39:13 survey perspective uh it’s makes it easier to like let them let the let the inference engine to know like when to actually stop. Because sometimes there are a couple of uh channels, but for inference to they’re just tokens, right? They’re essentially just token. Um So, EEO MFEOM that you you know end of message that don’t necessarily mean the the engine need to stop, but you can just keep going keep going streaming back the token and let the harness

39:44 decide which part should we present to the user which part should we like hide hidden from the user and then which part of should we actually asynchronously start to make some tool calls and then get back to the model. So, yeah. So, yeah, that that makes a lot of sense to me and uh maybe Corey can say more about how to how do we like from a modeling perspective uh how do we [snorts] make this decision?

40:16 >> Um yeah, I mean I think you covered it pretty well. Um I think the three areas there are reasoning, the content, and the tool calls. And yeah, I think you covered it. So, unless there’s like a follow-up question to that, I think we’re good on that. >> Yeah, awesome. Um I also feel like now uh a big use case are these always-on agents. And so, one uh really interesting thing about that and uh every kind of always-on agent harness has its own uh way to parse output from

40:47 the model. And so, if you a lot of these different um unique special tokens in the chat template, it allows the harness to do more customized lifting. Like, you can parse the tokens differently, you can enable different different harnesses to uncover different capabilities of the model. Whereas before you were kind of you were kind of duct taping over this in the chat template chat template might not have have those special tokens to parse around. So, I feel like that’s another

41:19 where we’re seeing like coding agents used to be the thing. Now, these personal agents that are always running in the background are are increasingly more and more popular. And now we’re seeing kind of the the ecosystem of models incorporate this into the model itself as well, which is which is super exciting. Um And maybe what one you mentioned that this model has kind of a very diverse input modality. It takes in text, image, video, audio. Um that’s not like not a lot of models

41:49 do this, right? So, Corey, can you talk a little bit about why you guys decided to make this model so multimodal? What were some of like the use cases that you guys were targeting? And have you seen um some cool use cases come out of this? Uh Troy? >> Yeah, um great question. I think that’s one of the things that’s really exciting about the model is that your ability to sort of like ingest pretty much any any type of data. And one of the things that is a little bit novel with this model that we haven’t seen as much of in the past is audio input. Um and I think that opens up a lot of

42:19 really interesting enterprise applications because there are situations where your data is locked away in audio somewhere. Like, you have recorded customer calls or whatever it might be, and you need to reason over that data or retrieve you know, um uh answers to questions about that data or sort of like be able to make use of it in a more accessible way without like transcribing thousands of hours of audio or something like that and make manually turning it into a text input because that’s the input that the model requires. But, being able to just, you

42:51 know, shove it into the model directly because the model natively is able to reason over audio input I think unlocks essentially a whole new type of data store for a business that might be operating in a way where some of their data is locked away essentially in an audio format. >> Yeah, I would agree. I I was building an app before this a couple months ago where I needed to take in like you could record your own audio and then you could find what the closest

43:22 text-to-speech model is that sounded like the recording audio and this is kind of when when customers are building these voice agents, they might have a voice that they wanted to sound like but they’ve got a catalog of that are supported with their TTS model and so I was building an app to help people pick the perfect voice and I realized that the only open model that supported voice in was like a 2B 4B model and so when you ask it like what basically the idea was give it an audio file and then extract out metadata

43:52 around how it sounds, what the emotion is like and I’ve got this like 2B 4B model which is the bottleneck to how well this app runs and so the fact that you guys have this like very large model that can now reason over audio input and you can ask it more things other than just transcription. You can ask it does the user sound like angry? You you can ask it do you can do all sorts of things like emotion classification. I feel like that’s a a very big unlock for especially a lot of these personal

44:22 agents like if you get if you’re sending this model some sort of audio file, you can actually do emotion extraction, have the agent do that and then respond differently based on based on that as well which is yeah, it’s it’s almost like outside the transcription STT modality. You’ve you’ve also unlocked a lot of these other interesting use cases as well that people can do now. Not only that but you can also fine tune on that aspect of it as well which is very very cool.

44:53 Um Awesome. So let me Joey, you talked a little bit about the serving stack disaggregated serving stack where you can separate out the pre-fill workers from the decode workers. We’ve talked a lot about the different use cases that people might want to leverage this model for. Can you talk a little bit about how you can use pre-fill decode disagg and then how you can optimize that configuration for like a coding agent use case

45:24 where you’ve got like a large high input sequence length, small output sequence length, really high cache hit rate versus like a chatbot use case where you’ve got really high sequence length, low cache hit rate. If you’re doing like a pre-fill decode disagg, how do you think about optimizing those different types of workloads? >> Oh, that’s a super good question. So, to my opinion, like uh PD disaggregation itself is like always targeting at

45:54 improving the throughput, like the high throughput region. Um but for different scenario, like if it’s coding agent, then it’s high cache rate high cache hit rate for pre-fill and long contest. So, it can we can do some like specific optimization for this scenario and there were also cases like maybe regular chat sequence sequence is not that long, but it the output might be long. Um Yeah, yeah, definitely different uh

46:24 scenario. Uh what I usually do is to maybe for for local testing for the for the for the general development, what I usually do is to figure out the like the the cache hit rate and then try to mimic the cache hit rate with on the pre-fill node and then then try to maximize the throughput for the pre-fill and then and then and then the decode and then and then just try to pair the decode with the pre-fill nodes to the correct ratio. Uh, I think that’s

46:56 basically can work uh in the most of the time, but in in production server it it could be a very different story. Like for for production server we definitely need the auto scale. So, not it’s not like something we we uh tune manually. Uh, what we really need to do is to find out the the correct metrics like for preview, uh the number of queue requests, the the preview TTFT as as a metric of auto scale. Then the preview node and the kernel can uh scale uh

47:27 they’re independently. Um, so in practice like if the the traffic pattern change from time like maybe in the morning at the the traffic in the morning and in the in the evening are actually slightly different, you can just changing from cuz in the morning maybe people are still sleeping, so then the most of traffic is to uh agent. Right? At at at the daytime maybe people also start to interact with with with the agents and like this this could could change. Uh, I think the right way and the scalable way is to to

47:59 correctly config everything to scalable and uh be able to adaptively uh adapt adapt adaptively change the ratio between P and D and the uh then we can maximum the throughput uh given the fixing given the different amount of GPU. >> Yeah, I think that that’s like a super challenging thing, right? You’ve got if you deploy a config that’s optimized for coding and then people start throwing chat traffic at it, how do you how do

48:31 you adjust to that? And so, um yeah, maybe maybe the maybe the the real answer is you’ve got multiple configs that are optimized for these different workloads and then you bring them up and then you can route traffic accordingly. Um, what one kind of interesting uh question that was mentioned Corey in your presentation, um a couple of the users that we have around Lake with with factory and Ollama mentioned in as well. Uh, you you mentioned this in your presentation as well. I I wanted to

49:02 touch on this cuz I don’t think this gets a lot of attention. Specifically, this model you mentioned that this model is good at tool calling, but not only that, it’s good at retrying tool calls. If it makes a wrong tool call, it it retries that tool call. Um, can you talk a little bit about what this looks like and how you guys, um, like how Yeah, in action, what does this look like? Is the model just really good at if it makes a mistake, fixing it? Or are there other uh, use cases for this? Um,

49:32 it’s interesting that a lot of the companies that are using it bought brought this up specifically. So, maybe Corey, I’ll throw it over to you and then Joey can comment as well. >> Yeah, sure thing. I mean, I think getting the tool calling right is something that’s really important for um, a lot of the business use cases because if it’s missing that uh, substantial portion of the time, it can be really hard to implement it into a production workflows. Uh, and so I think in this case, uh, especially, you know, in in the past, um, let’s just say that tool calling

50:03 isn’t always as good as uh, users of Llama might have hoped that it could have been, right? So, there was room some room for improvement there. And so, I think the the idea here was uh, one of the ways that we can make that better is by sort of like getting the model to check its own homework and retry if it doesn’t get it right. >> I see. And so, it’ll do that for tool calls, but it’ll also do that for retrieval. So, if you let’s say you’ve got an a knowledge base or you’ve got some database that it’s retrieving from, it’ll be able to look at the context and

50:35 then retry uh, tool calls uh, based on if the right thing wasn’t uh, retrieved as well. So, it’ll do like multi-turn tool calling where it’ll fix its previous mistakes as well, right? >> Yep, exactly. >> Yeah. Um, yeah, that’s pretty awesome. I feel like uh, Llama was built for the era of chatbots and this this thing is built for the era of agents and uh, tool calling is the unlock for for agents. So, uh, it’s interesting that a lot of these customers that tried it out brought brought this up specifically. Um

51:07 The The other question, maybe a follow-on to this, Corey, you you showed a kind of this table of comparing new Spark open to a bunch of other open source models. How do you How do you want developers to think about new Spark open? Uh What do you want them to use it for? How do you want them to kind of evaluate this model for their own use cases and and go about that model selection process? >> Yeah, I mean, you know, like we talked about before, one of the best ways to

51:39 evaluate the model for your own use case is if you have your own email, like just run it against the model and and see what shakes out, right? Especially if you’re in some of these sweet spots that the model is being built for, things like, you know, coding, multimodal understanding, and agentic workflows. Uh if you’re adjacent to those, it’s probably worth evaluating the model and seeing and seeing um you’re at. Um Um yeah, and I think those are those are the main thing. And the other thing is we also discussed a little bit about is fine-tunability of the model as well.

52:11 Um so, that’s that’s the other thing that I can see being particularly attractive about this model versus some of the other models that are available is if you have a use case where you really have like your your you have your own data set and you want to have a you know, a modern model that’s able to fine-tune toward that data set to get you a particular outcome, it’s probably worth taking a look at the model and seeing what new Spark open can do for you. >> Yeah. Awesome. Um Uh not only did we have the model supported on day zero, but I feel like

52:41 for this model specifically, because for the Llama family, fine-tuning was a big use case, um we had day zero support for fine-tuning as well, cuz I feel like that’s just as important for uh for this open model. Um so, maybe one one last question for you, Joey, and then I’ll I’ll throw it over to Corey to to take this home, but you mentioned that there’s a lot of different opportunities to optimize the the serving for this model. Um for a model that’s this large, about 800 billion parameters, and it’s um very sparse, so about 5% as sparse,

53:14 5% sparsity uh for for like uh a forward pass, what do you think uh has the most uh benefits of optimizing? You You mentioned spec decode, you mentioned KV caching, you mentioned uh PD disagg. What do you think is the biggest determining factor if uh if you had to choose out of all of the options? >> Um that’s a good question. So, if I have to choose, I think So, because of this model itself is big enough, then

53:44 uh a big challenging to put it into production is to finding the best like Of course, we can just do TP and we can use GP300 8GP300, then that it will definitely work, then uh but the problem is always like doing the trade-off between uh whether we want to maximize the the the the like the pre and the decoding that So, it’s either computer bound or memory bandwidth bound, but we want to maximize the hardware utilization, or we want to

54:16 uh give enough headroom for the KV cache so we can cope like host enough uh requests, then we can then in turn the maximize the utilization because uh from the for the first uh side, if we want to maximize the prefill capability or decode capability, then we will need to do use a smaller maybe smaller TP size, or even use PP uh pipeline parallelism, then we can save some uh the the the the the the gem problem shape will be look better cuz it did not

54:49 shrink that much, and the the gem will be large enough, and uh also there’s no or uh much less uh communication overhead, uh but the problem is once we do that, then the KV cache shrink too much, and uh we basically bottleneck at the KV cache, and the KV cache is not allowed the the engine to take more requests. The pre-fill and decode is always on light light light load. Um so it’s yeah so it’s always fighting

55:20 between like can we have more key we cache or how much is enough? Uh so in in in in this angle then I would say the most important factor would be Well, I was it’s hard to find the best thing the most important but but I think that the top two would be uh key we global key we cache for to have a external key we cache uh in addition to the device key we cache uh either host or disk or both.

55:51 And uh then the PD disaggregation it which allowed us to use a different strategy like for for pre-fill if the pre-fill cache is not a problem then for pre-fill we can use a small scale uh uh parallelism strategy but for decode we can use a slightly uh larger and to to to then uh take to enlarge the key we uh key key we key we cache size to host more requests. Yeah. >> Yeah, I feel like your your answer is also indicative of the times right now

56:23 because a lot of the traffic is just um agent traffic. So caching is probably the number one economic >> Yeah. >> Yeah. >> tool uh and then also you’ve got kind of very high uh pre-fill workloads that are you might have like 800,000 uh token pre-fill that’s coming in that might be from a session that’s been sitting there so it might be like a a cold pre-fill that comes in so PD disag is is also uh fairly important. Um Okay, so maybe I’m I’m I want to throw

56:53 it over to Corey. This is super exciting. This is like when you have a favorite movie uh and then a second part comes out and Meta open sourcing models again. This is a super exciting time. Can you maybe take us home? Tell us about um Um, why Meta is open sourcing models uh models and what you want to see from the community with this new model, and and what you’ve got coming coming down in the pipeline to get the community excited. >> Um, yeah, sure. So, I mean, I think I’m

57:24 also really excited about Meta open-sourcing models as someone who worked with Llama [clears throat] a lot in the past during that time. I think it’s a really important thing to do um, because uh, it’s almost like um, I don’t know how to I don’t want to steal Zuck’s tagline, but like the future is for everyone, right? And so, I think it’s a really important way to take the this the great benefits and output of this technology and put it in the hands of more people and allow more innovation and more use cases and more

57:55 people to have access to the technology as opposed to having it locked up behind like an API paywall where only a very small subset of people actually really have access to it. >> Yeah. >> I I think that is going to open up the door for some more interesting and innovative use cases that we just would not have seen otherwise. And another reason, I guess, we’re getting into a little bit of a personal note here, but like I think it’s important because a lot of the data that, you know, is training, you know, not just Meta models, but all of the

58:25 modern GAI models is coming from all of us, right? And so, like all of us should be able to have access to the models if we’re all providing [clears throat] the data for it. It’s only fair. So, I think it’s it’s it’s an important sort of like part of completing the cycle and is is, you know, if we’re training based on everyone’s data, then take the model and put it out there and give everyone access to it, right? >> Amen to that, brother. All right. Um, so with that, maybe I’ll end it off. Um, if folks have any

58:56 questions, feel free to feel free to send it to us. We’ll do a lot more of these. Um, this is awesome. Meta is open-sourcing models. We’re We’re back, baby. Let’s do it. All right. Thanks, everybody. See you soon. Bye.