Meta's New Models & Agent

Author: Sam Witteveen · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Meta’s New Models & Agent · Channel: Sam Witteveen · 22:22 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Local and open-source models

Summary

Meta released Muse Spark Open, an 816-billion-parameter open-weights model, alongside more capable closed models and a personal-agent product that uses a customized model. The video compares Spark Open with other open models, discusses its licensing and hardware requirements, and explains how Meta built its agent around behavior-specific evaluations. In demos, Spark Open performs well on function calling but shows variable output length and can use many tool calls; Muse Wonder generally reasons with fewer visible thinking tokens and performs better on the agent tests shown. The host sees the release as a positive sign for open models, while noting that Spark Open trails leading Chinese models and is too large for most people to run locally.

Key points

Muse Spark Open is a mixture-of-experts model with 816 billion total parameters, 42 billion active per token, a one-million-token context window, and native text, image, video, and audio input.
The model scores 39 on the Artificial Analysis Intelligence Index, below the open models discussed from Chinese labs and slightly below Qwen 3.8.
Compared with Reflections Beam, Spark Open scores higher on several cited benchmarks, while Beam is smaller, more token-efficient, and Apache 2.0 licensed.
Spark Open uses Meta’s Muse license, which includes attribution requirements and requires a separate agreement for model-as-a-service businesses above $20 million in annual revenue.
Running Spark Open locally requires substantial hardware; the video cites a setup of two M5 Ultra Mac Studios with 512 GB each and alternative large checkpoints.
Meta’s closed Muse Wonder and Muse Wonder Full models score higher than Spark Open, but Wonder Full is available only through Muse Code subscription plans, not an API.
Meta’s Muse personal-agent app uses a customized version of Spark, not Spark Open; the exact model and fine-tuning details are not publicly clear in the video.
Meta reportedly built the agent by defining more than 100 behaviors, creating evaluations for each, and using thresholds to decide whether the product was ready to launch.
In the demos, Spark Open performs well on basic function calling but can make many tool calls and produces highly variable output lengths; Muse Wonder scores better on the agent tests shown.
The host considers the open release valuable, especially for organizations seeking alternatives to Chinese models, and hopes Meta will continue releasing updated open models.

Chapters

Start	Chapter	Summary
00:00	Muse Spark Open and Meta’s model releases	The host introduces Muse Spark Open, its specifications, and the distinction between the open model, Meta’s closed models, and its personal-agent model.
03:10	Comparisons, licensing, and running the model	The video compares Spark Open with Reflections Beam, explains their different sizes and licenses, and outlines the demanding hardware needed to run Spark Open.
05:54	Closed models and the Muse personal agent	The host discusses Muse Wonder and Wonder Full, their access and benchmark results, then describes the traction and capabilities of Meta’s personal-agent app.
10:14	How Meta built its agent	The video recounts Meta’s agent development process, including defining more than 100 behaviors and evaluating each one before launch.
12:51	Model demos and reasoning behavior	The host compares Spark Open and Muse Wonder on SVG generation and long-form responses, noting differences in visible thinking and output consistency.
17:40	Agent tests and overall assessment	The models are tested on function calling, agent tasks, and website generation before the host assesses Spark Open’s place in the open-model ecosystem.

Quotes

It’s got what now has become kind of the standard 1 million token context window. — 01:37

Beam is the cheaper, cleaner licensed, smaller one — 04:18

They then trained the model against those evals — 11:18

Meta is definitely in the open models game again. — 20:50

Concepts and entities

Muse Spark Open, Muse Wonder, Muse Wonder Full, Reflections Beam, Meta, OpenRouter, Muse app, Muse Code, evals, OpenClaw

Full transcript (original language)

00:00 Okay, so last week Meta released the weights for Muse Spark Open, an 816 billion parameter model that they’re calling the most capable US open weights model to date. and I gotta say, based on the numbers that they’ve released, this claim seems to actually hold up but this isn’t the only thing that they’ve been releasing. Meta has been on a absolute tear lately with what they’ve been shipping so don’t forget it’s not much more than a year that the Meta Superintelligence Lab was actually created they shipped an open

00:32 model, Muse Glimmer, early in the year, Which at the time I talked about as being very interesting because of the fact that it was the first model from that new team. clearly that set a bar. After that, they had other people, leave from Google to join that team. And I’ve got to say that they’ve really been doing a lot of interesting stuff since So in the same week as them shipping this Muse Spark Open, they shipped two closed frontier models, One of which you can’t even get through their own API or through OpenRouter, et cetera and it’s important to distinguish that

01:06 what they’ve actually released here is not the model behind their personal agent app, which is doing really well here in the US and it turns out that model might actually be an entirely different model altogether So in this video, I want to try and untangle what Meta actually released And then actually show some demos with the open model Okay, so let’s look at the actual model. So Muse Spark Open is a mixture of experts model. It’s 816 billion parameters total with 42 billion parameters active per token

01:37 It’s got what now has become kind of the standard 1 million token context window. It’s also natively multimodal, which is really cool to see. and it’s not just text and images that’s making it multimodal, it’s video and audio in as well but as to be expected, it’s just returning text now Meta is pitching this as, a sort of general purpose model for coding, agentic work, a variety of different things. And they’ve made this claim that it’s the most capable US model.

02:08 It’s a very specific claim, right? So it’s kind of funny that they’re not saying it’s the best model out there, And I guess that’s clearly pointing at certain countries perhaps having better models And that kind of makes sense, right? if you look on the artificial analysis intelligence index, this is scoring 39. so the top open models right now are all Chinese. We’ve got the Mimo 2.6 Pro at 46. We’ve got GLM 5.3 at 45, Kimi K3 at 44, and even the Qwen 3.8 is just above

02:39 Spark Open at 40 so while I do think this is clearly a solid model, it’s not the leading model in the open field And I do feel that shouldn’t be something that stops us from sort of trying this model out. Definitely organizations like Meta tend to make models and train on data that is quite different than the Chinese models. so perhaps while there’s no benchmaxxing going on or anything like that here for benchmarks, it is interesting to see how this actually generalizes

03:10 to a variety of different tasks The other big release that just came out just after this, which was also a US open release, was Reflections Beam. So Beam’s weights literally just went up on Hugging Face last Friday even though they announced the model earlier on in the month, and actually at the time I wanted to make a video about it, but there were just no weights to even try it. so Meta’s weights have come out first, and that’s why I’m focused on this now clearly Spark Open is beating the Beam model, right? The Beam model scores 30 on that same index so Spark Open is clearly

03:45 a lot higher than that with 39 and when you look at the benchmarks that both companies publish, Spark Open is winning the majority of them. For things like tool use on MCP Atlas, It’s 89 versus 79. For DeepSWE, it’s 60 versus 44. although Beam does edge it on the terminal bench with 80 versus 76. But here’s the thing, in many ways it’s not a fair comparison. Beam is a much smaller model. Beam is only 501 billion parameters in total with 23 billion active.

04:18 and on top of that, they’ve really focused with that model to make it very token efficient and something that I do think we have to take into account is that model is fully Apache 2 licensed. Spark Open is under Meta’s own Muse license. Now, for most people, that’s gonna be fine, but you do have to display a built with Muse Spark Open in your product. and if you run a model as a service business with more than $20 million a year in revenue, you need a separate agreement with Meta to be able to do that so I

04:50 guess I would say that Spark Open is the more capable model, but it’s much bigger. Beam is the cheaper, cleaner licensed, smaller one now as for running this thing yourself, Meta worked with Apple on this. The headline setup is two M5 Ultra Mac Studios with 512 gigs each, and apparently Apple measured that at about 25 tokens per second now I don’t know off the top of my head how much that would actually cost, but from what I’m hearing, not only would it cost a fortune, but you’re probably gonna be waiting until January or February

05:23 for that hardware to actually arrive. Now, if you’re not buying Macs, they’ve got an NVFP4 checkpoint. It’s around 530 gigs. and Unsloth themselves have made a 2-bit checkpoint, That gets it down to around 300 gigs. but clearly for most people though, the realistic way to try this is gonna be one of the hosted service providers. and in that sense it’s been on OpenRouter since day one So I’ll come back to the demos for the Spark Open in a second, but let’s just talk about what else Meta has actually been releasing here.

05:54 Spark Open is clearly not their best model. It’s not even really close, to their best model. Meta clearly is back in the game of being a frontier model creator here. three days before Spark Open, they actually launched Muse Wonder and Muse Wonder Full with wonder and full being two words there. So these are closed models. and Meta has been benchmarking those against GPT-6 Astra, Claude Opus 5.5 and on the artificial analysis index, Wonder scores 52 and Wonderful scores 56,

06:29 which is pretty good compared to Opus 5, which is only two points ahead at 58 now obviously sitting under those is a bunch of different models, but also includes the Muse Spark 1.3, which scores 48. And I think a lot of people, myself included, were hoping that the Muse Spark 1.3 was gonna be the model that was opened up for everyone But that really wasn’t the case, I guess. The open model is actually nine points behind that sort of mid-tier closed model and 17 points behind their best model now one of the issues you’re gonna

07:02 have with getting access to these is that they’re just not all out there. so Spark 1.3 is on both Meta’s API and on OpenRouter at a $1.25 in and $4.25 out They also have a version of the model which is much, much cheaper at 10 cents in and 20 cents out. But obviously the catch there is that Meta gets to train on your prompts and your outputs. So just as we’ve seen the Chinese labs basically give sweet deals if they

07:34 can train on your data, Meta’s doing basically the same thing here, and I guess that’s not a huge surprise. the Muse Wonder model is on the Meta API and Open Router, and that’s $4 in and $20 out, which turns out to be exactly the same cost of what Opus 5.5 is. But Wonder Full, remember that’s two words, Wonder Full, isn’t on any API. It’s only available inside Muse code on the $50 or the $200 a month plans

08:05 So I do think it is a bit rich for them to basically make all these claims about this model and yet nobody can sort of independently test them because there is no API available for people to actually try this out. interestingly, on LM Arena, they put the Wonder model through more than 5,000 real agent sessions, and it only came back at number 12, in their agent arena Which is kind of interesting because it came in at number two on their text leaderboard. But clearly it’s a strong model, Perhaps on agent work, it’s actually

08:37 not leading… Honestly, I’m not sure what to make of the model coming in 12 for agents, because one of the areas that Meta is clearly actually winning a lot at the moment is not benchmarks, it’s personal agents. So as I’m recording this, I’m in San Francisco at the moment, and personal agents seem to be the thing that everyone is talking about. Meta’s Muse app is clearly the personal agent with the most amount of momentum at the moment it launched in the US on September 8th. Within a couple of weeks, it’s been downloaded more than two and a half

09:09 million times And it’s actually been the number one app on the Apple App Store just showing, that they are getting real traction, not just with the AI developer community so the big question there is what can the app actually do? Well, it does things like reads your email, books things for you, fills in forms, it can pay on your behalf, it can negotiate for you. It’s interesting because they’re saying that each user gets their own isolated virtual machine with the agent and their data in it. They’ve got a free tier, a $20 per month tier, and $100 per month tier in there

09:44 and while apps like ChatGPT and Gemini are still far more widely used overall, in the specific sort of category of personal agents, it does seem like Muse is the one people are talking about Now, like I said before, unfortunately Muse doesn’t run on Spark Open, right? It’s on a customized version of Spark, and we don’t really know too much about exactly which versions it’s running, If it’s running any specific fine-tunes of it, that kind of thing But there has been some really interesting info that has come

10:14 out about how they actually built this. So Alexander Wang actually went through this on, the David Senra’s podcast. And I’ve gotta say, it’s one of the more useful explanations I’ve heard about how a lab actually ships an agent product and I’ve put the link to that in the description. so basically it started back in February with Alexander Wang and Nat Friedman spent weeks living with sort of OpenClaw agents, and then a whole bunch of that team trying out these OpenClaw agents. And their conclusion was basically that the magic was real, but these things just

10:47 didn’t really work reliably so over the next few weeks, they worked on prototyping both the agent system but also the models to be able to support that and it’s interesting that it looks like the Muse personal agent really has come from seven months of grinding With what they’re saying is a team of under 200 people across both the model and the product for it I found it really interesting how they described that they actually did this. So what they did was they listed more than 100 specific behaviors

11:18 and capabilities that the ideal personal agent needed to have. And they put all those into a spreadsheet, and every row got its own set of evals. They then trained the model against those evals And they basically had sort of like a green light, red light system of knowing, okay, with this particular model, do these evals pass or not? and very quickly they could see what percentages were actually working and what percentages didn’t And you can imagine that they were then basically

11:48 using lots of reinforcement learning environments to do post-training, et cetera, to get these things to work. They kept looking at the use cases that didn’t work and then updating for them And I thought it was interesting that he talked about each row had a launch blocking threshold and a good threshold so there were lots of checkpoints where there were green on about 80% of the rows and red on about 20, and they literally decided that wasn’t good enough He mentioned that the first checkpoint that was green on every row was that specifically trained version of Spark

12:20 1.3 and I kind of feel what he described in there is really a fantastic way of how to build any agentic product. If you’re building your own agent, you probably don’t have 100 behaviors written down with evals for each of them. if you could start with 20 and then grow it from there, write down what the agent has to do, build some evals for each one, and then just don’t ship until they’re passing all of the evals so like I said, the link for that is in the description All right, let’s jump in and have a look at the demos of how this model actually goes.

12:51 but before that, I have an ask for you. If you’re watching this and my other videos and you haven’t subscribed, please subscribe. I’m really trying to get the number up by the end of the year It just helps a lot in making these videos get out there and let more people see them Okay, so jumping into the demo here, what I’m gonna do is compare both the Muse Spark Open and the Muse Spark Wonder. so both these models I’m running on OpenRouter. so you can see OpenRouter here for Muse Spark Open, there are actually quite a number of providers already.

13:23 we’ve got Modal doing quite quickly, Together AI, Nebius, Fireworks, Base10, et cetera it is interesting to see in here that Meta themselves are not actually running a version of this all of them here have the same pricing, $1.25 in, $4.25 out for the open model. If we look at Muse Wonder, it’s quite a different story. There’s only one provider being, Meta themselves. this one’s the $4 in, $20 out. Again, in this day and age, that’s not really something that’s very cheap there.

13:54 and the other thing I’ll point out too is that Meta is actually retaining prompts even though they’re saying that they don’t use them for training Alright. So I’ve basically just set these up here, where I can use them in my playground and I kicked them off basically doing the SVG test. And the reason why was that people were sort of criticizing, the meta models for graphics and for doing websites and stuff like that. I’ve gotta say for the Wonder model, it’s not bad here if we, sort of look at the, pelican SVG. it’s certainly okay.

14:25 It’s definitely not as good as some of the other ones, although more and more I’m starting to feel that this is probably not that useful of a test because the model companies themselves are, sort of overfitting on it. what I do find really interesting in here is just how little thinking, Muse Wonder does. So you can see that basically it’s just got 55 tokens of thinking in here, which is kind of just rephrasing, what I’ve put in the prompt, for basically putting it together. And then it goes straight into generating, the SVG in here if we look at Muse Spark

14:57 Open, it’s a very different story here. So you can see for a start, Pelican not looking so great, bicycle perhaps not okay, and actually the feet on the pedals is pretty good. but fundamentally it seems like there’s a bunch of things going on there. And more importantly, I think that the thing here is just how much thinking this does, before it, actually goes to generate out. So you can see that it generated a lot more tokens, than the, Wonder version in there really I think that’s a sign of basically how they’ve been doing the

15:30 RL here, The Wonder version, while being a different pre-train And almost certainly a much bigger model, et cetera. It’s also had just a lot better reasoning and long chain of thought done on it, as opposed to the open one. my guess here is then I’ll kick off, just the long prompt for asking them to basically do a long essay Meta clearly doesn’t want to give away how they’re doing the long chain of thought, right? so if we look at something like this where we’ve got the, open model, we can see that this has got 1,000 tokens of, thinking, which is actually, open

16:05 thinking, that we’re getting there. and if we compare that, you can see that’s come to a stop at the end there In this case, we actually have gotten quite a substantial essay out. that’s actually not always the case. So you can see that if I come back and run this again, this time we’re getting, less thinking tokens. but quite a number of times when I’ve run the test, while the quality coming out, is quite good, The amounts just vary widely of where sometimes it’s only giving sort of 3,000 to 4,000 tokens back, sometimes it’s giving 7,000 tokens back.

16:36 so this is with the open model. If we look at Muse Wonder, we see again, we’re getting just really low, amounts of thinking in here, But pretty good, quality output going on here Now going back to Muse Spark Open, you can see here this time, we only got 1,800 words. Last time we got 4,900 words. it’s just very erratic and while of course I expect that it’s gonna be different, each time, I wouldn’t expect it to have such a huge range of difference. The variation there is really high where you can see for Muse Wonder,

17:08 it basically went through and gave us a full, essay back there. Now, I don’t know what the details are of what they’re actually giving us back of reasoning. I suspect that we’re getting, perhaps like a summarization of the reasoning tokens back. This is something that’s probably worth looking into a lot more if you’re gonna actually use the paid Muse Wonder model, in here that said, for things like the RAG Q&A, this does, a decent job going through it. It’s just using a lot of thinking tokens. If I try turning thinking down to low, I can actually get

17:40 it to use a lot less tokens. But that certainly does seem to affect the output, of going through this Alright. So running the sort of basic agentic tests, of going through it You’ll see that the Spark Open actually does a really good job at the function calling and dealing with different kinds of, things going on in the function calling here. So the basic tests, it’s passing, 10 out of 10, for this not surprisingly, the Muse Wonder also scores perfect, 10 out of 10 for these particular ones

18:12 The story does change slightly when we look at the, agentic sort of tests in here. the model’s still doing very well, but it tends to basically end up using a lot of tool calls, I found. so for this one, where plenty of other models would have done this, very quickly, this is doing lots and lots of tool calls to actually do this. and this could actually explain why it was not doing great on the Agent Arena ELO scores there, is that it was just probably taking, a long time. that said, it does pretty well on most of the other things.

18:44 dealing with broken tools, it did, did very well with that. it was able to recover and just keep going and stuff like that. all the things with distractors and stuff like that in the prompts, it did pretty well. the open model scored seven out of eight If we come across to the Muse Wonder, it has no problems with these, and it scores, an eight out of eight, here. And you’ll notice that like with the other one, it just kept going and going and going for this targeted edit, but this one basically just a handful of tool calls, and it was done.

19:14 so that sort of separates the quality of the open model, versus their frontier model the last one that I’ll look at is actually the HTML one. I don’t think this model is very good for these kind of tasks. I don’t think either of these models is necessarily great for this kind of task. the Muse Wonder, very good at logic, very good at those kind of things, perhaps less so at coding. but that doesn’t surprise me. I don’t think that’s what Meta is sort of going for If we look at the websites here, This one has actually turned out not bad.

19:45 So this is the Spark Open, version in there. If we look at the one for MuseSpark Wonder, it’s pretty plain, I gotta say, right? I don’t think it’s, that great, compared to some of the other models that I, tested in here. Looking at this, both of them are using sort of six and a half thousand tokens to actually get here. overall I think the model’s pretty good. My guess is that the people who are gonna end up using this, ’cause I don’t think honestly I would switch away from something like Qwen 3.8-Next, or even the Qwen, 3.8-27B.

20:18 Don’t forget this model is so big that very few people are gonna be able to run this, locally or actually do anything with it locally. Most people are gonna be running it off an API. And I kind of suspect the main usage of the open model, is really going to be US companies, perhaps, some companies in Europe, et cetera, who perhaps for political reasons don’t wanna use a Chinese model or certainly wanna be able to have plausible deniability. And this is something that I’m definitely seeing, a lot of companies not want

20:50 to admit that they’re using the Qwen models in production purely for what it seems to me to be political reasons All right, so where does this leave things? Overall, I think this is a really good sign. Meta is definitely in the open models game again. they’re not doing the whole sort of trick that OpenAI did of just put out one model and then, abandon it and not do anything more. they’ve now got back in, they’ve put out multiple models. Hopefully they’ll keep going and update those models. the Glimmer model, their 30B Apache licensed model, came out

21:20 in August and when it launched, Mark Zuckerberg said that they’d open the weights to Muse Spark. And two months later, here we are, right? it’s a different version from perhaps what we would like, in that we would like to have that 1.3 version but at least it’s a good sign here and with this past week being a big week for US open models generally, with both the Spark Open and Beam both landing weights in Hugging Face it’s great to see these companies entering the game. and certainly good for a lot of other languages, et cetera.

21:51 Because Meta’s got a lot of multilingual data and they’ve got certain countries and regions that they prioritize, perhaps like European languages, certain Southeast Asian languages, et cetera, that are often just not very strong in the Chinese models So while they’re landing this model quite a bit behind the Chinese open weight models, this model does make sense for a lot of people to actually try out here. I’d love to see their next open release at least get to the Spark 1.3 level if we could see an open model coming out from Meta every three months,

22:22 you’ve got to think that that’s going to be great for the whole ecosystem. Anyway, I’ve put the links to the Spark Open, to BEAM, to the David Senra interview that I talked about in the description. Let me know in the comments what you think about Meta’s split between the sort of open models and the closed models. and as always, if you found the video useful, please click like and subscribe, and I will talk to you in the next video. Bye for now