Decisions, Microsoft-Decision-1, Decisions | Merge Conflict ep. 536

Author: Merge Conflict Podcast · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Decisions, Microsoft-Decision-1, Decisions | Merge Conflict ep. 536 · Channel: Merge Conflict Podcast · 39:31 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI models, Local and open-source models

Summary

James and Frank discuss the rise of specialized decision models, starting with how they differ from general-purpose language models and why their speed and low cost are appealing. They review recent releases, especially Microsoft Decision 1, and discuss benchmarks, latency, image input, robustness, and the role of fine-tuning smaller open-source models. The hosts also consider how these models might be used for games, robotics, and local applications, while noting that they have not yet found a personal everyday use for them. They close by describing an OpenRouter skill that can route decision-making tasks from a larger model to a specialized decision model.

Key points

Decision models take context and return choices, such as multiple-choice answers, rather than generating long responses.
Their simple outputs can make them fast and inexpensive, and allow multiple questions to be answered using the same context.
The hosts describe recent decision-model releases from OpenAI, Microsoft, and other groups as evidence that the approach is quickly becoming widely available.
Microsoft Decision 1 is presented as a small, fast model, with the discussion noting that it is based on a fine-tuned 9-billion-parameter Qwen model.
Microsoft reported results across a range of benchmarks, while the hosts caution that private benchmarks are difficult to verify independently.
Microsoft Decision 1 does not accept image input, unlike some competing decision models; the hosts discuss representing game states as structured data instead.
The hosts highlight robustness as an important measure: small changes to an input should lead to small changes in the output.
Fine-tuning existing models can make specialized decision models easier to develop and run locally, though suitable training data remains a challenge.
An OpenRouter skill can help a larger language model delegate choice-based tasks to a decision model, potentially saving tokens and credits.

Chapters

Start	Chapter	Summary
00:00	Opening and selling a Mac mini	James and Frank introduce the episode before James talks through the uncertainty of selling an expensive Mac mini.
03:16	How decision models work	Frank explains how decision models use context to answer choice-based questions quickly and cheaply.
06:32	Potential uses and recent releases	The hosts consider possible applications such as games and robotics, then introduce the growing number of decision-model releases.
10:15	A wave of decision-model clones	Frank discusses the OpenAI and Microsoft releases and how the approach became something other model developers could quickly reproduce.
17:09	Commoditization, benchmarks, and speed	The hosts discuss decision models becoming interchangeable products and examine how Microsoft compared its model, including its latency.
21:22	Images and representing game states	They compare models with and without vision input and discuss representing a game’s state as structured text or JSON.
26:10	Fine-tuning and robustness	Frank explains Microsoft’s small fine-tuned model and highlights robustness as a useful engineering measure for decision models.
31:01	Specialized models and local fine-tuning	The hosts review other decision models and explain how fine-tuning can adapt existing models to specialized tasks and local use.
35:16	Running models locally and delegating decisions	They discuss fine-tuning smaller models, local inference, and an OpenRouter skill for handing decision tasks to a specialized model.
39:01	Wrap-up	The hosts thank each other, mention the next episode, and sign off.

Quotes

it’s fast, it’s cheap — 05:29

So what you’re describing is called commoditization. — 17:09

Computers can be fast if they’re programmed to be fast. — 20:19

small changes in input should produce small changes in output. — 28:19

The biggest problem in ML is data. — 35:48

Concepts and entities

Jev, Microsoft Decision-1, OpenAI, Microsoft, OpenRouter, Qwen, robustez, modelo de decisão, fine-tuning

Full transcript (original language)

00:00 Welcome back everyone to Merge Conflict, your weekly developer podcast talking about all things in software development and AI and all those goodies. I’m one of your hosts James Monttoagno with me with with me. I’m a little sick, Frank. With me as always, Mr. Frank Krueger. How’s it going, buddy? >> Oh, it’s going wonderful. Especially now that I know that you’re sick. I I’m doing wonderful. I’m healthy. The room is 120 degrees because my neural network keeps training. I had a nice fire going this morning. Um, spoilers. The Seahawks lost, so it’s a little depressing, but aside from that, life is good over here.

00:34 I have no decisions to make. I’m I’m just running on autopilot. Everything’s easy. >> Well, I’m making a few decisions. One, I am sick. Like, everything is just like happening. It’s like coming and it’s still going [laughter] and uh that’s not great, but I can at least talk and like I feel okay. I feel okay. It’s just it’s like uh up in there. Um, now we are we already recorded next week’s episode. So if people are like, oh, like you know, some big events happened last week, next week we’ll be having our big Microsoft

01:04 hardware reveal podcast. So stay tuned for that. Um, so we have a whole bunch of stuff to say on that and also the Windows announcements and so much more. So we really wanted to to to make sure that you’re liked, subscribed, and done all the stuff to be here. Uh, and I’m currently having a decision to where to sell my Mac and Mini Frank Krueger. So, that is >> tough one. What are your choices? Where where do people sell computers now? Do you just like put a lemonade stand out in front of your house? I mean, you’re in a pretty affluent town. You could probably just put a sign up and within a

01:36 day sell it, I I assume. >> Okay. So, I’ve sold stuff on I usually almost always sell stuff on Craigslist. That’s like my go-to. But now, no one’s on Craigslist and everyone’s on Facebook Marketplace, >> right? It’s like Facebook Marketplace is like the go-to, but it’s like all this junk on there. Um, however, I mostly just go to like eBay. >> Junk, James. >> Yeah, that’s true. Yeah, I I’d usually just go to eBay. So, I figure it’s like it’s it’s Facebook Marketplace local or or eBay. Now, eBay has a 13% fee, so it’s like

02:09 >> hefty, >> you know. However, like I’ve sold stuff on Facebook Marketplace and usually these are items that are, you know, 50 bucks, h 100 bucks, whatever. Not like a big deal. This would be like a 2500 to $3,000 purchase. So, the question is, do you >> like in here is a little Mac Mini that’s like $3,000, you know what I mean? Like and meet up with somebody at the local movie theater or gas station. You know, it feels a little >> a receipt, an invoice. Do you have a tax form? you fill out a warranty like, “Yeah, what services are you going to

02:42 provide? >> What if they say it’s broken and you’re like, no, it wasn’t broken. Now it’s broken.” It’s a tough It’s always scary selling something big, but I mean, it is roughly the price of a car. You’re basically [laughter] selling a car at this point. >> Yeah, the smart car was $4,500. So, I mean, just saying, it’s not that much more than that. >> Yeah, this thing has more RAM in it, that’s for sure. But uh yeah, lot lots of decisions to be had. But what we wanted to do uh this week is break down kind of almost a followup of a podcast

03:16 we did recently which was about a bunch of little tiny models. Like we talked a lot about Luna and things like that. However, um one of the models we talked about was Jev. And Jev had its like moment. You know, every model seems to have like a week of of its moment. And Jev specifically is a >> you I think you were like this is just a decision you know model. It’s making decisions in and out. Why don’t you give a a Jev TLDDR and then Jev update before we get into like Jev lookalikes.

03:47 >> Okay. Sure. Um so Jev is still a large language model because it takes in natural language. So the idea is you can still fill it with a context. give it a bunch of source code or give it all your company’s data or give it the collective works of Shakespeare or something and then you can ask it basically multiple choice questions and it doesn’t even have to be multiple it could be just yes no questions and things like that. So the nice thing about it though is it’s tuned for that scenario. So it’s tuned to just give you, you know, given

04:23 options A, B, C, and D. Choose which one of those options to output. The nice thing is there’s no like JSON parsing of the output or waiting for um the re recursion to happen to generate a long response. Basically, it’s fast. I mean, that’s one of the nice things about it. And then it gives you an answer to your question. And it also gives you a probability with it because that happens naturally with all these networks anyway. So they’re just throwing that in. And then the nice thing is because um its output is so simple, you can

04:55 actually ask it multiple questions all at once. And if your context is much bigger than your questions, then those questions become very cheap to have it answer. Uh it’s it’s cool. Well, it kind of took over the world because it’s just another way to use these advanced LLMs that we’ve really um fine-tuned. Oo, that’s can’t use those words. Specialized. >> We’ve refined. [laughter] >> Refined. >> Yes, we’ve refined how LLMs work for the past few years. And it this is just a

05:29 nice new mode, a nice new way to use LLMs um to just answer questions and make decisions and basically do it for cheap cuz you know what the all the other models are getting more and more expensive. So it was kind of nice to be like, “Hey, here’s a new model and it’s cheap.” And you’re like, “Oh, that’s nice. I do like cheap.” It’s been a while since I’ve heard the word cheap. Um but people like it because it’s fast, it’s cheap, and it it’s it’s one of those things where like constraints breed creativity. You’re like, well, all it can do is choose from a multiple choice output. So, hm, I wonder what questions I can pose to it. That’s

06:02 basically a multiple choice output. If we want to be reductive, we can say it’s like the old classifier networks that used to be out there. Hot or not, hot dog or not, >> [laughter] >> um, sentiment analysis. It’s like one of those, but a more general purpose and a more modern version. One of those that can do a lot of things. >> Yeah. Yeah. And I’ve seen a lot of people do a whole bunch of crazy things like you know drive like games, make like you know different forecasting like monitor like decisions like when to purchase tickets or whatever cuz it’s

06:32 all decision based right? So it’s like here’s decisions based on the historical part like when do we do these things XYZ instead of these more general models or models that are fine-tuned for uh coding for example um that are out there. So, have you yet to find a use case in your daily workflow to use a Jev type model? >> Absolutely not. >> Okay. [laughter] >> I want to like every day I wake up, I’m like, could I use Jev for this? I’m like, I’m too lazy. I’m just going to

07:02 pay $5,000 and ask [laughter] GPT7, Astra, whatever the heck, you know, I’m just going to do that. Um, that doesn’t mean I’m not still intrigued by them, but [gasps] for a person like me that’s been in the ML world for a while now, these classifiers are not new things. I know it’s new because people have only been introduced to AI through LLMs and natural language input and output. So, this is kind of new to them, but to me, this is kind of old tech. It’s an old

07:36 It’s an old thing with a new spin, which is cool. Totally get it. But um I never had a use for that old tech. So of course I’m not going to have a use for this new tech. But that’s not to say that like I’m not intrigued by things like um you said you’ve seen it do some things like a fun one that I see it do a lot is do like video game input. And if you think about video games, what are you doing? Well, you’re pressing if you’re a if you’re a keyboard one of those disgusting keyboard game players out there, you’re you’re pressing your

08:06 WASD buttons, your WD, and think of that as a choice. Do I press W? Do I press AD? Or do I press the space bar to shoot or something like that? Um, so there’s actually like kind of fun games and um this it’s almost AI. Look at that. but like NPC kind of AI and I think about it for robotics and things like that, but there’s problems with inputting data to them still. And there’s still this latency of you’re still having to go out to this foreign network over the internet and you’re going to have your

08:40 Starlink latency and you’re going to have the network providers latency and all that. So, it’s prevented me from doing anything serious with like robotics or anything, but I still think about it a lot because it is a creative use of a modern tech that we think about a lot. So, to answer your question, nope, have no use for them, but I’m still intrigued. I would say while you maybe have not found a use case and you have been intrigued at least many other people have been extremely intrigued and it seems like people are getting to work

09:10 because there’s not just like one decision model now there’s many decision models and it’s only been like two or 3 weeks since Jev was announced. So, what I really wanted to do today was talk about some of the things that have come out because I actually didn’t know like you sent me a note and I was I’ve heard of the OpenAI decisions, but I haven’t heard a lot of these other ones. But the big sort of news in the last week was Microsoft decision one that came out that really took at least my Twitter. you maybe there’s some bias there since I work at Microsoft, but like there was definitely some um really

09:43 um storm I would say of tweets that came out and uh all relatively extremely positive for being even cheaper and even faster and even higher quality on it. Uh, so I figured we’d dive a little bit deeper into this new world of decision models that we have access to, right? When we do find that use case, what it means today and what it means going forward. So Frank, what has actually happened in the last three weeks? Because you have a list, you have a bucket list and a whole b whole bunch of like [laughter] things

10:15 here. What is happening? >> Well, basically, everyone who knows how to train a model has released a new Jev clone. That’s what’s happened, James. Because it is it’s intriguing. ML people find it intriguing and we’re all like, “Well, I can do that.” Now, it’s one of those things. It’s like obvious in retrospect, but it takes a smart person to like push it through and be like, “Hey, here’s something cool.” And then we’re all like, “Yeah, that is cool, and I can do that, too.” Because now that you’ve demonstrated it, now I can be like, “Oh, I can do that, too.” And so, we have a lot of the um, oh, I can do

10:46 that, too. Come out. So, I think actually um Open AI beat Microsoft to the punch. Um Open AI released a decisions model and an API which is kind of nice because we haven’t had a new um API for these things for LLMs in a while. There’s been the chat completions API that we all use. And so when and the joke is the entire industry just copied OpenAI’s API. Um and now we’re all copying Open AI’s decisions API. Oh jeez.

11:18 >> Yeah, exactly. [laughter] So, OpenAI released a model. We’re not really sure what it’s based on, but it’s a little bit expensive. It’s it it’s it’s Open AI, you know, it’s going to be good. It’s going to be expensive. It’s going to be a tiny bit slow. That’s Open AI. Um, so the cool news, the one that you mentioned is Microsoft released one also. And I really appreciate Microsoft’s because mostly because they wrote a good blog entry talking about what they wanted to achieve, the pros and cons they saw in the world and um

11:48 and how there’s compares to everyone else’s because again everyone’s releasing these puppies now. And as an ML person, it was nice. It was a proper breakdown. I not even sure it’s definitely not a white paper. It’s not even really a technical report um because they didn’t disclose a lot of information, but there’s enough nuggets in it that we can see um how they were trying to compete. And I feel a little bit bad. I want to say this first place, like I feel a little bit bad for the Jev people because it’s terrible to be like,

12:20 “Hey, we worked on this thing for two years. Isn’t it cool?” And everyone’s like, “Yeah, that’s super cool.” And then everyone comes and copies it. It’s just a little disheartening. It just hurts a little bit. [laughter] >> Well, I mean Microsoft comes. I mean, I think to that point though, like that’s almost the world we live in to some extent. Not that like there’s there’s copying, but it’s like, hey, listen, like this is a viable use case and we are now able to >> to sprint like we’re sprinting right at at this pace. And you kind of see this sometimes when you know and there’s all

12:53 these articles it’s like oh and there’s been this software that’s exists for 20 years and then oh someone just like open sourced a thing or someone created some really cool thing like well I can just like clone it right and the question ends up coming down to >> okay anyone can at this point >> with enough tokens can clone a thing can do a thing like you can make it happen but what is the actual >> like business behind it what is the quality behind it what is the long-term viability of that piece of software or that product that’s out there. Like whether you’re buying a car, we talked

13:24 about this with the Slate truck, for example, it’s brand new, it’s a brand new truck. What is the long-term uh viability of that company of that? I was just talking to my uh friend who lives a few blocks away and they got solar on their on their house and we just got solar on on our house and um the company here has been around for a long time which doesn’t mean anything actually but at least all the equipment is all NPhase which has been around forever again which doesn’t mean anything and I was talking to them I was like oh what’s your setup and they’re like you know I don’t really know but

13:55 it’s it was this company and then you know they were here for a bit and then they got bought out and it’s from some other company and something and you’re like like so I don’t even know if something goes wrong what happens right so it is and it’s it’s and it’s the same thing when you go and buy a product in the store right you you know when you’re going to no shade on the dollar tree when I go to the Dollar Tree uh which I go to quite often I know what I’m getting myself into and and there’s there’s a variety of quality metrics at the Dollar Tree or Dollar General wherever you’re at right when I’m buying something for $1.50 50 compared to going to another $5. It doesn’t actually mean

14:30 that the $5 one is higher quality. What it means is like I’m going to do a little bit more deeper research on this one that is cheaper and and and faster and maybe it is just as good of quality as the $5 one. I’m paying retail markup because that one was first to market or for example that one has a brand name. If I buy a white t-shirt that’s a Hannes versus one that’s a a Nike versus one that’s a a Yeezy shirt, right? These are all different price points, but you know what it is at the end of the day? It’s a white t-shirt. Now, there’s a lot of different elements inside there. So, I think that this is why it’s really

15:02 interesting because we’re going to see more and more of this just like we’ve seen the models have a huge variance of of low, medium, and and high and extreme. you know, you get your your haikus, your your sonnetss, your your your opus, your fables, you get your your your Lunas, you get your your your Souls, you get your Astros, for example, like you’re talking about. And and now we have this, you know, not just here’s the major LLMs, but now we have a lot of

15:32 different companies that are putting out things and now we have these new specialized models that are also coming through the same vein, too. So we we get this long one just for example of like hey we talked about on the the local LLM one and and and what you’re running and also oh can I just use a Quen model or or a >> [snorts] >> um maybe an Nvidia model it’s like running on open router or whatever a Deepseek model right and I could be getting the same exact quality you know but at a different cost or I could be thinking I’m getting the same quality at

16:03 a cheaper cost you know what I mean it’s really it’s really we we have a lot of these different um uh decision branches in our head that we can now evaluate deeper if we will. [laughter] So anyways, I just think I think it’s interesting because with these models, you can also apply it to things in real life, right? Because you know, a good example is this. >> Think of um if you’re replacing the floors in your house, right? You know, if you’re buying carpet, there’s many carpets, but if you’re buying uh floors like hardwood floors, you have you have, you know, have like vinyl, luxury vinyl,

16:35 manufactured hardwood, actual hardwood. I’m sure there’s something beyond and beyond that, right? You know, there’s all these these different tiers of it and it doesn’t mean that one’s better than the other. It depends on your use case and your scenario, right? If you’re a family, you got a bunch of dogs, maybe that LVP is the best for you in this point in your life compared to something where in a high humidity area where maybe that that manufactured hardwood maybe isn’t exactly what you need. I’m just saying. So, any I’m just saying large slab of marble. I mean, you you really can’t that just one huge one huge

17:09 slab of marble marble. That’s it. That’s all you need is is all Frank Krueger needs for [laughter] the floor. Perfect. >> Yeah. So, what you’re describing is called commoditization. It’s just surprising how fast it happened with us. >> Um, and it it always reminds me of that Steve Jobs quote. I think he said it about Dropbox. He says, “This isn’t a company. This is a feature.” And I think um sadly for it’s hard [laughter] >> well open AI and Microsoft just proved that. They’re like hey um we have this

17:42 thing called Azure Foundation and guess what’s on Azure Foundation now? [laughter] Here’s your own decision model. Look at that. Look how quickly we spun it up. But I want to talk about why they were able to quickly spin it up also. Um a few quick things about it though. um they definitely focused on speed. So, it’s interesting to see what people start competing on when it becomes commoditized. Like, okay, what are we going to discuss here? And I think this is I like what Microsoft did because they basically established here are the um posts we’re going to use to measure

18:15 these things and they released a bunch of bench um benchmarks and we’ve had we have terrible benchmarks for LM right now, but we have a lot of them. [laughter] So, maybe with a lot of them that makes you know um quantity over quality. So there there’s something to there. But um Microsoft uh rated their model against 36 different benchmarks. Some public, some private. I hate private benchmarks cuz there’s no way to prove what it did. So

18:46 >> Microsoft, don’t mention your stupid private benchmarks. Like it’s throw those out. Like it’s public or GTFO, honestly. Um and anyone can make up a private benchmark and say that you win at it, you know? Come on. Come on, Microsoft. But, um, to their credit, they they competed with a bunch of public ones, and there are some standard public ones. Like, there’s one named Jev Bench, >> of course. >> Um, ironically, Jev is not on Jeb Bench. [laughter] >> Why? >> For one very good reason. Um, Jev is

19:18 still kind of a a beta kind of thing. Like, I have access to it, >> but part of the terms of services, you can’t benchmark us. [laughter] Oh >> yeah. So Microsoft’s in a tight spot. They can’t actually compare themselves against Jev. Um so if you want to compare Microsoft’s decision thing or OpenAI’s decision thing against Jev, you’re just going to have to do it on your own and you’re going to have to not release that data. >> So but what there is thankfully is a million other people have released their own decision models. So we do actually

19:49 have a nice baseline for what’s good and what’s bad on Devbench. And there’s a few that have trickled to the top already. But um so there is the quality there. That’s fun. But then there’s speed because speed’s a big deal. Speed is a part of this because some of the earliest demos of Jev were like, “Hey, it can control a robot. Hey, it can play a video game.” So this isn’t like your I tell Astra to do something or Opus to do something and now I go get a cup of

20:19 coffee for a half hour and watch it think in circles. This is I ask it something and bam, I get a response back. And that’s kind of the delightful part, too. You’re like, “Oh, yeah. Computers can be fast if they’re programmed to be fast.” >> Yeah, [laughter] >> it’s real refreshing sometimes. Another Steve Jobs quote. It’s like giving a glass of water to someone stuck in hell. [laughter] So, it’s um I got Steve Jobs on my mind for some reason. >> I see that. I see that. [laughter] [gasps] >> So, I uh So, it’s fast. Uh if you look at their chart, it’s at the top. It’s

20:50 fast. 125 milliseconds to get a response. You can get eight responses per second. That’s almost fast enough to play Unreal Tournament. No, >> but you know, we’re getting there. >> Well, six 60 frames per second. That means you’re processing, you know, oh, I guess it would be >> Yeah, second. It’s actually pretty fast. Like, um, >> it’s pretty good. >> Yeah, >> that’s faster than you can press a keystroke. That’s faster than you can press >> Yeah. >> make a decision to press something. You’re shooting. I wonder what they call

21:22 >> Frankenstein. [laughter] >> Can I play Doom? Yeah, I bet. You know, I saw >> I think that was one of the demos. >> I saw it playing Space Invaders and I saw it playing Tetris obviously as a canonical. >> Okay. >> Okay. Let’s talk about because these decision models. So, you do give them context and context is usually natural language, but we’ve advanced in the LL LLM world where you can actually give images to LLMs. not like here’s a screenshot of my app, tell me how to improve my app, blah blah blah.

21:53 >> Um, so the Jev can take in images, open AAI decisions can take in images, Microsoft one cannot. >> So that’s um one little post there. Um so like they have great numbers and good benchmark scores, but it is an important distinction like it can’t do video. So then you’re like, “Okay, but one of their benchmarks that they say they did awesome on was playing Space Invaders.” So what’s going on, James? What do you think they’re doing?

22:26 >> Well, you would just take the Well, you know, you would take the grid like every snapshot in time, every frame has, you know, some sort of backing data to it. So you pass that data into the decision tree and represent the invaders, the ship, the asteroids that you’re behind or the bases you’re behind, whatever it is. And you would also then send in all of the data such as uh it shots in your shots. So you should be able to in near in real time be able to pass in all of

22:57 that data back and forth to make decisions on what to move left, right, shoot on. I would assume given the context that you’re there. In fact, I have to imagine that’s faster than sending in images in which it has to then parse through because the confines so for example the confines of space invaders are like deter most I would say deterministic it’s it’s >> and even if let’s say you were playing

23:28 um another game such as Fortnite and the randomness which There’s a human control here is that frame snapshot the snapshot of the data that comes through obviously would be much larger but that is still a defined snapshot in time of the the state of the world because the game loop itself you know the game loop is processing that data every frame so you would just be passing that data off to I guess Mai or Microsoft decision one in

23:59 this case >> is that is that what they’re doing I assume >> pretty much the biggest problem with this benchmark is it’s a private benchmark and it’s not released to the public. So, we all have to kind of guess what they did. So, your guess is honestly as good as anyone else’s guess. But the fact is you do have to take this graphical representation and represent it as text in some form or another. >> The dumbest possible worst way to do that would be actually do like an asky conversion of the graphical screen. You know how people do like image viewers in

24:29 the terminal? Don’t do that. They’re not doing that. They better not be doing that because that’s dumb. These models are not trained on that. Models are trained on natural language number one and JSON. Honestly, JSON is our universal data representation. So, most likely what they’re saying is here’s a list of aliens. Here are maybe their spatial coordinates XY position. Here’s the player position. Here are the bullets moving. I I forget all the parameters, but yeah, you got to capture your state space. So if I wanted to do a robot, I would have to capture my state space of the robot into some kind of

25:04 structured data, most likely JSON and not just natural language. And so everyone’s assuming that’s vaguely how they’re doing it. Unfortunately, that means it can’t do the simple things of like play an Atari game. Um whereas the other ones can because they do accept vision input. And I do want to push back a tiny bit on what you said about um text is guaranteed to be smaller than an image. It’s not actually guaranteed because um >> these numbers vary all over the place, but normally these networks will take an

25:37 image, break it into like 32 pixel x 32 pixel cells. Each one of those 32x 32 pixel cells becomes a single token. So your resolution of your image will tell you how many tokens you have. Yes. So a high-res image, if you’re capturing every little bit of it, yes, that absolutely can take a bunch of tokens. But for like an Atari screen, it actually is not actually too many tokens most likely. Um, but that’s all guess work. I just wanted to bring out the point that um, our large language models, language models actually do more

26:10 than just language. They actually do vision also. >> And honestly, it’s a little bit frustrating. I’m jumping a little to the end here, but um, Microsoft was able to release this thing because all it really is is a fine-tune of an open-source model. It’s a fine-tune of the beautiful, lovely Quen model that I love because it runs on my cute little RTX 390. I’m g ding. I I think it should get a bell. Like I every time I mention on the podcast [laughter] >> cuz I’m in love with it. And it’s actually a small Quinn. It’s actually the 9 billion parameter Quen. So you can

26:43 easily run the model. You could run Microsoft decision one on your own machine easily if they would release it publicly. But they haven’t. It’s just private. But there are plenty of um these decision models on hugging face if you want to go looking around and all that and maybe we’ll include some links to some of them on here. So I just want to put it out there that one of the reasons they’re getting the speed is because they’re it’s just a 9 billion parameter model and they threw out the vision side of it. So they’re just doing

27:14 the tech side of it and it’s a 9 billion parameter model. So Microsoft making good engineering decisions also benefiting from open weights Microsoft. So maybe you should be contributing more open weights to the world also. You know, if you take something, you should give something back. Just saying. Hashtag just saying. Um [laughter] so, uh the reason it’s fast is because it’s it’s a smaller model. It just is. >> But um one thing I really appreciate in their blog post was how they spent a lot of time ensuring robustness.

27:48 And this is something that we I I haven’t seen too many other people test. And this is something that um I think about a lot with these models of like if you just change a small amount of wording in something, how much is that actually affecting the output? >> So this is actually an engineering definition in control systems. What we So this is a different field. This is my field control systems. But we have a pretty strict definition of robustness and it’s very simple. It’s small changes in input should produce small changes in

28:19 output. It just means it’s not a twitchy thing. You know, you you change a little phrase here or there, you’re still going to get roughly the same answer out of the machine. And Microsoft basically turned that into a benchmark or whatever. They use that as a guiding principle for how to train this thing. So, they tested like does it matter what order you give um I said they’re all multiple choice questions. So, does it matter what order you give the answers to it? Does it matter if you rephrase the input a little bit? And to their

28:52 credit, their model is not the best at this. Other models outperform them. The models that outperform them are much bigger and much slower. So you would kind of assume that, but it’s it’s nice to see um proper engineering benchmarks because I feel like a lot of the benchmarks we see in the world are horrendous [laughter] and have absolutely no engineering foundation to them. This is the first time I’ve seen someone actually have like a serious benchmark and it’s it’s robustness. This is a thing I care quite a bit about cuz

29:25 I’ve trained plenty of ML models where as long as you don’t mess with their inputs too much, they produce great outputs. But the moment you give them a variation on the input or something it hasn’t seen before, it just acts like an idiot. And so to their credit, um, they really pushed on that. And also in robustness is a variety of topics like is it did they train it to predict the stock market? Probably not. But, you know, that’s that’s one universe of things. Did they train it to make good life decisions? You know, should I break

29:58 up with Sally? You know, that’s that’s kind of an important question. Do we have a benchmark for that? And [gasps] so just be [laughter] Yeah, exactly. Just because it’s a decision model doesn’t mean it’s a simple problem. In fact, it’s an incredibly hard problem if you’re just saying like there is a universe of things you can make decisions about. And the training data is going to be honestly very small. You It’s hard to get a good random sampling of every possible decision humans make ever. Yeah,

30:28 >> that is a very large universe of open-ended questions. Um, so it was fun to see them tackle that and acknowledge that. Um, >> in some ways it’s easier than LLMs because you’re not asking it to, you know, generate a book or anything. But in the same thing, it it really shows you how hard LLMs have it too cuz they are generating a book in this universe of possibilities. And if you just say, “Well, my universe is the universe of the possibilities plus this multiple choice thing.” You’re like, “Well, the input is still the universe. It’s still big and still difficult.” So, it’s fun.

31:01 It’s just a good blog entry, by the way. Also just well written. So, I hope everyone goes and reads it. >> We’ll put a link in the show notes, of course. And yeah, I think what’s what’s nice is like when you’re going through, you’ll see like obviously Jev, Quietette, Rune, H2O, Lightning, obviously, Luna Decision, which we talked about Deck. There’s all these different ones that are in here. So you get to learn about all the other sort of decision models that are out there that are in these top of the the dev bench that are there too. And I would also point out because I’m not sure about Luna exactly if it’s fine-tuned off of Luna. I assume maybe or something, but

31:34 you learn in the and the thing I was going to ask, but then you you have gone ahead and solved it for me, which is how did these things come about so fast? And what you end up learning as you go through is that many of them are fine-tuned off of other models that are out there. And you know what’s the important part here to to your part? Obviously I agree with you on on um you know giving and taking but also uh transparency is also really important too and many people uh called out uh right after the post went out um that it

32:06 was you know fine-tuned on quen um 3.59B and well that was right in the blog post it was literally like right in the beginning it’s like here and here’s what it is and here’s how we got here right >> yeah and you see quadette is on gemma 4 31 billion parameter fine tune model and all these other ones that are here too so it’s not to say that that is the only way to get there. But that that does means that you’re able to, you know, fine-tune on top of these models that are out there to do very specific things. And that is the the world, you know, that that we’re we’re we’re leading into to to to create some of

32:37 these things faster than ever, right? Um, so it’ll be really interesting to see how this evolves. I do think that it’s uh I think the only thing the only thing I mean I actually the one part in the blog post that I think is funny but also is a good I’m glad they I’m glad that they included it even though it seems a little bit like shade which is they have in a lot of these graphs that that the team has on here it is also against GPT6 soul >> and the reason for that is not to like throw a bunch of shade on GPT6 soul but

33:08 it is to point out that GPT6ole is not a decision ision model. So if you are specifically trying to use GBT6 soul for decision making, then you are dramatically going to be getting slower um much more expensive results coming back and and maybe not even at the accuracy that that you desire. So something to think about there. But so I am glad that they did include it. I think it’s funny because it is the one that is like the biggest bar. Um which is funny. I wish they put more big LLMs in there, you know, honestly because I

33:41 want to know like am I out competing the bigger models like >> true small models >> and >> and the reason it is out competing them is because all these big models are fine-tuned to to do a certain kind of task and that task is usually chat with a human and run agent junk. [laughter] These are the two fine-tunes that we roughly do on all these models. Now, so the whole the the the thing everyone liked about Jeff is like, “Oh, what if we fine-tune on something else?” Because

34:14 these are fine-tuned. And that’s important because we don’t want to train these things from scratch. They have to understand all of English, all of German, all of Arabic, all of Chinese. You know, it takes a lot of money and resources to get them to understand all of that. The question is, okay, now that they understand all of that or whatever to varying degrees, throw in your own verb there. Um, what can we do with the outputs? How can we um make the outputs more efficient? What are more interesting tasks we can do with these

34:45 outputs? And I think that’s why the ML people are also excited about this. It’s not just marketing hype. We’re like, oh yeah, we have these very sophisticated things. Let’s fine-tune them to do something other than people please in a chat interface. Because honestly, [laughter] as much as we all love chat GPT, like the chat interface, >> it’s cute. It’s cute, but it’s it’s it’s going to go away. We we all know it’s going to go away eventually. Um, everything will have a little bit of a chat interface. And the joke is these

35:16 models can do the chat thing. It’s just as you fine-tune them to be question and answer things, they’re just going to get worse at chat because we’re moving them away from that fine-tune. >> Yeah, makes sense. Well, it’s very cool. It’s cool to to get down into it. And uh if everyone else is fine-tuning these, can why don’t you just create one, Frank? >> Exactly. Well, because my my GPU is currently busy doing something else right now. Have I mentioned how hot it is in this room? [laughter] >> Good. It’s cold. I got a jacket on. >> But yeah, um especially because it’s a 9

35:48 billion parameter model, like you can easily go to like Google Collab. You don’t even need a GPU and in probably just a day or two fine-tune one of these puppies. The biggest problem in ML is data. So if you have a big question and answer kind of database and I mean big, you need a big one. You can also go download some examples like on Hogging Face, you can fine-tune these things yourself, which is really cool because they’re small enough they can be local models. So you might have some trouble running like Quen 27 billion parameters

36:22 with your small amount of video RAM, but one a 9 billion parameter model. Oh, that’s runnable. A lot of people can run that thing. And then you can run your own decision maker locally. And that’s when you can really start um don’t cheat at video games people, but that’s when you can really start cheating at video games [laughter] and doing that because it’s all just kind of running locally. You’re not paying for external servers and all that. Don’t tell Azure Foundry I said that. I try to stay in their good graces. But the nice thing about small models being fine-tuned is that you can

36:53 run them locally. >> Well, you also have Foundry local. I’m just saying. So, you can just pull down those models. You’re totally good to go. Now, it’s all there. I mean, we’ll we’ll hear quite a bit next week when we follow up on the Surface and Windows event talking about local models. We’ll really deep dive uh into these new devices that have 128 gigs of RAM and an RTX uh Spark. So, pretty excited about that, too. But I do think that this is good almost lead up to it because >> um well we were not going to talk about the devices until next week but one thing that is really important is sort

37:24 of this even more and more local models running and making decisions on your behalf. So we’re only going to get better from here smaller tighter faster um as these continue to go. So I do think that this makes a lot of a lot of sense and like you’re saying to be able to run these locally just just and why not? Yeah. And and the I one thing I want to bring up and this goes back to your question of have I personally found a use for these uh and the answer again is no. But um Open Router has released a skill a skill.md.

37:59 By the way, Open Router is fully in on these things. So they they have the same API endpoint as Open AI. So you can start um if you have an Open Router account, you can start hitting these. You can even hit uh Microsoft decision with it because that model on Azure um open router will go and hit that puppy. So uh there’s a skill file, a markdown file that you can import and it tells your big fat slow LLM, hey, if you have a decision point to make, here’s a way

38:29 that you can access a decision model that and here fill it with this context. give it these multiple choice questions and then let that thing burn on it. So, another good way to just preserve your own tokens and your own credits and all that kind of stuff. So, I I plan on giving this skill um a little bit of uh a little run, see how it goes and all that. So, I just I’m excited for that skill because otherwise I didn’t feel like building a whole harness. You know, there aren’t a lot of harnesses built around this thing because we’re all

39:01 still playing with big LM harnesses at the moment. No, I’ll put all the links in the show notes to the decision one blog and to that open router scale too and I’ll link to a few other things in there. Uh, which super cool. But Frank, thanks for breaking this down for me. I appreciate you did all the hard work this week and you have a nice note here of all the different uh different different links too. So, I really appreciate that. >> Yeah, I I love giving you links because I I just assume that you’re not going to open any of them, but I think you actually read some of them this time. >> I [laughter] did. I did my best. I did my best. Uh, well, that’s going to do it

39:31 for this week’s Merge Conflict. Do like, subscribe, do all the things. Next week we have a banger uh covering the Windows and Surface event. We’re going to get deep into the Ultra and the Devbox. So, you do not want to miss that one. But that’s going to do it for this week’s merge conflict. Decision made. Frank Krueger. Boom. One in the chapter. I’m James Monttoagno. >> And I’m Frank Krueger. Thanks for watching and listening. Peace.