Microsoft Joins the Local AI Push

Author: Sam Witteveen · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Microsoft Joins the Local AI Push · Channel: Sam Witteveen · 13:25 · Watch on YouTube

Topics: AI agents, AI infrastructure and hardware, Local and open-source models

Summary

Microsoft presented a vision for Windows that routes AI tasks between local models and the cloud, which it calls hybrid intelligence. A GitHub Copilot demo showed local models handling much of a multi-agent coding task, including most of the 1.6 million tokens processed. The event also featured highly quantized models, llama.cpp support in Windows ML, execution containers for agent security, and new RTX Spark hardware. The video’s central takeaway is that Microsoft’s support for local AI could help make running models on personal devices more commonplace, though model quality and routing behavior still need to be tested.

Key points

Microsoft described hybrid intelligence as routing AI work between local models and the cloud.
In the Copilot demo, the model picker’s auto setting could choose a local model as well as cloud models.
A harder coding task used a cloud model for planning and local sub-agents for much of the reading, accounting for most of the 1.6 million input tokens.
The local Microsoft AI code model was presented as a three-bit quantized model with a 256K context window.
Microsoft also discussed running Nemotron at two bits and DeepSeek 4 locally at an average of 1.6 bits; the presenter questioned how well quality, long-context performance, and tool calls would hold up.
Windows ML is adding llama.cpp, which could make open GGUF models and hardware compatibility more accessible to Windows apps.
Microsoft execution containers (MXC) provide a sandbox where organizations can set policies for what agents can access, with agent actions attributed to the agent.
The RTX Spark hardware was presented with up to 128 GB of unified memory and full CUDA support, with the video noting a starting price of around $2,600.
The presenter sees Microsoft’s backing as an important step toward local AI becoming normal rather than exceptional.

Chapters

Start	Chapter	Summary
00:00	Microsoft’s local AI demo and hybrid intelligence	The video introduces Microsoft’s event and explains how Windows may route AI work between local models and the cloud.
03:44	Local models and the limits of quantization	The presenter reviews the models Microsoft showed and raises questions about quality, long context, and reliable tool use at very low bit rates.
08:05	Windows ML, llama.cpp, and agent security	The video covers llama.cpp support in Windows ML, the possibility of using custom local models, and execution containers for controlling agents.
11:12	RTX Spark hardware and pricing	The presenter discusses the RTX Spark’s capabilities, expected cost, and how it may compare with other local AI hardware.
12:17	Microsoft’s broader commitment to local AI	The video concludes that Microsoft’s support could help make local AI more widely adopted and invites viewers to share their views.

Quotes

So Microsoft is calling this hybrid intelligence. — 01:03

1.6 million in and a bit over 10,000 out. — 02:37

It means open models on day one. — 08:37

Every action the agent takes is attributed to the agent, not to you. — 10:09

making all of this the norm rather than the exception. — 12:53

Concepts and entities

hybrid intelligence, GitHub Copilot, Microsoft AI code 1.1 flash, Nemotron, DeepSeek V4, llama.cpp, Windows ML, Microsoft Execution Containers (MXC), RTX Spark, Microsoft

Full transcript (original language)

00:00 Okay, so yesterday Microsoft held their Windows and Surface event in San Francisco. In this video I want to talk about some of the interesting things that they showed there. So they had an interesting section where they showed 1.6 million tokens. That was how much one coding agent session sent into a model during one of the demos. And they made it clear that the bill for that was zero. Not because Microsoft was picking up the tab or anything like that, but because the model was running on the laptop itself. And I got to say that it didn’t look like a toy demo either. The agent was investigating some GitHub

00:32 issues on the Windows terminal code base. And maybe even the most interesting part here was that they claimed that it wasn’t actually even the developer who decided it should run locally. It was the tool. And while most of the coverage that I’ve seen about this event really focused on the Surface Laptop Ultra and the Nvidia RTX Spark chip here, both of those looked really cool. Although definitely a steep price tag there. A lot of that was actually announced back in June. Really the big thing here I guess is the shipping dates, the pricing, that kind of thing. But for me the interesting thing here

01:03 was what is Windows actually going to do with this powerful processor? And it seems one of the key things that it’s going to do is decide which of your tokens run on your machine and which of them go to the cloud. So Microsoft is calling this hybrid intelligence. They had the person who leads the Windows team actually described it as intelligent routing. Basically the models, the runtime, and the chip set all working together. So let’s start with the router that’s the piece that actually made the demo work. So in the

01:34 GitHub Copilot demo, the presenter had the model picker set to auto. And the new version of auto doesn’t just choose between cloud models anymore. It can also choose a local model. So when she gave it a pretty basic sort of clean up task on the Windows terminal code base, it actually sent that to the Microsoft AI code flash model that was running on the laptop. And you could actually see the GPU ramping up on the screen. She even pointed out that you could put the whole machine into airplane mode and it would still keep going. After that

02:07 though, she then gave it something harder to go through all the issues that were tagged needs repo. When she did that, what happened there was a whole different thing. So basically a cloud model, it was one of the open AI models GPT-6 or something, did the planning, then three sub agents ran locally on that Microsoft AI code model. So this is a very standard pattern we’ve seen, right? The expensive model does the thinking, the local cheap models do the bulk reading. And that’s where we get to the actual 1.6 million tokens coming in

02:37 here, right? The majority of those tokens, in fact, I think all but maybe like 10,000 or so, were running locally. And that’s where the 1.6 million tokens come from. 1.6 million in and a bit over 10,000 out. That’s roughly 150 to 1. And more and more this is the kind of pattern that you see if you’re sort of building things with multiple agents, where the context window keeps getting longer and you keep sending it through to generate something small out. And I thought this this was really interesting to see basically how they handled this

03:11 routing between the local model and the cloud model. Because I do think that this is something we’re going to see happen more and more going forward. While it’s not in the interest of companies like OpenAI or Anthropic to really talk about that, it is in the interest of any of the hardware makers and even companies like Microsoft who are making some of the software and the hardware, not just the actual models. So this is going to ship in the GitHub Copilot app on October 15th. And it seems that that app is getting both local context, local actions, local models over the next few months. And

03:44 it’s really going to be something that’s very interesting to sort of look out for. Because we know that every router lives or dies by what happens when it gets it wrong. If the locomotive can’t handle the task it was sent, does it then know that and send it back to the cloud, or do you just get a worse answer and never actually know what failed and what didn’t work out the way you wanted it. Unfortunately, that wasn’t something that they showed here, but it’s certainly going to be interesting to see when we’ve seen things like the whole model router idea with things like PAIR from NVIDIA through to other projects

04:16 and ideas that people have put out there of how this is actually going to work. All right, so what did they actually say runs locally? So, first off, this was the Microsoft AI code 1.1 flash model. This is the coding model that Microsoft introduced at the Build conference. And the thing that kind of amazes me here is that they’ve taken it down to three bits, which has cut the size by close to 80%. And they’ve got it running on these devices with a 256K context window fully local in there. Now, I’ve got to say

04:49 that I’m on the fence a lot about a three-bit quantized version of a model, especially when you’re trying to use it for something like coding. I think time’s going to tell either they’ve worked out something really good that they’re actually doing there, or this is going to be calling the cloud for any sort of coding task that’s remotely difficult. And that same idea carries into the second model, right? So, they talked about a new coming Nemotron model that runs at two bits in about 20 gigs of memory. And interestingly, they’re pitching that as a good fit for the RTX Spark. So, don’t forget the RTX Spark

05:24 now is not just going to come in a 128 gig version. We’ve already seen the DGX Sparks include a 64 gig version, and it seems that the RTX Spark might even have variations that go down to a 24 gig version. Now, the third one that they showed, and I got to say I’m a bit suspicious about this one. I really hope I’m proven wrong, but this was them running DeepSeek 4 locally at 1.6 bits in about 60 gigs of memory. The line that they used on stage was something along the lines of that a year ago this capability was only available from the

05:58 Frontier cloud models. And for most people that is kind of true. Now you can run DeepSeek 4.1 on multiple DGXs. I actually do that. It’s one of my favorite models that I really like. But it’s going to be really interesting to see what a 1.6 bit version of that model actually is like. Now to be fair, they’re actually saying that the 1.6 bits isn’t average. So in practice that means some layers are going to be kept at higher precision and a lot of them are going to be pushed very low. So you try and keep the layers that are the

06:28 most important at higher precision levels and then layers that don’t get activated as much, you then go for lower precision on those. The real question is though, how much of that quality actually survives? So the claim on stage was that it outperforms on coding and reasoning benchmarks. But they didn’t really clarify what that meant. And I would say that there really sort of three key things here. One, does the coding pass rate hold up against a higher precision version of the same model? I’m very suspicious of whether that actually does. Secondly, does it fall apart at long context? And then

07:01 thirdly, if you’ve got sort of complicated use of tools, does it actually still produce clean tool calls? Because structured output is often the first thing to break when you push quantization too far. And for any sort of agentic task, a broken tool call is generally going to be a failed task. The other thing you got to remember too is that if you’re really going for a sort of long context like 256K, that’s not going to be free, right? At that length your KV cache takes up a lot of memory on top of the actual weight.

07:34 So there’s a lot of nice tricks of where people quantize that KV cache where that’s down to 8 bit, 4 bit, perhaps even lower than that now. But my guess is you’re probably going to need one of the bigger versions of this machine with a large amount of memory to be able to do that. Now, another interesting point just to raise here that I haven’t really talked about on the channel before is that it’s a very interesting debate of do you go for a bigger model with a lower quantization rate, or do you go for a higher quantization rate with a

08:05 smaller model? And I personally I’ve been seeing as long as your quantizations don’t go below 4-bit, actually going for the bigger model usually gives you a better win. But, that’s mostly me saying that anecdotally. I can’t recall off the top of my head any sort of papers or anything that have been talking about that. All right. So, to be able to run these models on the machine, you’re going to need a runtime. And one of the announcements that they had, I think, is just a clear win here was the whole idea of llama.cpp inside of Windows ML. So, Windows ML is Microsoft’s runtime for

08:37 deploying models across GPUs, NPUs, CPUs. And the cool thing here is that they’re bringing llama.cpp into it. So, why does that matter? Basically, it’s going to allow for a lot of compatibility. It means open models on day one. So, when a new model drops and there’s a GGUF quant that shows up a few hours later, you’re not having to wait for Microsoft to package it. And that’s especially true if you’re not just using the normal versions, perhaps you’re using an uncensored version, a specifically fine-tuned version, which

09:08 Microsoft is never going to convert for their sort of system. So, having llama.cpp in there now gives you that ability to just pick the models that you want to run and actually run them. The other cool thing here is that if you are building a Windows app that ships with a local model, you’ve now got this one runtime that handles whatever the hardware your user happens to have. I’m guessing as long as that hardware probably is reasonably recent. The thing I I know yet, and I’ll be looking into, is how you can actually plug your own

09:38 local model into the Copilot router here or whether it’s only the Microsoft picks. I got to say, if you could plug in any model there, this could be a huge win for Microsoft in that you could just basically decide, “Oh, I want Qwen 4 35B mixture of expert model plugged into my OS and it just gets automatically used by a whole bunch of different apps or something.” All right, the next thing up was once you’ve got capable local models here is obviously how do you stop them

10:09 from doing things that you don’t want them to do. And in many ways what Microsoft announced here was their version of Nvidia’s open shell or their version of Docker sandboxes, etc. And this is called the Microsoft execution containers, MXC. It’s basically a runtime sandbox that organizations can set policies for what an agent can access, and then Windows itself enforces them while the agent runs. The bit that I think here is really quite interesting is the whole thing around agent identity. Every action the agent takes

10:41 is attributed to the agent, not to you. And so you can imagine that this helps both in sort of tracing things, but also when you’re actually working on something, if the agent actually goes to do something, you can actually block it and then it loses access to take that action, but you as the logged-in user still can have access. All right, lastly up I guess was the hardware itself and I’m going to say Microsoft’s done a really nice job at design here. These look really impressive. And don’t forget, these are just the first of the

11:12 RTX sparks that we’re seeing coming out. These machines are up to 128 gigs of unified memory, they have full CUDA support, and it’s all running on a Windows laptop. You don’t have to go into Linux to actually do this. So reading some of the coverage, I think Engadget puts this GPU at around about an RTX 5070 level with 20 CPU cores and it won’t draw more than 80 watts. Now, the prices for this started I think around about $2,600, but obviously the more memory that you put into this, the more that price goes

11:44 up. And I guess there’s really nothing that Microsoft or Nvidia can really do to prevent that. The whole memory and RAM issues that are going on at the moment is just making all of this hardware get more and more expensive. Along with a laptop version, they’ve also got a desktop hearing there. And interestingly, they even teased a Windows version of the DGX Station coming later on this year. So, that’s going to be interesting to see. I think more for unlocking the ability for other people to be able to actually use these. So, hopefully at some point I will get my hands on an RTX Spark and be able to

12:17 run some tests on it myself and see just how these things perform and work out for which group of people is this going to be a viable alternative to a DGX Spark, etc. Anyway, so to pull it all together, I thought this was actually interesting than usual event from Microsoft because it really planted a flag of them saying that you’re going to need to be able to run models locally and not just use the cloud. And I think by them getting behind it really is an important step in the direction that local AI is going. So, no longer do we just have sort of enthusiasts or ML people or even just startups sort of

12:53 pushing for local AI now. And while we’ve had Nvidia clearly pushing for local AI this year, having Microsoft join that bandwagon is definitely a step forward in making all of this the norm rather than the exception. So, anyway, let me know in the comments what you think. What’s your take on the RTX Spark? How do you feel about Windows adopting Llama.cpp rather than just making their own protocol or their own way to do it? I’m really interested to hear from people about this one. I do think Microsoft’s gone in an interesting direction here. Anyway, as always, if you enjoyed the video, please click like

13:25 and subscribe and I will talk to you in the next video. Bye for now.