Apple Hasn’t Put an LLM on Your Watch. So I Did.

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Apple Hasn’t Put an LLM on Your Watch. So I Did. · Channel: Better Stack · 11:27 · Watch on YouTube

Topics: AI agents, Model inference and optimization, Local and open-source models, Multimodal models

Summary

The video demonstrates a 90-million-parameter Falcon H1 model running fully offline on an Apple Watch Series 6, with local text generation averaging around 15 tokens per second. The creator explains why Core ML is not suitable for Falcon H1’s Mamba 2 state-space layers and how llama.cpp can be built for watchOS despite its lack of official build support. The watch’s arm64_32 architecture retains full 64-bit ARM instructions but uses 32-bit memory addresses, limiting addressable memory to about 2 GB. In demos, the app answers questions and uses Wikipedia and weather tools, though generation slows during longer responses; the creator concludes that even older watches can run small local models.

Key points

Falcon H1, a 90-million-parameter model, runs locally and offline on an Apple Watch Series 6.
The watch averages around 15 tokens per second, and one demo reaches 24 tokens per second.
The Series 6 performed about 50 times faster than the first-generation Raspberry Pi in the creator’s earlier test.
Core ML is unsuitable for this model because it lacks a state-space building block for Falcon H1’s Mamba 2 layers.
The creator got llama.cpp working on watchOS with a handful of build flags and a one-line source guard.
The watch’s arm64_32 target retains the full ARM instruction set, including NEON, but uses 32-bit memory addresses.
Those 32-bit addresses limit the watch to addressing about 2 GB of memory, making large models impractical.
The demo app includes Falcon H1 and a 135-million-parameter Small M2 model, plus Wikipedia and weather tool calls.
Longer generations slow down, with the depth-first search explanation dropping from about 20 to about 10 tokens per second.

Chapters

Start	Chapter	Summary
00:00	An offline LLM on Apple Watch	The creator introduces a locally running Falcon H1 model on a Series 6 and outlines the plan to explain setup, deployment, and performance.
01:02	Why Core ML is not the fit	After comparing the watch with an earlier Raspberry Pi test, the creator explains how Falcon H1’s Mamba 2 layers conflict with Core ML’s capabilities.
04:09	Building llama.cpp for watchOS	The creator describes getting llama.cpp to work on watchOS and explains the implications of its arm64_32 memory addressing.
06:43	Models, tools, and live demos	The app is introduced with two small models and tool calls, then tested on factual questions, weather, and a longer algorithm explanation.
09:51	Results and what they suggest	The creator sums up the successful local inference demo and asks why device makers have not shipped general-purpose on-device LLMs.

Quotes

There’s no streaming over the cloud involved here. — 00:00

The answer turned out to be about 50 times faster. — 01:32

And fortunately for us, it does work. — 04:09

32 bits gets you to about 2 GB, and that’s it. — 05:42

I’m surprised that Apple still hasn’t shipped a general-purpose local model you can prompt freely on the device. — 10:23

Concepts and entities

Apple Watch Series 6, Falcon H1, Small M2, Core ML, llama.cpp, Mamba 2, arm64_32, Andras, Better Stack

Full transcript (original language)

00:00 This is an LLM AI model running completely offline on an Apple Watch Series 6, which is a 6-year-old watch that came out back in 2020. There’s no streaming over the cloud involved here. Everything is working on edge and streaming text at around 15 tokens per second on average. So, a few days ago, I challenged myself to see if Apple Watches can actually run AI models locally. And to my surprise, yes, they can. And with a few tricks, you can make

00:32 them pretty efficient as well. Now, the reason this works is because behind the scenes, it’s running a 90 million parameter model called Falcon H1. And because it’s so small, it’s surprisingly performant. It even supports tool calls, and thanks to the watch’s own voice input, it’s really easy to prompt it as well. So, in today’s video, we’re going to take a look at how to compile and run an AI model on an Apple Watch locally. We’ll talk about why this works and what the best practices are when you’re

01:02 deploying these models, and then we’ll run a quick demo to see how it performs and where it still struggles. It’s going to be a lot of fun, so let’s dive into it. >> [music] >> So, a few months ago, I did another video where I tested whether a first generation Raspberry Pi could run an AI model completely offline. And it turns out that it can, but in my test, it was struggling at around 0.3 tokens a second. So, it worked, but it wasn’t

01:32 really usable. But that got me thinking, what if I put the same model on my 6-year-old Apple Watch? What kind of performance would I get out of a proper Apple wearable? And the answer turned out to be about 50 times faster. The Raspberry Pi 1 had a single 700 MHz core and 512 MB of RAM, while the Series 6 has two cores at 1.8 GHz and a full gigabyte. So, on paper, this was always going to be a more powerful setup, but I

02:04 didn’t expect it to be better by such a long shot. Now, if you’re thinking about how to run inference on a watchOS device, the first thing that naturally comes to mind is Apple’s own Core ML. It is Apple’s on-device machine learning framework, and the idea is that you convert a model into Apple’s format, hand it over, and let the system deal with it. But, that doesn’t work well with every model, and Falcon H1 is one of those models because half of its architecture doesn’t use attention

02:34 layers. The way Falcon works is that alongside the usual attention, every layer also runs a state space model called Mamba 2. It keeps one fixed-size chunk of memory and updates it one token at a time, deciding what to keep and what to override as each new token comes in. And that sequence of updates is called a scan, and because each step depends on the one before it, it has to run in order. And nothing gets cached or stored for later because it all gets

03:05 compressed into that state. So, it’s less precise than attention, but it’s very memory efficient. And a thousand tokens in, it’s using exactly the same amount as it was at token one. And Falcon runs both of these at once, the attention layer and the Mamba layer, and adds the results together, which is great for small devices. The problem is that Core ML has no state space building block. It only knows how to convert attention layers, so there’s nothing for the Mamba layer to convert into. And the

03:37 deeper issue is that Core ML wants a fixed graph decided up front, while the scan is a loop whose length depends on how long your conversation is. So, your options are to unroll it at a fixed length, which locks you into one context size, or carry the state yourself and thread it through a conversation pipeline that fails by converting perfectly and then giving you subtly wrong numbers. So, Core ML doesn’t work for our specific case, but there’s another option we can use, which is

04:09 llama.cpp. But, that one has another problem on its own. It ships with a script that builds for macOS, iOS, tvOS, and visionOS, but watchOS is not on that list. But, nowhere in the project does it say that it doesn’t support Apple watches, and there’s not a lot of information about this specific use case on the web, either. So, I decided to test it and see if we can actually get llama.cpp to work on watchOS. And fortunately for us, it does work. All it

04:39 needs is a handful of build flags and a one-line guard in the source. So, here’s where the confusion arises. Everyone assumes the blocker here is the architecture. watchOS doesn’t use standard arm64. It uses something called arm64_32. And that name makes it sound like it uses some kind of 32-bit cut-down, half-speed variant of arm, which is not actually the case. Because the instruction set is completely intact.

05:10 It’s the same full 64-bit arm. Nothing is removed, including neon, which is the arm’s vector unit that allows batching math operations. So, the actual math performance runs exactly as fast as it would on any other arm chip. The one thing that is different, however, is how the chip refers to memory. So, every location in memory has an address, which is really just a number, and on a normal 64-bit system, those numbers are 64 bits long. But, on the watch, they’re only 32

05:42 bits. And Apple did that because a watch only has a gigabyte of RAM, anyway. so making every one of those addresses have the size is freeing up memory across the whole system. But that does put a hard ceiling on things because a smaller number can only count so high. 32 bits gets you to about 2 GB, and that’s it. A watch physically cannot address more memory than that, which means it can never run a large model no matter how aggressively it’s quantized. But Falcon

06:13 is only 57 MB, so I’m not remotely close, but this is something you should keep in mind if you want to deploy other bigger AI models on the Apple Watch. And the nice part is that none of this requires porting anything or changing anything. ARM64 32 is just a target you tell the compiler to build for. You just need to pass different flags. So, to compile llama.cpp correctly specifically for watchOS, you need to use this command. But in the watch app project we’ll be working with, it is already

06:43 compiled. And I’ll leave a link down in the description below to the GitHub repo that you can clone and run it for yourself. And as you can see on Xcode, the project is quite simple. We just have a models folder where you can import different GGML files of whichever model you want. Just be aware that it is a very small wearable device and you won’t be able to successfully run a multi-billion parameter model on it. But for this specific demo, I have two AI models added here. One is the previously

07:14 mentioned Falcon H1, which has 90 million parameters. And I also added a slightly bigger 135 million parameter small M2 model just to see how they compare on the same set of tasks. And then in the tools section, there’s also a tools samples file where you can check out some of additional tools that I’ve added for the use of this app. And if you just rename this file to tools, it will register these specific tool calls instantly and you can also modify this file to add or remove any other specific

07:46 tools that you might want in this project. All right, so now let’s see how it works. All right, so let’s open the app, and here you can choose which model we want to prompt. So first, let’s try it with the Falcon H1, and we can just push this button to start speaking. What is the capital of France? And when we’re done, we can just click done, and then press the ask button. And as you can see, it’s using the Wikipedia tool call, and it answered almost instantly that

08:17 Paris is the capital of France, which is correct, and it did it with a speed of 24 tokens per second. Now let’s try the same thing with the small model. We can ask the same question. It’s using Wikipedia again, and this one was a bit slower, 14 tokens per second, because this is a bigger model, so that is expected. Now I also added a weather tool call. What is the current temperature? So let’s ask that, and as you can see, it’s using the weather tool call, and

08:49 the current temperature is 14.5 degrees Celsius. And I do have to say that for this tool call, I hardcoded the location as Ottawa, just because I didn’t want to add the location permission on this app, but if you use this in your own application, you would want to set it to reference the current location rather than hardcoding it. Now let’s ask it something more difficult. Can you explain the depth-first search algorithm to me? Wow, it gave me the answer almost

09:19 immediately, and look at that, it is streaming so quickly. And the longer it goes, the slower it becomes, because I guess we’re pushing the boundaries of the memory the watch can hold. So I think we started off at like 20 tokens per second, and now it has dropped to like 10. Yeah, you can see how the token count per second is dropping. So, it’s probably not ideal for very long responses or difficult questions that require multiple paragraphs of

09:51 explanations. So, as you can see, both of the models have no problems answering questions with detailed responses, and they’re doing it at a relatively quick pace for such a small wearable device. So, I’m super happy about these results. So, there you have it, folks. We now have a functional inference application. This means that even old devices from 2020, like the Apple Watch Series 6, are powerful enough to run AI models locally. Apple recently came out with

10:23 their new Series 12 watch, and although it does have an on-device neural engine that can perform certain light AI tasks, I’m surprised that Apple still hasn’t shipped a general-purpose local model you can prompt freely on the device. I think tests like I just did and similar experiments have shown that the industry is now in a stable enough state to actually have the capacity to run LLMs locally. So, my honest question is, why are device makers still hesitant to make

10:53 the jump to on-edge LLMs? Do you have any thoughts on that? Let me know what you think in the comment section down below. And folks, if you like these types of technical breakdowns, please let me know by smashing that like button underneath the video, and also don’t forget to subscribe to our channel. This has been Andras from Better Stack, and I will see you in the next videos. >> [music] [music]

11:27 [music]