Forget the Series 12. My Series 6 Runs an LLM Locally!
Read the Portuguese translation →
Author: Better Stack · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Forget the Series 12. My Series 6 Runs an LLM Locally! · Channel: Better Stack · 01:01 · Watch on YouTube
Topics: AI agents, Model inference and optimization, Local and open-source models, Multimodal models
Summary
Section titled “Summary”An Apple Watch Series 6 runs a small language model fully offline, streaming text at about 15 tokens per second. The watch uses Falcon H1, a 90-million-parameter model, and supports tool calls and voice input. A demonstration shows it answering a question about the capital of France using Wikipedia and explaining depth-first search. The video concludes that older devices can run local AI models and asks why device makers are hesitant to adopt on-edge LLMs.
Key points
Section titled “Key points”- The demonstration runs on an Apple Watch Series 6 released in 2020.
- The model runs completely offline, with no cloud streaming.
- Text streams at an average of around 15 tokens per second.
- The watch runs Falcon H1, a 90-million-parameter model.
- The model supports tool calls and can be prompted using the watch’s voice input.
- It uses a Wikipedia tool call to answer that Paris is the capital of France.
- The video also demonstrates a request to explain depth-first search.
- The speaker argues that older devices can run AI models locally and questions why device makers remain hesitant.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | An offline LLM on an Apple Watch | The video introduces a local, offline language model running on a Series 6 and describes its speed and small size. |
| 00:31 | Voice input and tool calls | The watch uses voice input and a Wikipedia tool call to answer a question, then receives a more difficult prompt about depth-first search. |
| 01:01 | The case for on-edge LLMs | The speaker concludes that older devices can run local AI and asks why device makers are hesitant to adopt on-edge LLMs. |
Quotes
Section titled “Quotes”There’s no streaming over the cloud involved here. — 00:00
Everything is working on edge and streaming text at around 15 tokens per second on average. — 00:00
it even supports tool calls — 00:31
why are device makers still hesitant to make the jump to on-edge LLMs? — 01:01
Concepts and entities
Section titled “Concepts and entities”Apple Watch Series 6, Falcon H1, Wikipedia, edge computing, tool calling
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 This is an LLM AI model running completely offline on an Apple Watch Series 6, which is a 6-year-old watch that came out back in 2020. There’s no streaming over the cloud involved here. Everything is working on edge and streaming text at around 15 tokens per second on average. The reason this works is because behind the scenes, it’s running a 90 million parameter model called Falcon H1. And because it’s so small, it’s surprisingly performant. It
00:31 even supports tool calls, and thanks to the watch’s own voice input, it’s really easy to prompt it as well. What is the capital of France? And as you can see, it’s using the Wikipedia tool call, and it answered almost instantly that Paris is the capital of France, which is correct. Now, let’s ask it something more difficult. Can you explain the depth-first search algorithm to me? Wow. And look at that, it is streaming so
01:01 quickly. This means that even old devices from 2020, like the Apple Watch Series 6, are powerful enough to run AI models locally. So, my honest question is, why are device makers still hesitant to make the jump to on-edge LLMs? Do you have any thoughts on that? Let me know what you think in the comment section down below.