Building AI Assistants with On-Device Memory
Read the Portuguese translation →
Author: DeepLearningAI · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Building AI Assistants with On-Device Memory · Channel: DeepLearningAI · 01:38 · Watch on YouTube
Topics: AI agents, Prompt and context engineering, Local and open-source models, RAG, search and information retrieval
Summary
Section titled “Summary”The course introduces how to build AI assistants with on-device memory, allowing applications to work locally and continue functioning without a network connection. Students build a voice assistant that turns notes, recordings, and photos into embeddings and stores them with context for later retrieval. The course covers semantic search, metadata filtering, deleting memories, and combining different kinds of input in an offline assistant. It also teaches the device to recognize new objects from examples added to its memory.
Key points
Section titled “Key points”- On-device AI can keep data close, respond locally, and work without a network connection.
- The course is taught by Dylan Cusack and is built in partnership with Quadrants.
- The assistant stores experiences as embeddings in vectors along with useful context.
- Memories can include text notes, voice recordings, and photos.
- The assistant can retrieve notes and recognize objects taught through photos.
- The course uses the open-source Quadrant Edge vector search engine.
- The memory system supports semantic search, metadata filtering, and deleting memories.
- The application can run on devices including Mac and Windows PCs.
- The completed assistant can recall experiences, forget them, and recognize things it has been taught.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | Why on-device memory | The introduction explains the benefits of local AI and presents the course on building assistants with on-device memory. |
| 00:32 | Building a memory-enabled assistant | The course assistant stores experiences as embeddings with context so it can later retrieve voice notes or recognize objects without sending memories to a remote service. |
| 01:04 | Memory architecture and vector search | The course covers the on-device architecture and building a memory system with Quadrant Edge, including semantic search, metadata filtering, and deleting memories. |
| 01:38 | An offline assistant that learns | The assistant combines notes, recordings, and photos to answer questions about the user’s day and learns to recognize objects from examples added to memory. |
Quotes
Section titled “Quotes”It can respond locally and continue operating even without a network connection. — 00:00
The assistant will retrieve a voice note about an appointment or recognize an object that you taught it from a few photos. — 00:32
You’ll store notes as vectors, search them semantically, filter them using metadata, and delete memories — 01:04
you’ll teach the device to recognize a new object by adding examples to memory. — 01:38
Concepts and entities
Section titled “Concepts and entities”on-device memory, embedding, Quadrant Edge, Quadrants, Dylan Cusack
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 For many AI applications running directly on the device like your phone, laptop, or this, opens up possibilities that could be harder to achieve than when everything depends on the cloud. The AI can stay close to the data it’s working with. It can respond locally and continue operating even without a network connection. And for some applications, you might want to give the device a memory of their own experiences. I’m thrilled to introduce Building AI Assistants with on-device memory built in Partnership with Quadrants and taught by Dylan
00:32 Cusack. In this course, you’ll learn how to build on-device AI applications that can remember what they’ve seen, heard, and been told. The voice assistant you’ll build throughout this course will turn these experiences into embeddings and store those in vectors together with useful context for later retrieval. The assistant will retrieve a voice note about an appointment or recognize an object that you taught it from a few photos. With all of this happening on the device itself without sending memories to remote service.
01:04 >> Thanks, Andrew. You don’t depend on a specific hardware to run the application you will build in this course. You can run it on any device, including your Mac or Windows PC. You’ll start by learning about the architecture inside the device where the camera, [music] embedding models, and vector memory all work together on the device. [music] Then, you’ll build the memory system step-by-step using our open-source Quadrant Edge [music] vector search engine. You’ll store notes as vectors, search them semantically, filter them using metadata, and delete memories when you want the system to
01:38 forget. >> [music] >> Next, you’ll combine text notes, voice recordings, and photos into an offline assistant that can answer questions about your day. And in the final lesson, you’ll teach the device to recognize a new object by adding examples to memory. >> Memory is important for [music] customizing experiences to a particular user. And by the end of this course, you have built an [music] on-device assistant to control experiences, recall them, forget them, and recognize things that you taught it. I hope you enjoy the course.