I Built JARVIS With GPT-6 Astra (It's INSANE!)

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: I Built JARVIS With GPT-6 Astra (It’s INSANE!) · Channel: AI Andy · 16:39 · Watch on YouTube

Topics: AI agents, Task and workflow automation, Data, databases and embeddings, Multimodal models, Open source and software ecosystems

Summary

The video demonstrates a voice-controlled Jarvis assistant that can see the screen, use voice dictation, and coordinate multiple coding agents. The presenter walks through its interface, model and voice options, knowledge graph, and workflow for assigning agents tasks and receiving their updates. He also shows hands-free use and LLM-assisted dictation, then describes the app’s available tools and open-source extensibility. The video ends with an offer of a prompt pack and a downloadable version of the software.

Key points

Jarvis can open multiple agents and let the user assign tasks to them by voice.
The assistant can inspect the screen and use a knowledge graph to connect and find local files and folders.
The presenter says the app can use several models, including GPT-6 Astra, Claude, and locally run models.
Users can choose from locally run or integrated voice options.
Jarvis acts as a lead assistant that relays tasks to agents and summarizes their responses.
The demo includes asking an agent to commit a playlist change and checking calorie-tracking work in a phone app.
Voice dictation is available outside the Jarvis chat and uses an LLM to clean up speech with context.
The presenter lists tools such as searching files, reading email or Telegram, opening apps, and managing agents.
The presenter says the software is open source and offers a prompt pack and a downloadable version.

Chapters

Start	Chapter	Summary
00:00	Introducing Jarvis and its knowledge graph	The presenter introduces Jarvis as a voice-controlled assistant for coordinating agents and shows its interface and connected-file knowledge graph.
02:39	Models, voices, and agent workflows	He demonstrates voice interaction and outlines the available model and voice options before explaining why he uses multiple agents.
04:44	Assigning work to multiple agents	Jarvis opens three agents, summarizes their findings, and lets the presenter direct tasks to specific agents.
08:52	Committing changes and checking the phone app	The presenter asks an agent to commit a playlist change and another to verify that calorie tracking is available in the phone app.
11:58	Chat history and voice dictation	He shows chat history and demonstrates using context-aware voice dictation in a document and speaking to Jarvis without being in its app.
14:35	Tools, extensibility, and setup	The presenter lists available tool calls, describes the open-source app, and explains the prompt-pack and download options.

Quotes

Just like that, we got three agents. — 00:00

This allows you to essentially have Jarvis as the lead and he can control all the agents for you. — 05:45

I’ve told the creator growth agent to commit the podcast’s playlist sir and to leave the backups and the other loose files out of it. — 08:52

And I think the best part about this is that this is completely open source. — 14:35

Concepts and entities

JARVIS, GPT-6 Astra, Claude, Opus 5, Qwen3, Graphify, Claude Code, Codex, AI Andy

Full transcript (original language)

00:00 Hey Jarvis, can you open three new agents for me? >> Just like that, we got three agents. >> Agents one, two, and three are up in their own windows, sir, sitting idle until you give them something to do. >> Can you give agent one the task of fixing some of the Jarvis issues we’ve been having recently? Just like that, >> I’ve told agent one to go through the recent Jarvis issues in the Jarvis repo. Fix them one at a time. >> Hey Jarvis, can you see what’s on my

00:32 screen right now? >> I can, sir. You’re in Chrome on the XTink X4 Pro, a Pocket e-reader marked down from $109 to 999, which I’m sure is purely research. Is this the best pocket reader or are there GPD6 Astra came out so I decided to make Jarvis? But it’s not a smart assistant like Gemini or Siri that can check your calendar and send emails. This allows you to control multiple agents at once just by talking

01:02 to Jarvis. It sees your screen and through your camera, so it’s easier to prompt [music] and voice dictation from anywhere, so you can directly talk to Jarvis or into any document or chat. But setting this up can be quite hard. So, I’ll give you the prompt pack by the end so it’s easy for you to set up even if you’re not techsavvy. Just imagine chatting with Jarvis with voice and he goes out and does real work. across

01:33 multiple agents. I’m super excited to show you this. So, let’s begin. So, take a look at the screen right here. On the left side, we have Jarvis, which is an orb that we can change into many different styles. By the way, we’ll take him in just a minute. But on the right side, we have a knowledge graph. And if you never seen a knowledge graph before, it has all the files and folders on your computer so you can easily find them and they have connections in between them.

02:06 As you can see, these nodes over here connect with each other. These nodes all the way down here connect to each other. And you can see exactly what they do. This is based on Graphify, which is one of the most popular GitHub repos with 119,000 stars that makes your AI agents a little bit smarter because it connects all this knowledge into easily findable files and folders and the connections as I said before. So here is more of a 3D visual

02:39 representation of exactly how your folders and files looks like. And we also have the classic view for the people that love graphify and its abilities. But over on the left side, we have Jarvis himself. And so if I hold down the option key on my keyboard right now, hey Jarvis, what’s the weather in Bangkok right now? >> I’ll check. Sir, Bangkok is 31° with light rain showers. Sir, feels like 36. >> So just like that, you can speak to it with just voice input and it will tell

03:11 you basic stuff. We’ll go into more advanced stuff in a minute. But in the top here, you don’t only have access to GPT6 Astra. You can also use Claude with this. So I have connected my Opus 5 subscription into it. I also included a Quen 3, which is a completely uncensored model into it as well. Running completely locally. It easily integrates with any LLM. So if you want Kimik K3, if you want DeepSync, you can just add it here. easily. But on the left side

03:43 here, we have Jarvis the butler. And this is more personality type stuff. So here we have an assistant that can do things. And this is more of controlling the agents that I’ll get to in just a second. And here you have the Opus 5 that is running on a subscription that is more like a Gemini that answers a little bit faster, but it doesn’t have all the abilities of the software itself. If we go to the right here, you can see that we have any voices from

04:14 Cocooro, which is a free voice model that is running locally. And we also have the Apple integrated one. If you have Microsoft, just a Microsoft integrated ones, super easy to use. And if so, if you want to change from this British voice to that British voice, it’s easy to do. I find that Daniel here and George is actually super awesome for having the butler in Iron Man, right? But the most important feature that I

04:44 think at least is the agents tab. So here you can see that we have one agent panel up and this is opening a new T-Max terminal that allows you to use any of your agents. Now, if you haven’t figured it out yet, if you use chatbt or claude with just the chat mode, they’re not as good as if you’re using them in the coding mode or now called work mode. Essentially allowing you to code

05:14 different apps, make a lot of things, and have a completely different memory and folder structure that just when I started it completely blew me away. And the real problem here is if you just have a normal LLM chat and you ask it to do three different things like check my email and see if there’s anything important and fix this bug in a software that I have and start making content ideas around this one thing. It’s going

05:45 to do that sequentially. And if you used AI before, what you might do is you might make a new chat for one of them and another chat for the next one and another chat for the next one because you don’t want it to just sit and wait for 3 to 5 minutes for it to complete its task. And that’s one of the big issues that I’ve seen with other like Jarvis or smart assistants is that they can’t really open new chats. So, this allows you to essentially have Jarvis as

06:18 the lead and he can control all the agents for you. So, let me just show you exactly how that works. So, I have a little bit of a low limit on GBD6 right now. So, I’m going to just go to Opus 5. Let me just show you this other design how it looks like. By the way, I want to have a community poll in my community page where you can vote for your favorite design of Jarvis as well. We have a bunch of them that I think look really cool. Uh, let’s test this one out

06:48 for now. So, let me just prompt it by holding option and I’ll show you. In the bottom here, you can see that we’re listening and sending directly to Jarvis. Hey, can you open three new agents? One for agent flow, one for goal flow, and one for creator growth. And as you can see, uh, these things like agent flow, goal flow, and creator growth is something that normal dictation tools has a pretty hard time with. But here you can see that they’re all opening,

07:18 three of them. And you’ll see them here in the UI as well, open pretty shortly. So you can see that >> agents one, two, and three are up, sir, for agent flow, goal flow, and creator growth. And they’re all waiting on you for something to do. And you can now see that once they’re coming back to us with a correct answer or >> the agent flow agent has read through the app and is ready for a job. Sir, it hasn’t changed anything. Though it did spot a fair pile of uncommitted work you left in the Swift files. What would you like it to do? >> And so here I can tell it directly which

07:51 agent I want to do what and of course if we get some >> creator growth agent is ready too, sir. And it found a podcast’s playlist you added but never committed which is already live on the running app. What should it do with it? There’s one more waiting. >> So there you can see that this type of workflow where they ping me back and Jarvis summarizes it is a little bit >> flow’s agent has had a look around and is ready too sir and it has found uncommitted work of yours there as well. What would you like it on? And in this

08:22 way, you can kind of not just read a bunch of chats, but rather just chat with it like it’s Jarvis, right? And so, this is a better way of working because you don’t have just one smart assistant that can do tool calls. You actually have three agents and one leader, which is Jarvis itself. And Jarvis speaks back to you. You speak back to it. And it’s a very easy and fun back and forth that you can do with it. So let’s do one

08:52 prompt or something just to showcase this. Awesome. Can you commit uh what the creator growth agents want? And now you will easily see that he is going to send a message directly to the other assistant and it’s going to run itself and I can just >> I’ve told the creator growth agent to commit the podcast’s playlist sir and to leave the backups and the other loose files out of it. >> Now he got back to me. So, he’s going to speak to me. >> Committed, sir. Only the podcast

09:23 playlist went in and the backups and loose files were left out, but it’s sitting on main and hasn’t been pushed. >> I guess let’s just demo it a little bit more. Let’s make an app feature on the phone. Awesome work. Can you tell the goal flow agent and ask it if we have committed the calorie tracking inside of Goal Flow so I can see it on the phone? So, here you can see the app. It’s basically a to-do list app that controls >> the gold flow agent whether the calorie tracking is committed sir and that you

09:54 want to see it on the phone. >> We have this calorie tracking that is just a super easy calorie tracking app that essentially I just tell exactly what I eat and put in the calories. It’s very very basic but helps me a lot in tracking. I just don’t have it in the phone app yet. So I’m just going to as you see I just asked it. We worked on it a little bit yesterday, but actually didn’t get it through. And so I want to see if it actually comes through here. Just listen. >> It’s committed. And on the phone now,

10:25 sir, under food at the end of the tab bar, you’ll have to close and reopen the app first. One catch, the Mac app has to fill in calories for anything you log on the phone. >> All right, let me check it on the app that I have on my phone. And yes, I can now find a food tab in the bottom here. And if I click over, you can see that I have my ramen and my sushi from yesterday and a little hot pot mala from the day before. And just note, I didn’t

10:55 use any fingers. Like I use one finger to hold option down, but you can also toggle on the hey Jarvis. And if I say hey Jarvis, I can everything I speak after that becomes completely hands-free. And even crazier, if I actually want to change anything with Jarvis itself, I can just ask Jarvis to make an agent that changes Jarvis. But I know that you have some questions and something else that I want to show you as well. And that is, can you type to

11:27 it? And of course, you can talk to it right here. So, if I say, “Hey, Jarvis, what is up?” It’s just going to work as a normal chat. And of course, we have um the history You can see >> both commits are in and the phone has the food tab. The goal flow agent is still waiting to hear whether you want the Mac app rebuilt so it can fill in the calories you log on the phone. >> Perfect. So as you can see you have the chat here and you also get the Jarvis uh

11:58 what is it called? Input into it as well. So this is what it noticed. So basically the agent sent this to Jarvis and then it sent this message to me. So it’s kind of sick workflow. And of course if you want to go back in the chat you can easily just go up in the history right here. And if you want to have a new chat, a completely new one, you can have a new chat here in the top. Uh but I’m just going to close it right now. So that’s the typing and the

12:30 history. But one of my favorite features is voice dictation because you can use it anywhere. So if you just close this down, you can see that I have my own type of Google Docs. And if I make a new document here, for example, I can just hold down the Fn key, which is different. One of them sends to Jarvis, one of them sends to any chat that you’re in. So, if I just talk out loud right here, it’s going to just pop up immediately right here using an LLM

13:01 cleanup that is super fast. So, it’s not only using like dictation that may make a lot of mistakes, it actually uses an LLM to clean it up and makes it a little bit more customizable to you. And the LLM that it actually goes into also understands it based on context. So I if I say Google flow, it’s going to read my documents with the graphifier and say ah it probably means goal flow, right? But

13:31 I can say things like check my goal flow and my agent flow and go to creator growth and then give me some information about Sarpsburg in Norway, which is kind of hard, right? But it literally probably nailed the whole thing and it does it really fast. So if you tried using any other voice dictation apps and probably if you’re paying for some, you can just yoink this and you will have it for free unlimited,

14:02 which is one of my favorite features. And again, if I want to send it to Jarvis without even being on the screen, option right, hey Jarvis, uh just say hello back to me and it’s going to chat to me without me having to be in the app. So, three, two, one. >> Hello, sir. >> And that was it. Beyond what we just talked about, there is also a bunch of tool calling here as well because when you’re using the assistant mode, you can ask a bunch of different things as well, like ask the graph, make invoice, look,

14:35 see screen, send to Telegram, read Telegram, read email, find files, list agents, read agent, spawn agent, open app, tell agent. And I think the best part about this is that this is completely open source. So you can just code whatever you want yourself into the app. And so if you’re advanced into agents and you know a lot about agents, you probably already know this that you can add more features. But if you’re a beginner, one thing that you can ask is for example, hey, can you connect to my

15:06 Gmail? And it’s easily going to connect with your Gmail or whatever calendar app you have. If you have clawed skills or chat GPT skills in the back end, you just put that into the same folder, which again, you can just ask the AI to put it all together. But how do you actually start building something like this yourself? I’m going to give you a prompt pack where you can easily just paste these into cloud code or codeex or even grock if you want to and you’ll get a similar result to mine. We got a bunch

15:38 of prompts. And what I recommend is doing one by one. So build this out first, then the second one, and then the third. And after a little bit back and forth, you will literally get to exactly what I have kind of, of course, a new version of it. But because it takes me months to make some of these tools, I also have the 5minut download that you can instantly download the exact version that I have and in just 5 minutes it will be up and running. So, if you want that, you can get that for free for 7

16:09 days. There’s a 7-day free trial to join AIMATE. Cancel at any time. And once you go into the classroom, you can click on my software. And here in the top, you see AF Jarvis. So, this is your own Jarvis, exactly the same as mine. And all you need to do is download this zip file and then copy and paste this XAP prompt into cloud code or codeex. And just like that, you will have the full version immediately. So click the link

16:39 in the description to join my school community or check out the free prompt pack. If you want to watch more of my videos, click on this video next and I’ll see you there. Peace.