Everything is LLM - Vibe coding a robot task

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Everything is LLM - Vibe coding a robot task · Channel: sentdex · 1:13:03 · Watch on YouTube

Topics: AI agents, Local and open-source models, Multimodal models, Robotics and autonomous systems

Summary

The video follows a live attempt to use GLM 5.3 Flash and a coding assistant to make an XGO mini robot locate and pick up a cube. The creator builds and iteratively tests a control interface, refines the robot’s pickup pose and movement, and uses color-based cube detection to guide an automated approach. After several failures involving delays, arm movement, and grip strength, the robot successfully approaches, grabs, and lifts the cube. The video closes with ideas for a reusable robotics harness, broader object detection, and future tasks such as delivering objects to a mapped box.

Key points

The creator uses an XGO mini quadruped robot with an arm, gripper, wheels, and multiple cameras.
An LLM can help choose and coordinate robotics tools, while low-level locomotion remains a job for reinforcement learning or existing gait control.
The live coding process builds a front end for camera streams, movement, body posture, and arm control.
Repeated testing reveals problems with command delays, weak wheel movement, and inconsistent body pitch and height.
A repeatable floor-pickup pose and a safe sequence for moving the arm are essential to reaching objects.
An HSV mask detects the green cube, and image position guides the robot’s approach and fine adjustments.
The automated pickup eventually succeeds after fixes to the gripper squeeze and arm retraction.
The creator argues that an LLM can coordinate fast OpenCV methods with more general vision models, rather than doing every perception task itself.
Future work could include a robotics harness, obstacle-aware arm movement, mapping with LiDAR, and carrying found objects to a drop box.

Chapters

Start	Chapter	Summary
00:00	Introducing LLM-controlled robotics	The creator introduces the goal of using an LLM to help a small robot locate and pick up an object.
03:11	The XGO mini robot and its hardware	The video tours the robot’s wheeled quadruped design, arm, gripper, and camera placement.
06:29	Setting up the coding session	The creator supplies the assistant with the robot’s SDK and network details and asks for a basic control interface.
10:12	Planning cube detection and a robotics harness	The creator considers color masks, segmentation, and vision models, and proposes a reusable harness for robotics tasks.
13:21	Testing the first control interface	Initial tests expose nonworking movement controls, lag, and problems adjusting the robot’s posture.
18:10	Tuning the floor-pickup pose	The creator tests body height and pitch targets and works toward a consistent pose for reaching the floor.
24:15	Defining a repeatable arm sequence	The assistant is given specific requirements for the pickup pose and the order in which to move the elbow, gripper, and shoulder.
31:15	Planning the cube approach	The creator outlines how the robot should spot the cube, keep it in view, approach it, and deploy the arm.
38:28	Testing detection and fine movement	The cube detector works, but arm stowing, command delays, and several nudge controls still need attention.
43:49	Moving from manual controls to automation	After testing a successful arm deployment, the creator asks the assistant to automate the full pickup process.
46:03	First automated pickup attempt	The robot tracks the cube and approaches it, but delays and a claw issue prevent the pickup from completing.
52:08	Fixing the lift and arm recovery	The approach and grab work, but the robot drops the cube while trying to lift it and stow the arm.
56:30	Improving the grip	A further attempt gets close, but the cube is not gripped tightly enough and the arm still risks hitting the robot’s head.
1:00:15	Successful pickup and future possibilities	The robot completes the pickup and lift, after which the creator discusses model trade-offs, a robotics harness, and future tasks.
1:10:21	Ideas for continued development	The creator summarizes possible improvements, including object-finding tasks, LiDAR mapping, and delivering items to a box.
1:12:32	Closing and upcoming robots	The creator wraps up, mentions a tracking link for the robot, and previews other robots expected on the channel.

Quotes

I think the way forward is actually use an LLM to help you figure out what is the best way to get whatever done on that robot you want done. — 11:14

Oh, that’s perfect. [laughter] Oh my god, it actually worked. — 43:15

Then you just send your the SDK information to an LLM and that’s it. — 1:04:01

Like this is a fast model if especially if you can run it locally. — 1:09:17

Concepts and entities

LLM, GLM-5.3-Flash, XO mini, XGO Mini, OM harness, Whisper, OpenCV, HSV mask, IMU, VLA, Segment Anything, Raspberry Pi, sentdex

Full transcript (original language)

00:00 So, I guess it just got like maybe a little too close. Oh, it’s so fast now. He just like bang goes for it. It’s kind of cool to see that. He probably should have gotten a little closer ideally, but we’ll see if he stands up this Oh my god, [laughter] that was way too fast, bro. No, I don’t think he actually did the full grab either. Oh man, we were so close. I could literally taste victory, y’all. What is going on, everybody? Welcome to a video on everything is LLM now. And uh what I want to talk about today is using

00:34 LLM for robotics, specifically this little XO uh mini robot, but there’s you can do this with just about any any robot, including like fixed based arms, all of that. And what you see here is actually GLM 53 Flash controlling basically the highle control. So you wouldn’t use this for locomotion or like a gate like how to walk. you would still use like reinforcement learning and extreme scale and just get that done in like 30 minutes and you’re done. Um, but for the purposes of what do you want to pick up? Where where might I find that

01:06 thing that you want me to pick up and then how do I actually use the tools in my tool belt to go get that thing on that robot? Uh, LLM are turning out to be very good at this and much better than the competition and um much more enjoyable than the competition. [laughter] So uh the competition is like VAS or ACT or some other visual motor type of model where you have to either try to use simulation but simulation is very tough because you have to build that whole simulation. uh and that takes a lot of

01:38 time and then you have a problem of sim to reel especially when it comes to camera information or you have teleyop where you have to like manually move that robot and do that whatever you’re trying to do manually a bunch of times and at the end of the day you still have this it’s not sim to real but it’s more like uh if anything in your environment changes like the lighting changes enough or the task just changes just enough like if you have this with the GLM model you know pick up the green cube You could also pick up any other color cube.

02:08 You could probably pick up any other color ball. Anything that will fit in that gripper once you kind of have the idea for how you want to get it done. Um what you’ve done there will work and it will translate very well to different different surfaces, different lighting conditions, everything. Um so it’s very compelling. And so anyways, uh that’s what I’m going to be kind of showing today. We’ll see. I’m going to do everything live. Uh there’s already like I’m going to be using the OM harness so it knows some things that I’ve done because of memory. I’m going to be working on the XCO which I already kind

02:39 of have files there. So it’s almost certain the LLM is going to like look at some of the other files. Um, but I just kind of want to show you the way that I would work through a problem like this and uh give you an idea for like what you can do with LLMs um for for tasks like this and like little robots because I think we’re living in a really nice time for for doing robot development both in terms of how do we imbue hardware with intelligence but also um just deploying an SDK and playing with small robot like this like certain some

03:11 of these things that we’re going to do today in like one minute or 5 minutes would be things that would take you hours or days or even maybe a week. Uh, and I just it’s it’s crazy where we’re at. So, uh, the robot I’m going to be using is this, uh, XO mini twowe robot. Um, I actually really like this. I’ve been following this company now for, um, I think it’s been like a little over 5 years now. Uh, and I’ve got I think I have four of these like quadriped things and I I think I have maybe two have the arm. Uh, I don’t have any with the LAR.

03:42 And the big new thing this time around are these wheels. And I have wanted a little quadriped with wheels for so long because what the wheels do for you is you can walk on like not ideal terrain. Like you could take this almost certainly and put it on like pravel or just even like full size like you know 57 rock. You could probably put it on there. And the thing is you can you can either walk without the wheels spinning. You can walk wheels only. So you can just roll or you can walk with the wheels spinning and the legs stepping.

04:16 And what this gives you is like if you come up to like a ledge or something like that, it’s like an inch high. Sometimes it’s kind of hard to get over that ledge and then you need all you need to like load a different gate entirely to get over the ledge. Uh whereas in this case, you actually don’t. You’re just spinning the wheels. So as soon as you hit that ledge, even if you didn’t quite get that foot right over that ledge, well, that wheel’s spinning. So it’s just going to pull you over that ledge. And uh that’s really cool. I also uh the arm is quite nice. It does actually also rotate. So, it has uh a rotation here and then the shoulder joint pivots and then it’s got like an elbow joint that pivots and then you

04:49 have the gripper. Um I like that. Uh that has always been the case, but the new thing is the shoulder rotates. Uh which is kind of cool. I haven’t found a good use case because you still you have to see what you’re doing. And I think if I would change anything on this robot, it would be there’s the camera is like right here. I actually kind of wish the camera was here on the gripper because sometimes the gripper actually does run into the to the camera. Like one, it’s kind of in the way. Uh two, if it was here, you’d still always have that forward- facing camera. But the problem is when you’re trying to pick something

05:21 up off the ground, probably the hardest part with this robot is getting close enough to the object where you can still see that object, but then also make contact with the gripper. So, another option would be maybe have the gripper have longer uh like longer tongs. I don’t know what to call those. Uh those could be a little longer. The arm, like the forearm could be longer, so you could get a little more length and then it would be a little more versatile. Uh so, these are all things that you could probably just modify, right? It would be very easy. You could just like honestly tape some some like plywood or something

05:55 [laughter] to the dogs um and achieve a better objective most likely. Alternatively, you could have like a hood over that camera and then the hood has a mirror on the top. That would also help. Um, so there’s a lot of little stuff that I might modify, but I’m trying I’m going to try to keep everything stock for today. Uh, anyway, let’s go ahead and just start working with it and I’ll show you just how easily we can get into all this stuff. So, uh, I’m going to move this out of the way. I am going to make a new folder here. We’ll call it Xcode. We will open that up. And I’m just gonna

06:29 open terminal in here. Uh, and I don’t really think we need this anymore. Okay. So now, uh, we’re going to try to do everything with GLM53 flash NVIP4. I have done all this with the native precision. NVIP4 seems to perform quite well with GLM53 flash, but we’ll find out. So, uh, the first order of business is I kind of just want to pass a little copy pasta over. So copy pasta.

07:02 So what I just passed, let me do this is um I’m just explaining to the uh to GLM, you know, hey, we’ve got this XGO mini uh whatever. Uh here’s the SDK, here’s the URDF, here’s the STLs, the meshes, all that. And then also Wi-Fi is set up. Here is the IP address and the password. And I’m just now realizing I didn’t turn on the robot. >> [laughter] >> Let me go uh let me go turn it on. Okay, try again. Also, from here on out, I do

07:33 also have speech to text. I know a lot of people use uh services for this. I implore you to consider how those services work. You have to send in your voice data and almost certainly they are storing your voice data. This was many years ago. I pulled my uh my Google takeout data and one of the most jarring things was all of the I think it’s Hey Google or something like this. all of those recordings were still there and they were they were like five plus years old and I like why are those saved [laughter] and from that

08:05 moment forward I don’t use uh I’m like unwilling to uh to use anybody’s like speechto text service and also you don’t need to we have whisper you can run whisper locally it’s a tiny tiny model you have you will have no idea it’s running so don’t even bother um so I’m going to be using that so sometimes I’m going to be speaking into my mic and hopefully you’ll understand the difference. Uh so, okay. In password. Yeah. Yeah. Yeah. Okay. Okay. Cool. [snorts] Okay. Great. Uh could you please uh give

08:37 us like um a very basic front end? I want to see the streaming head camera on the uh XO. I want very basic controls for the gate. So for example, we should be able to say, hey, we want to go forward, backwards, rotate, uh change the body height up, down, and then also change the body tilt based on the uh the IMU. So um probably ideally just allow us to address the IMU specifically, like what angle do we want? Cuz eventually what we’re going to try to do is pick

09:08 something up off the floor. And in order to pick stuff off the floor, we need to be able to lower the body as much as possible, but then also, you know, squat those front legs as much as possible and keep those rear legs stretched up as much as possible. Does that make sense? So, I need a front end that will allow me to do that while simultaneously being able to move the robot with all of the available gates. So, that’s stepping only, stepping plus wheels or like mixed, and then also wheels only. Uh, and then finally, there’s a third camera

09:41 that we’re going to use potentially for some debugging. It’s the uh Logitech camera. Could we also stream that camera live and do all that on like a really basic looking front end that we can run on this machine? Okay, we’ll see. Uh, we’ll see how it goes. Uh, if anything is processing for too too long, I will just uh I’ll just like edit it out. Uh, so we’ll we’ll see. Um, so the so the objective is I think today

10:12 I what I would like to show on video if we have the time anyways is at least getting this exco to go pick up that cube and then we’ll kind of call it there. So it’ll just be like a cube hunter and um and just kind of show how you can go about doing that. Now, how that’s going to happen, like I already kind of like I know at least the the physical um orientation that’s going to be required for this. Um but then how we’re going to actually do it, we’ll see if it how if it does it the same way that I’ve

10:43 seen before. But there’s many many different ways because you can you can like you one you could just mask out uh you could mask out the green, right? And then just just hunt green, right? You could also use GLM for image analysis. So maybe I’ll show a little bit of that happening. Um, and then I’m trying to think what else. So, you you could just use like an HSV mask, honestly. You can also use like some of these like segment anything type uh models that will that would also work quite well. And at the end of the day, what we’re trying to do

11:14 is not it’s not like we want to do everything with an LLM necessarily, but like some of these like VALAs, like some of the bigger VAS, um, that’s kind of what they do is they kind of force you to do everything with that VA. And I think the way forward is actually use an LLM to help you figure out what is the best way to get whatever done on that robot you want done. And eventually what would be cool is if it was fast enough to just do it on the fly. We were definitely not totally to that point. And that has made me wonder about

11:44 something like some sort of like we have a lot of coding based harnesses and like assistantbased harnesses. I kind of feel like we’re on the verge of uh like a robot harness for LLMs being very useful. Uh because we just need something to like kind of band-aid and paper over that gap until it gets really good. Like same thing with like coding harnesses. Kind of the the way that harnesses came about is it’s like we have this really smart intelligence but we’re kind of like doing the same similar patterns to ex you know get

12:16 value out of them and uh what if we just like automated that and that’s hence harnesses. Um so we just kind of want to do the same thing but for robotics. So I’ve been kind of trying to trying to like figure out how how would we do that? How would we like make it aware of all of the various models that exist or the techniques like with the OpenCV that exist for doing some of these tasks very fast? Because if you try to use V uh an LLM to simply pick up a cube and identify in every picture where’s that cube, that’s going to cap you to something like one to three frames per second of understanding. Let make sure

12:48 he’s not. Okay, cool. Um whereas if you’re using like HSV masking or something um yeah you can probably for sure 30 plus frames per second. Now if we’re using there’s like a little Raspberry Pi Zero on here, you know, maybe it’s more like five or maybe 10, but you can run that literally on board. Um so anyway, uh that’s what uh we’ll see. We’ll see how it pans out. [laughter] All right, it says it’s done. Here we have our uh front end here. So this is our front camera. This is that side camera. Oh, another nice benefit of

13:21 this robot is this is like the power cable to charge. And because the battery is like embedded into the robot, you can’t have just like multiple batteries that you just like hot swap uh or just swap in general. Um so the battery lasts like maybe like 30 40 minutes, something like this, depending on what you’re doing. And this version, I don’t think any of the other previous versions work while plugged in. This one does. And sometimes the the C because the robot’s so lightweight, it’s real easy to get like stuck into like one of the cables

13:52 and impact like if you’re trying to rotate the cord can kind of still like get in the way. But for the most part, doing an R&D while it’s plugged in is really nice. Um you probably should still unplug it and cycle the battery [laughter] just to keep sure make sure everything stays balanced, but overall I really like that I can dev with it plugged in so easily. So uh okay, so we have gate mode. Uh, let me adjust this so I know if I’m off my screen for you guys. Boop. Okay. Uh, so we have stepping only wheels. Uh, let’s go ahead

14:22 and just test it. Uh, gate, trot, walk, high walk, slow trot. I don’t know any of these. We’ll do slow trot. Pace. We already have. So, like what’s the difference? I don’t know. Uh, let’s hit forward. Nothing happened. Uh, we’ll come down here. Do we have any no error recording? Uh, okay. Uh, I’m just going to say, “Hey, bro. Uh, I try to use that front end.” I click forward walk. Uh, and nothing happens. Come on, dude. Do some testing before you tell me you’re done, fool. Uh, okay.

14:56 So, we’ll do a walk. Let’s I just want to try steping wheels. Yeah, none of these are working. Uh, come down here. What about these? Do these work? Nope. That don’t work. That don’t work. None of it works. What the heck? Freaking junior devs. What are we going to do with these guys? Um, [laughter] there you go, boss. I did my job. See you tomorrow. [laughter] All right. After a stern talking to,

15:27 he’s saying he’s done. Let’s see. Uh, whoops. Uh, we’ve got our cameras. We’ll just come down here and we’ll try forward. No way. Oh. Oh, I clicked it twice. Uh, that took a long time. Why did it take so long? Yeah, it’s taking a long ass time. That’s weird. It’s almost like you got to like double tap it. Uh, okay. Okay. Why did it take so long the

15:59 first time? That’s so weird. Uh, also Uh oh. Why is it laggy? I needed to We need to figure that out ASAP. Uh steps, wheels. Uh okay. What about this? Yeah, it’s it’s very uh it’s strangely laggy. Uh yeah, it kind of works. Uh, the problem is it is strangely

16:34 laggy. Also, I’m lacking any ability to set uh well, never mind. I don’t even want to bring that up. Uh, but when I change targets, like the IMU target, it takes like like five seconds plus just to to make that modification, which is weird. It should be It should be instant. Why Why is it taking so long from the UI? Um Oh, you already have a floor pickup pose. Let’s see that. Also, when I’m on

17:06 wheels only and I try to go backwards, nothing happens. Wheels only, I try to go forward. Waiting 5 seconds to see if anything happens. Doesn’t look like it’s going to. And then we’ll try steps plus wheels. Oh, I can’t even I can’t even select step plus wheels in the UI. What is going on? You need to get back and do your freaking job, dude. Don’t [snorts] tell me something’s done when it’s not freaking done. You need to do uh you need to do some actual testing, son. Uh biggest thing is the the delay. Uh stepping only. I would like to step plus

17:37 wheels probably. And we also just need adjust wheels to work. That’s how we’re going to fine-tune our positions. That’s probably that could be a function anyways of how much uh force you’re attempting to use as you uh run, but I don’t know. Figure it out, dude. All right. Our little coder minion has assured me we are back. Uh here is our front end. And uh oh I didn’t even realize I could change the frames per second on the camera. Okay. Anyway uh allegedly uh we can let’s see come forward

18:10 back. Okay. So we have a little more we what we Oh let’s try wheels only too. Okay. That we need a little more force on those. I think we also Let’s see if Let me make sure y’all can see everything here. Let me Okay, that’s your arm. Okay, the IMU. Let’s swap the IMU stuff. Let’s get it a little That’s interesting. A little taller. And let’s try forward again. Okay, so the wheels are a little weak. So, let’s go ahead and report that real quick.

18:43 Okay, the wheel only uh the wheel only movement is a little weak. So, it’s like too weak to actually do anything. So, we need to have a little more force there. Uh beyond that, the other commands seem to be working pretty good. Um and then let’s see what else. Let’s see the floor pickup pose. Okay, clicking the floor pickup pose. Um, it almost looks like it does things in like a sequence. If if that’s true, then

19:16 I guess that might be workingish, but uh I wonder if it has something to do with maybe the set body height. So, like right now I’m seeing the act. So, I clicked floor pickup pose and I’m seeing the actual IMU reading 7.52 degrees. I mean, that’s that’s literally terrible. It should read we should we should be hitting, you know, our goal would be to hit like 20 degrees. Um, but we’re probably going to hit some sort of like maybe 18 degrees, 16 degrees. And the

19:47 big thing that we really want is to have whatever the number is, we need it to be super consistent. Like we need to always hit if the target is 18, we need to always hit 18 with the same body height. Otherwise, the ability to grab is going to be like if if it varies at all significantly enough, then we can’t have any like rules about picking up things because it’s always going to be a different scenario. So, could we potentially let me try tinkering with uh let me Okay, so if we set to like 22 even. So,

20:20 if I set the body pitch to 22, I’m watching it. I’m watching it. Uh, we are hitting looks like the max we’re going to hit is 1432. Uh, so like 14°. Now, interestingly, if I set body height up, [laughter] I’m sorry, that’s funny. Uh, why would that be? I do not know. Let’s go back down. Beautiful. [laughter] Oh. Oh, that’s too fun. But um yeah, I’m still having a hard time hitting the

20:54 target IMU reading and this this might be a problem uh going forward. So like again we need to have a consistent reading and ideally we definitely we’d like to have a more like 16 to 18 degrees even if 22 is not reasonable like if we can’t hit 22 for some reason. Um, so when I hit max dip or floor pickup pose, honestly, we need to figure out what is the right combination. So maybe if you would if you would just like do

21:26 some abilations or something to hit to figure out how do we hit a 75 millimeter body height, but then also the maximum IMU target, like what is the series of things we need to do to hit that? Uh because oh well actually right now apparently the IMU is reading 183. [snorts] So then the next thing uh what is this? Oh this is the shoulder. Okay. So for the arm the other thing that we need to probably just pre-program is getting that gripper into position. So I’m going to Okay. So the shoulder we actually

22:02 Okay. So we would want to first move the elbow. Whoa, that’s so weird. So, probably set elbow to -41° and then the gripper actually don’t even know what’s open. Yeah, fully open the gripper. So, -65 before we do a grab. And then the shoulder we would move.

22:34 But you would before you move the shoulder, you definitely want to move the elbow. Is this open? Yeah, probably actually to like 53 and then come down maybegative 53. I can’t really tell. That looks like it’s real close to the ground though. Uh, okay. So, this is the shoulder. I’m just Okay, that’s ground. Okay, so actually Okay, so the position the robot is in right this instant is actually a really

23:07 good position. Um I wonder if somehow we could save that because then we would be able to see what we’re about to try to grab. Does that make sense? So my readings here are I’m trying to decide I’m trying to decide if we actually want to touch the ground. Okay. So, if we swung the elbow out this way and then I think that’s our play. Okay. [clears throat] Let’s say we want to have the exact positions that I’m in. So, the

23:40 body height is 75. The body pitch target for IMU is 22 degrees. Our actual reading is 183. Now, it’s it’s essential that we get as close as possible in some sort of repeating manner to 183. We then when we’re ready when we’ve achieved that position, we’re ready to deploy the arm to actually do the grab. In order to do that, what we need to do is first release that elbow like move the elbow to uh 57°, open the gripper to 65°

24:15 and then we can move the shoulder to 80 degrees. And this will put the gripper basically like right in front of the camera. So then all we need to do is get the object like in front of our robot. So what I need from you is can you help us to get to a repeat like the actual like quote unquote floor pickup pose? We need that to be repeatable when we decide we want to do a floor pickup. We need that to be repeatable such that we pretty much

24:45 always get this IMU reading of around 18°. the body height 75 and then the arm as already described. Do you think you can do that for us? All right, we’ll let him type all that up. That was a long long amount of information, but we’ll see if he can figure it out. I need to just like paste this probably into into clipboard instead of having it actually write out.

25:18 Oh my god, [laughter] it’s a book. Okay, very good. All right, we’ll see. We’ll see what he comes up with. Hopefully, we can get back to this like every single time. That would be that would be the dream. I’m going to save it while I can. Uh cuz I think if if we could get back to this in a repeating manner, that would be that would be nice. Uh cuz then we can actually see it. And historically, I’ve actually had to grab where I can’t actually see what I’m grabbing. But this allows us to get that gripper all the way actually to the

25:49 floor and we would theoretically be able to see what’s in that gripper. So I think that that’s I want to keep pulling on that thread. Go little buddy. Go. All right. This part’s actually just kind of interesting. So I had to move up the the shoulder a bit. So, I think it’s like ablating from various positions to see if it can’t continue. And I don’t know. I’m trying to decide. I wonder if that’s actually gonna No, I don’t think that’s going to keep updating for us, unfortunately. I wish it would tell me like right here. Uh, but I think what it’s doing is trying to go from like

26:19 random positions to get to that IMU position. But like here, I actually think he was a little higher. And I’m gonna [clears throat] I’m gonna voice that concern because I think we actually want to get like the head needs to be a little lower otherwise the gripper won’t be in the same spot. Like I think if we brought that arm down actually kind of want to do it. Let’s see. Is it this way? Yeah. What was it? 50. I already forgot. What was the uh I saved it in a photo. Uh shoulders 80 degrees. Okay. Wow.

26:51 Okay. So if I went to Oh, I think that’ll still touch. No, it won’t touch the ground. And so like the front end is not low enough, I think. Interesting. Is the elbow in the same Yeah. Oh, okay. So, he eventually got to that height. So, that’s weird. Now he’s smashing the heck. Hold on, buddy. Take it easy. [laughter] Can you please pick up the shelter? I don’t want to keep smashing the ground. Oh my god. Oh, that’s so funny. [clears throat] Okay. Um, got it.

27:24 I think you are doing it, but make sure that besides just the IMU, you have to get like the body height as well. It looks like you’re probably doing that. I just want to confirm that you you hit all of those requirements, not just the IMU. Cool. All right. I guess we’ll just let him I mean, like that looks pretty good. Um, so if we could if we could just like consistently hit that, that would be that would be ideal. And then we can hunt down that cube. But like [snorts] this, that’s too high, right? So, I’m not really sure. And maybe he’s like

27:58 doing that first. I don’t really know. Kind of want to make him go forward so I can see him better. You think it’ll go forward or no? Oh, it was wheels only. Hold on. There we go. See, like that’s too high. He needs to come down. I’m not really sure why. At least what I’m watching. like as you’re iterating like you are routinely getting like a good angle but you’re not as low as you will all like sometimes

28:30 you you get a little lower than other times and we need to make sure that we’re consistently getting also as low as possible. All right, we’ll let him cook and we’ll be back. All right, our little worker says he’s done. We’re going to see. Is he done? Let’s see. Uh, that looks good. Let me floor pickup pose. I wonder can I get away with I can’t even remember now. Is it this to get it up? Yeah. Okay, cool. So, that’ll get it up. 80 degrees

29:03 is where we want it. Now, what I want to do is if I hit neutral, does he reset? He Oh, okay. It does. Uh, let’s move forward ever so slightly. Hello, buddy. Okay, moment of truth. So, he’s squatting down, tilting down. Okay, still going. I wish it would go quicker, but we’ll see. I’m still just seeing Oh, maybe I have to hard refresh. Let me do uh a hard refresh here just in case.

29:41 Max dip h Oh, okay. I’m not even sure. I’m not sure what I’m looking at here. That’s pretty good though. Oh, he’s going for the grab. Hey. Okay. Okay. Floor pickup hose. I don’t know what max dip. What is What are we talking about here? H90 reliably gives. So if Okay. So um

30:12 floor pickup H75H90. Okay. Okay. So real briefly, what is the difference between max dip and then the the floor pickup? Like why did why did you retain both of them? Is it just because floor pickup is a little less spready or or what’s the deal there?

30:43 Oh, I see. So, floor pit. Oh, height 75. I see. The body is lowest. Okay. All right. [sighs] Um, I think we’ll go with this because we can see that. Well, um, okay. So, then the question is, [clears throat] all right. Um, I like that. I like that a lot. [laughter] That’s cool. Um, all right. So, now what I want to do then is okay, that looks really good. Uh, what I think we should do next is,

31:15 um, we need to pursue the cube and get the cube center screen and towards the bottom, right? Such that when we drop the gripper down, we would actually be able to grab that cube. Does that make sense to you? So, I’m going to pick up the arm. Um, and what actually what would be cool potentially is to be able to put the arm back like on the back and then be able to just instantly put the arm down into this grab position without needing to like manually do each of the joints. Does that make sense? So

31:48 like our uh shoulder like we have to like pick up that arm. Oh my. So like the shoulder could go to 54 degrees and then the elbow could go to Oh, I see. Uh yeah, the shoulder could go 54, elbow to1 or or something like that. I actually kind of I’m trying to see. Yeah, how squished up can we make this? Oh, we can make it real squished. Um

32:24 hm. Okay, let’s set the shoulder to -68, elbow to 31, and then when we actually reach the position and then enter into that floor pickup pose, I actually don’t know. Um I’m going to hit neutral real quick. Do you think he’ll Oh, he does level out. Okay, so another thing is the neutral pose. Oh, he does actually kind of Okay, he levels out. That’s cool. So, I think what we want to do is when we spot the cube, we need to get over to the cube. As we get close to the

32:58 cube, we’ve got to angle down as we approach it. Otherwise, we’re not going to see it like we’re going to lose it in the camera. So, probably once we spot once we’ve identified the cube, like we see a cube, we should immediately uh tilt the IMU target to I’m waiting for him to adjust. probably like 10 or maybe even more. I’m trying to see maybe 14. And then once we get real close, we

33:29 would want to uh So maybe I don’t know, maybe I imu pitch to like 18 and then wow. See, look how much he can straighten out those legs in that pose. Like that’s what I don’t understand is like uh if he can do that, how come I can’t just enter 22 here and then go down? I don’t really know. Okay. Um, so then if we turned left Oh, he’s still busy. I think he’s busy trying to hit this uh target.

34:00 Let me just do that. Uh, yeah. I’m trying to like I’m trying to make him turn. It seems like he’s uh busy. But you know what? I think the first thing is show me how you might detect where is the green cube. like what are the methods you might use to detect where is the green cube on that head camera like what are some of your ideas for that and then also how should we approach cuz I think once we’ve identified there is a green cube we want to tilt up that body but I was oh okay now he’s working okay so now the now the

34:32 turn is working it just like wasn’t working so like I’m thinking you see the cube in the distance you immediately go to pitch like 17 degrees and then you would start to approach the cube cube. Probably slower than that. Oh, that’s quite quite good. And you approach the cube till it’s at the bottom of the frame. And then we want to make sure we recenter. Now, one issue I have is I don’t actually seem to have it away. Let’s see if this works yet. Okay, wheels only still doesn’t seem to have enough force or it just straight up doesn’t work. Be advised. Um, and then the other thing is once that cube is

35:06 kind of centered, my issue is um we don’t we don’t have a good way in the drive mode to to walk forward with less force or less time. And we really need something like that. And also we need wheels only mode so when we get closer we could do fine-tune adjustments. Um, and then beyond that uh we need to what else do we really want to do? We we just need to work on that approach of the cube. So, we want to center the cube, approach the cube, and as we get closer and closer to that cube, we need

35:37 to begin to enter the floor pickup pose. I’m going to go ahead and test that. Oh, when I click for floor pickup pose, I immediately see you’re not moving the elbow like I told you to do. And you’re actually getting caught on the I already forgot shoulder. We need to set that shoulder down if we can. Now, he’s all pissed off. Uh, when I said floor pickup pose, you moved the I I’m trying to decide. I think you moved the shoulder before you moved the elbow. And you really have to move the elbow before you move the shoulder.

36:11 Otherwise, you’re going to get caught on the front head. So, you need to fix that. But then, yeah, when we get into that floor pickup pose, you essentially want to enter every part of the floor pickup pose except for the fact except for putting that arm down because we need to first get kind of close and then when we’re in a good position, then we would get close. So, I’m going to Yeah, we’re Wow, that actually works really well. Let’s see how far forward he goes. [laughter] Oh my god. Uh, let’s do one more stripe. Let’s see if this works.

36:44 Is it right in front I think? [laughter] Yeah, we need a way to like fine-tune adjust unfortunately. But yeah, and then as we fine-tune adjust, we really need that gripper to be out of our way. So, let’s do all of those things. Uh, and then see what we can come up with. Okay, that was that was a lot of information. That’s probably going to take a while. Um, but yeah, we need to do something like that. we have to work on how how we’re going to actually make that approach um and then do it like automated wise like I can’t I mean I

37:18 know how I would want to do it manually and we don’t even really have to do it I don’t really need those manual controls but it would be nice for like R&D purposes but I mean I’m very confident like that’s that’s a good way to do it and you would be able to see if it was in the gripper so I I really like the pose that we have here this this is actually the best one I’ve ever seen um but we need a way we need that gripper out of the way so we can kind of fine-tune our way get that get that cube like at at the very bottom. I.e. probably like a rule that we would have is like so that no sliver of anything but the object, you know, like below there’s nothing below

37:52 that object basically. I think that would be like a good rule to use. Like no pixels underneath it. Like that’s how close you want to be. And then send the arm down, make the grab. I think that will work. So, um that’s what I’m going to head for. We’re going to see if he’ll he’ll get us there. What is he doing? Oh, he’s I don’t even know what he’s doing. Anyway, um I’m gonna pause here and we’ll come back once he’s done uh doing that. [laughter] All right, our little dude says he’s done it again. Let’s see. Uh cube off. Let me I don’t like that pose at all. Oh, nudge. Nice. Okay. Okay. Let’s enter

38:28 a neutral pose. Okay. You got to put that arm away, though. Uh okay. Let’s try stowing the arm. It always [laughter] you son of a bee. You’re killing me, man. You’re killing me. Okay. Uh we Oh, he’s just Okay, he’s just going to drag it over. Okay, we need to fix that. That’s one thing we’re going to fix. Um Okay. And then we also have the cube detection.

38:59 Let me tilt this guy forward. Okay. So, let’s try cube detection. Oh. Oh. Oh, it’s so sexy. Oh my gosh, look at that. [laughter] Heck yeah. All right. Um, so let’s tilt down a little bit if we can. For some reason, it doesn’t want to do that. Uh, that’s a nice cube detector. Uh, let’s go back. Okay. So, I don’t know if we’re waiting or I’m not really sure what we are stuck with. Um, okay. I’m going to start my report here because he’s being a little bee. All right, man. Uh, the cube detector

39:33 definitely works. The arm sto uh works. The only problem is um like the arm is hitting the face as it attempts to go back because I think the shoulder started pulling back first before the elbow like the elbow needs to open out first and then start putting it back and then and then curl in the elbow. I think you tried to do it the wrong way. So, it got kind of stuck. Um I do see cube steering. I’m not really sure what I’m looking at. Um, I would love to test the

40:05 nudges, but Oh. Oh, now things are working. Oh, how fantastic. [laughter] Um, when I clicked buttons, it seemed to have gotten stuck. And now I think everything is coming through. So, that’s unfortunate to see. Um, okay. Yeah. Okay. We’re back to having commands be laggy, and that’s super annoying. So, I’m not really sure why it says like control resource busy or whatever. Uh, is there not like some way we can just like when we do a new command like flush out the other command and get them get them up out of the way

40:36 because it’s getting really annoying. Uh, and then beyond that. Okay, so my turn works. Okay, so my my nudge turn doesn’t do anything. Nudge forward sort of works. I’m going to try to nudge one more time. Nudge right does not do anything. Uh, so a nudge rotation right doesn’t do anything. A nudge left rotation doesn’t do anything. Let’s try a strafe. I do think the strafe makes an attempt. I I don’t think it does anything. It It attempts to strafe, but

41:08 it actually does not strafe. Let’s do a full strafe. Um, honestly, I don’t think a full strafe does anything either. [laughter] They just don’t do anything. So, I’m going to do a full turn or a full rotation left. Uh, that appears to work. I’m going to do click that one more time. Okay, that’s quite nice. Uh, I’m going to do a nudge forward. Nudge forward works. I do like that. So, as we go forward, uh, we need to adjust the IMU. For some reason, that IMU is just ignoring everything

41:40 about me. Okay, good. Uh, now I’m going to start nudging forward again. Well, I thought I would. Oh, there he goes. Okay, so the nudge forward kind of works. Let’s do that one more time. He made no progress at all. [laughter] Oh my god. Get closer, dude. He’s not getting caught. Okay, so that’s pretty good. And then I’m going to click uh floor pickup pose.

42:11 Once again, the elbow contracts completely and I’m concerned that we’re going to wind up probably opening the shoulder joint first and that’s going to cause a problem. And what are you doing now? Okay, maybe he’s entering. Okay, floor pickup pose. So then before we deploy the arm, I need to We got a couple of issues. One, we’re going to nudge forward. See if that works. Nice. That does work. My only concern is that might not be center, but I bet that is center. Okay,

42:43 so the cube is there. Now I’m going to say deploy arm. And when I do that, I better see that elbow move before the shoulder. I’m I’m worried it won’t happen. Let’s see what happens. I hit deploy arm. Nothing has occurred yet. Oh, it says at grab range. Bottom 100. Okay. So, it definitely seems to know it’s at grab range. I didn’t even notice that. I’m hitting deploy. Oh, okay. The second deploy arm. Okay, I can confirm the elbow does open. Then the grip opens and then the shoulder is moving down.

43:15 So, that appears to be functional. Oh, it’s so beautiful. Oh, that’s perfect. [laughter] Oh my god, it actually worked. Okay, then all we would have to do is close the gripper from there. Uh, is it this way? Oh, yeah. Oh, yeah. So, we close it in the gripper and then all we would have to do is lift it up. Oh, I’m lost. Okie dokie. Is it that way? I don’t even know. I guess we actually probably want to move the shoulder. Uh, for some reason I’m trying to adjust

43:49 the shoulder after the grab and I can’t get the shoulder to like lift up. I’m gonna hit stow arm and we’re gonna see what happens. [laughter] Okay, I hit stow arm. I’m not seeing anything just yet. We’re probably just more stuff is flooded maybe. I’m not really sure what’s happening there, but everything else seems to have worked pretty good. So now what I’d like to do is automate the whole thing. So, we know that we have all the little pieces that we need. We know how to center the cube in frame. Uh, we know how to

44:22 uh nudge forward and stuff like that. So, I think all those things like the turn left, turn right, or the rotate or whatever, that kind of like the full one works just as good as the nudge. But maybe we want to fine-tune the nudge. But other than that, everything else looks very good. So, I think we’re ready to actually like automate this. Like, I’ll put the cube in the distance and let’s see if you can actually get to it. What do you think? Um, and then we also just need to have the uh putback actually function.

44:54 That might take four years to fill up. I do like speech to text though. That that makes things very convenient. I would not want to have to have typed all this. There we go. Uh, okay. So, I’m going to let him work on that. And I think what I’ll do is reset the cube and all that and then we’ll see. We’ll just see if it can get to the point of going and picking up that cube. I think it should be able to. I believe here we’re using an HSV mask. Uh which is perfectly fine. Segment anything will probably be better for more general um objects. This I

45:28 don’t know what he’s doing there. Um hold on. I’m going to send that one more time. Uh okay. The pose that you’re taking right now is not an acceptable pose. It makes like the the actual legs and the feet come up off the ground. Don’t do this. Uh, okay. I’m going to let him work on that and then we’ll be back. All right. I had me some breakfast and uh GLM assures me that we now have a working uh

46:03 variant. And now let’s go ahead and see. I’ve reset the cube and the uh Xcode robot and let’s see what happens. So, we’ve got a few new buttons. I can just click auto pickup allegedly. Let’s go ahead and hit it and we’ll see what happens. Uh the cube we could turn on hopefully cube tracking. We’ll see if that works. Uh interestingly enough, it looks like maybe maybe GLM broke that aspect, but we’ll see what happens. Um I’m going to report that. Maybe I needed to

46:35 wonder if I needed to hard refresh. That might have been that might just be like a hard refresh away. Not really sure why the zoom it looked like the zoom just changed. What? Oh, I wonder if it’s this text. So, this is like some debug text. Basically, it kind of like tells you what’s going on. So, I think Yeah. Okay. So, spotted floor pickup. So, it enters into the kind of pickup state. Oh, there’s the cube. Maybe a little slow on the uh red detection, but now it’s using the wheels to inch forward. All looking good so far.

47:09 Let’s see. Let’s see as we make our approach with our vibe coded [laughter] while we have breakfast. I mean, what a time to be alive. I mean, we’re just going to eating and we have our uh robots learning to do things. Let’s see. That’s probably close enough. We’ll see. Oh, he’s going to inch. Okay, very good. He’s following the rules. I did say he have like no pixels underneath was the goal. Let’s see if he actually goes. So aligned, no pixels, bottom at grab range. Okay, sounds like he’s maybe

47:40 going to go for it. Not really sure what the delay is at this point. I’m just going to note that. So I’m going to copy this just in case. Hey, let me Okay, I can’t do that. [clears throat] I’m just going to tell him. Okay, so we’re running the auto pickup. Uh, the approach looked pretty good, I would say. Uh, the only weird thing is we are now at the cube. It says at grab range. And I’m just not sure why there is any delay because the grab is all um it’s

48:12 all rulebased, right? Like we just simply go for the grab. So, literally as soon as we’re in grab range, how come the grab doesn’t just immediately occur? And maybe that’s a bug. Once again, it could be I’m on the wrong like I didn’t uh hard refresh potentially. [snorts] [clears throat] We’ll see. See what he says about it. Oh, let me pull him over here. We’re close though. It’s definitely very close. And I like this one a lot better

48:44 actually than than the other one. This pickup is so much nicer because you actually like the angle is good. And I actually think that we can get those legs to be a little taller. I think the the way we have to do it though is uh individual servo control as opposed to the higher level API. So I I didn’t want to like get too deep into the weeds, but I think we can actually get a little steeper. But if we get steeper, well, hold on. We’ll talk about that in a second. Let me uh the grab is rulebased, but rulebased includes Okay. Yeah, but we still

49:19 that’s fine. Like, but why haven’t we gone for the grab yet? Like, everything looks good to me. And then why is cube off? That don’t make no sense either. Hold on. Let me uh we’ll do a hard refresh. I don’t think that’s what we need to do. I need to start paying attention to maybe what build we’re on or something. Um yeah, it just doesn’t make any sense. Found it. Up. It’s my claw lock overreach. Okay. I love it when the when the AI just blames itself. That’s like the most enjoyable thing in the world. Uh,

49:51 recheck claw after gate fix. I like when it start when it uses words like latched and uh I’m trying to think of some of the other ones that like just you always see and it’s like man can we just like stop. So like this is a perfect example of why I really want to harness because a lot of this stuff should just be automatically caught and overcome. Um, okay. Let me see here. Fix the bear. Why not grab? Is it going for a grab now or no? Uh, I don’t think it would be. Okie dokie. Press auto pickup once. Um, I kind of want to like back up and

50:26 do it, but I’ll just press auto pickup just to see. Oh, he’s going to probably lose the cube if he’s doing that. Let’s see. I want I want to see my cube as well. Rotating to find cube. It’s right there, brother. [laughter] Auto spotted. Uh, we’ll see if he turns. He needs to turn right. Historically, it has looked okay at doing that, but yeah, he’s he’s just unfortunately he’s too close to the cube. I kind of thought that might be a problem. [snorts] Oh, he did locate it. Okay.

51:02 Got to center that just right. Oh, yeah. See, [laughter] it’s going straight in. That’s why I was like, it should be like rulebased. Like as soon as you’re like there, it should be very fast. Okay. Very good. Very good. Let’s go. Look at that. Oh, he’s got it. No cube in frame. That’s kind of weird. He should just know that he has the grab. Uh the only thing I don’t know. So, we have the grab. How come he doesn’t just like put up the arm?

51:34 Because I I bet I could just sit uh hit stow arm. Oops. I don’t know if I actually had Oh, okay. He just kind of like dragged it. I think the stow arm also is a little flawed. I think he just dropped it. Son of a bee. Okay, we actually approached, did the grab, all that really nicely, but once we actually did the like we had the cube in our grab, we failed to actually like pick it up. I’m just going to back up. Actually, I’m going to hit stop. Can I just Is this going to work? Anyway, uh we failed to actually pick it

52:08 up and then when I tried the stow arm, but then it like rubs the cube along like the head and it ended up dropping the cube. Uh I’m pretty sure we have a pickup that shouldn’t do that. And also stow arm should just not do that. But most importantly, we actually did not pick up like we grabbed the cube perfectly. All we needed to do was like pick up the cube and then stand up and that was it. And then that would be like the full the full loop. So, um, can we analyze maybe any issue that we added that is causing that trouble and, um,

52:39 fix that? And then I think we’re we’re basically done. I mean, that was that was a good approach. A good grab. Everything actually worked pretty darn good. I think we we just need that last pickup. Um, and then after that, I think I want to add like putting it into a box, like putting the thing like grab stuff up off the floor and then put into like some bin location. And that at that point so far that LAR really hasn’t had any use in my opinion. But eventually what I want to be able to do is like map the environment and do like an occupancy grid type situation and then just note

53:13 where the box is like where is the you know dropbox essentially for like ground floor stuff. And then with using the LAR, you can find out based on patterns where are you in that whole like layout of the building that you know and then you know where the box is in that layout and then go there and then eventually you can as long as you go towards that box facing it you would be able to identify it no different than kind of what we’ve done here. Um, the other thing I’d like to add is segment everything potentially or some other

53:43 really small model potentially to um to have a much wider range because eventually I don’t really actually want to pick up cubes necessarily like Lego bricks is a good one. Um, I would like to pick up those. Um, but then there’s all kinds of stuff like stuffed animals, socks, shoes. Well, I don’t think this this robot is not going to be able to pick up shoes, but later we’ll have different robots that could theoretically pick up a shoe. Um, so we want to be able to handle for that. But already you can see that I mean this is like I mean I’ve done a lot of yapping to you. I don’t know exactly how long

54:16 this video is going to be. But with very very minimal effort we have a robot that in um just about any changing condition. We could change the lighting. We could change the flooring. We could change a lot of stuff about this can go and pick find a cube and pick it up. Um and then and it doesn’t have to be a cube. We could change that cuz basically it just needs to be something we can detect and put a bounding box on and then from there we can approach literally anything. So it might cost us a little more time maybe to do that initial detection but not really. I mean most of these things you can do a detection even

54:48 GLM53 flash I was doing detections with that in about 200 to 500 milliseconds. Um so even if you just want the most powerful nonsense uh image you know detector or object detector you could literally just use GLM53 flash. So, um, so yeah, that that’s super cool. I do hope, um, I don’t want to finish this video until we literally have the approach, grab and stand up. I just it’s going to feel wrong if I stop us here. So, um, I’m very confident we are going to get there, but I have to I have to see it. So, um, interestingly, see, like

55:20 this is a mistake. Oh my god, if he’s going for this grab, I’m going to lose my mind. Please don’t go to try to grab [laughter] that. I was going to lose my mind. Yeah. Um, test sto Oh, I guess it’s just testing stowing. I’m not really I have no idea. I I’m not sure. It’s clearly saying it’s doing like some R&D stuff here. So, I’m going to let him keep going. Um, yeah. Uh, one thing I will add is a lot of people are using models like Astra and stuff to actually do this. Hold on, let

55:54 me make sure. I’m going to restart it if I can. Uh, Okay, I’m going to do a hard refresh just to be 100%. Uh, we are going to cube on lift st auto pickup. Uh, that might be in place of stow arm. I’m not really sure for manual recovery. I got it. Okay, so auto pickup should work. I’m going to hit it. Full send, boys. Let’s see. >> [clears throat]

56:30 >> Oh, that detection don’t look good. [laughter] What is going on there? What is going on there? That’s goofy. Let’s see if it actually makes the approach. I’m going to wager that’s a bug, but we’ll see. Oh, yeah. Some sort of weird bug. Vibe coding. This is probably This isn’t even very hard because you have to rotate. But I believe I think the other grab was requiring a rotate. So, you’ll have to take my word for it or just have watched that to know that it definitely worked. Um, I do like this. This is cool. Let’s see.

57:03 Is that good enough or is it Oh my god. He’s got to get straight floss. Interesting. So, I guess it just got like maybe a little too close. Oh, it’s so fast now. He just like bang goes for it. It’s kind of cool to see that. He probably should have gotten a little closer ideally, but we’ll see if he stands up this Oh my god, [laughter] that was way too fast, bro. No, I don’t think he actually did the full grab either. Oh man, we were so close. I

57:35 could literally taste victory, y’all. I could taste it now. He’s got the hand the the gripper in the face. Okay, hold on. All right, we are so close. I can taste victory, man. But uh you made a few mistakes. one is uh we probably just needed to get slightly closer, but to be honest, I think we had a good grab. I think you just simply did not grab it tightly enough. Like you the the cube didn’t even move when you uh went to go pick it up. So, I think the main flaw here was you literally just did not grab

58:06 it tightly enough. So, we need to fix that. Um, beyond that, uh, when the arm is going to actually go back to the, you know, lift position, it still is trying to rub on the head of the, uh, XO robot. That’s stupid. We should not be doing that. We should pro, we could even like swing the elbow out slightly or something first and then retract the shoulder and then retract the elbow or something. Like, hitting the face is stupid. We got to stop doing that, man.

58:39 Um, beyond that, the approach, the grab, everything is good. Like I said, I actually think you dropped the cube purely because um purely I honestly I was I was trying to get it so it’s like not centered. I think it’s like still in auto pickup mode. Um anyway, uh I forgot I was recording as I was talking to you guys, the viewer. But anyway, um so yeah, let’s let’s fix that. When we go to grab, we need to make sure we are grabbing tightly. like you can probably just keep trying to

59:11 grab tighter and you will grab tighter. [laughter] I think you need to do that. Um, other than that, I think we’re good to go. But yeah, we got to we got to fix this. We got to grab a little tighter. We got to stop hitting ourselves in the face with the object that we just tried to pick up and let’s get it done. Like, let’s get it done. Okay, I’m going to send this. Um, and hopefully we will have this. I I thought maybe, just maybe, as we were watching it was going to do it. Uh, but I can taste it. We are so close. We are so close. Um, so we’ll let him work on this. Uh, hopefully hopefully we are

59:44 just moments away from success. All right, GLM once again is asserting it’s done it all for us. Let’s see. We’ll see what we got. Apparently, also a bonus bug paid for all day. [laughter] What the heck? Uh, don’t know. Current pro silently break. Interesting. Um, okay. Yeah, let’s, uh, let’s send it. Let’s see what happens. Uh, we will hard

1:00:15 refresh and, uh, can I just do this? I just want to force it to. So, cube on. I wonder if cube. Oh. Oh, interesting. So, the cube is the steering. That’s hilarious. And then auto pick. So that’s just going to always be steering. Hold on. I I’m sorry. I just have to mess with it. That is funny. That means you could probably just like start moving the cube. Okay. Um I’m not going to do that though. We are going to go auto pickup.

1:00:47 Let’s see what we get. What do we think? Is this finally it? It’s a little It’s almost not fair because you guys have um spotted on what verify height and pitch and then approach. I’m not really sure. Uh, you guys know how much time is left on the video, so it’s a little It’s kind of not fair. He’s ready to pick up, bro. [laughter] He’s definitely not in the right position. Okay, now he’s in the right position.

1:01:21 He’s on the prowl. [laughter] I kind of like the original Twitter one, uh, like the gate as he’s like approaching and like all the actions that he’s taking as he’s approaching that cube. I kind of liked that. it just looked better. So, I might have to bring that back in. But, like I said, I just I just wanted to walk through kind of the process that I go through on something like this. People are always asking for like a vibe coding video. And to be honest with you, I just think most of the time vibe coding is probably really boring to watch. That’s just that’s my estimation. Um, so hopefully this wasn’t

1:01:55 like super freaking boring. Um, I’ll like I said, I’ll try to cut out all the weight times, but Okay, it goes in. Did he grab that all the way? Okay, that looks all the way. Okay, please just pick it up and please be nice. Uh, okay. Shoulder verified squeeze ladder. Okay, now what I don’t understand it, so it has the grab, but it’s still doesn’t like why doesn’t it want to lift? Okay, beautiful approach. Uh, beautiful

1:02:25 grab. Everything’s looking good, but for some reason, like why why after the grab, after the full grab, why are we not automatically lifting that cube? Oh, well, I guess I’m wrong. Oh my god, that’s so aggressive. [laughter] We probably need to chill out the aggression. Uh, it’s a little slow. That’s a little silly, but Oh, he’s going to go back. Okay. Okay. probably would prefer it to go in a full rest position. But you know what? I’m

1:02:59 going to take that as a win. I’m not going to waste y’all’s time uh fixing all that. It would literally just be me telling it, “Hey, will you do this?” So, um overall, man, I am I’m super happy with that. Uh that’s really nice. I I I I think that’s that’s pretty cool. Uh I think for us to be where we are today where I’m still recording on my transcribe. Hold on. I got to fix that. Okay. As I was saying, I for us to be here, right, where you know, it wasn’t that long ago. I mean, I would say like three years ago

1:03:31 when you bought like a new robot, like you can literally take this robot out the box. You don’t even have to look at the documentation for the SDK. Like, you don’t have to do anything. You do have to, let’s see, you initially have to set up your Wi-Fi. So, when you first get this particular robot, you need to plug in a mouse and a keyboard, load into the uh Linux on the Raspberry Pi, and then you need to log into a Wi-Fi so you can make an SSH connection. After that, that’s it. That’s all you ever need to do on this robot. You just get out of

1:04:01 the box. Then you just send your u the SDK information to an LLM and that’s it. That’s all you have to do. And then you just start asking, hey, I want to build XYZ. And like in the case of this task, we could do it with an HSV mask. really simple, you know, basically you’re just filtering for color and uh achieve this task. For more complicated things, we could probably use segment anything model. Um I think there’s one and two. There’s a model two for that. So you could use that and if that fails, you can literally use a vision-based LLM

1:04:32 like GLM53 flash that will also detect literally anything and and it can draw bounding boxes and it can do it at up to about three frames per second, which is honestly very good for like stuff like this. Like you can do very general purpose things. The other thing you can do with GLM53 flash is you could literally say like you’re looking at a refrigerator. You could say, “Okay, find the refrigerator in in this frame, but then also where is the handle? How should I grab the handle?” And then from there, you can actually pass your robots

1:05:03 uh model files like to um to the LLM and determine, okay, how many joints are in my arm and like what position, what pose should I be in to grab that fridge handle and open that fridge. like these are the things that you can do when you’re using a just huge general purpose language model as opposed to a VLA. Now, there are some really nice VALAs that you can do stuff like I just described and it like basically will output a structured JSON, but they’re just not as intelligent and useful as an LLM and then but it turns out you can just use

1:05:36 an LLM to do everything. one, it can it could output straight up JSON for joint positions, but it can also help you to do the HSV mask stuff because that’s faster like for for a lot of stuff, we can just use really rudimentary OpenCV techniques to do a lot of cool stuff. So anyway, um I hope you guys enjoy it. I hope you find this interesting. I also just again I think it’s cool that all of this on GLM 53 Flash. I I’ve seen a lot especially lately of people using Astra for robotics. The one thing I would say is or even like Claude Code or uh what

1:06:09 do you call it? Uh Fable, [laughter] it is so dead to me. Um Fable Astra [clears throat] also I I would love to never use Astra again. I do. I mean it there’s no question Astra is smarter than GLM53 Flash. Um, uh, I just recently I had a computer that, uh, it was one of the computers I tried to use to do the AI server and I, while I was in there doing things, I pulled out the Windows hard drive or the NVME for Windows and I

1:06:40 guess it was in the middle of a Windows update. I don’t I don’t freaking know. I can’t believe this. But when I pulled that drive out and then started turning things on, I started getting a blue screen. even if I was trying to go to Iuntu, like what the [laughter] what the hell? Um it was the weirdest thing. And then so I put the Windows drive back in, but then everything was just messed up. I could not get into Windows was constant blue screens and the recovery disc was like not working or the USB drive was not working. So then I used Astra, gave Astra shell into the Linux because that machine dual boots into

1:07:12 Linux and Windows. Gave it access to the Linux and it went in and I don’t know started around on the Windows [laughter] and like fixed it. I could not figure that out. I was never going to figure that out. There was no amount of Google searching and forum reading that I was ever going to figure that out. And I just threw Astra at it and it fixed it. And that’s crazy to me. I mean, that’s that’s that’s wild. Um, but it is also cool how close GLM like these these open source models are to that. And we’re not

1:07:42 really that far away from having open source models being trustworthy enough to do something like that with. I still won’t use like even Astra. I won’t let Astra run full RIP on I almost would have let GLM 53 Flash because really what’s the worst that could happen? I the worst is I have to reinstall Windows or something, you know, like who cares? So I I kind of I almost want to like brick it again and [laughter] and see if GLFI3 Flash could do it. U because it was I I started off letting it try and then I could see what it was starting to try to modify and I was like that sounds risky. Um [sighs] so yeah. Um and then

1:08:16 also just be I was doing it like on my main computer. Well, what if GLM well what if either of the models kind of screwed up and instead of doing it on the remote machine just started doing commands on the my main machine, you know? So, like I it makes me very nervous. Um, but yeah, I would like to I’d like to try harder stuff like that. Like I was like, “Oo, that’s that would be a good benchmark question.” Um, and I I bet it’s relatively repeatable. [laughter] So, anyway, okay. So, that’s it. Oh, the other thing I was going to say about Astra is it’s just so slow. So, even

1:08:46 this this took me a lot longer, I think, cuz I’m talking to you guys trying to do things and like and then also like I’m actually like sitting here like a lot of times if I’m doing something like this, I’m going to be doing like other things at the same time. So just it’s like you give your input and then you leave and then you have the result, you know. Um but one thing with like Astra is it’s so slow. Like at least with GLM53 Flash, you really you can kind of like dev live with like with a project like this, you’re pretty consistently giving your

1:09:17 input, waiting for results. I’m usually like reading the text as it’s flying by. But as you can see, I mean, it it goes pretty quick. Like this is a fast model if especially if you can run it locally. like I’m getting like around 200 tokens per second on this um logo and um yeah it it’s enjoyable but if you try to use Astra for it it’s just so slow and like using Astra for literally like live play out like using Astra as if it’s a VLA it’s really painful in in my opinion uh it’s just not it’s just too slow whereas if you can use a model like this or even

1:09:49 smaller and get them even faster that would be cool like um DeepSc V4 flash you know we we can get like 400 tokens a second and get very fast, especially if you’re using Dlash and then uh uh LM cache. So if you’re if you have all of those things coming together, you are your response rates are so fast like you could you could do things really quick. So we’re I think we’re very close to um like GLM53 flash, we don’t have any LM cache going on. So, like if we could get that even um our we we probably could just do like we could do any detection

1:10:21 we wanted live with GLM53 flash and like no problems and um especially if we’re tracking like the same similar types of objects or something. I think that we could we could speed a lot of things up. So anyway, um okay, I think that’s uh that’s everything for now. Uh if you want to see more on this concept, uh let me know below. Like I said, I I really want to add I want to make some sort of harness because a lot of these things are like obvious and I feel like a lot of these things are going to be the same no matter the robot, no matter the task where a lot like like hitting yourself in a like like running into like the the

1:10:53 head camera or running into your body with the arm just doesn’t need to ever occur, you know, like we have the model files. We could like literally just pull it up in like a majo and understand like was contact made or not, you know, those kinds of things. Um, and then like sensor fusion would be kind of cool. And anything with a LAR, you can pretty much always make an occupancy grid. So, I kind of want to do stuff like that. I don’t know. I haven’t really fully decided what I want to do or what I want to continue with this robot. Um, what would be the thing? I mean, clearly this robot should pick stuff up off the floor. And it’s a very unique um like probably smallerish stuff up off the

1:11:27 floor. But maybe one of the things I would like to do from here is yeah have like a box in some designated location and then we can map with uh the LAR and then from there know where we are at all times and then once we have a thing go to the box and just kind of repeat like keep looking for stuff on the ground that like needs to go into that box. Again an LLM is great for this. So like obviously you might have certain things on the floor that like are supposed to be there but like many of the things are just kind of out of place and like VLMs and LLMs like you can literally just say some you know find me anything on the

1:12:00 floor that is like a kid’s toy find a kid’s toy on the floor and it will do that or you know find me something on the floor that probably should be picked up you know and it would be able to do that. Uh you could also probably say something small on the floor and it would find that. So anyway lots of really cool stuff. I hope you guys have enjoyed. I hope that wasn’t too boring uh watching someone vibe code. Um I’ll do my best to edit it and make it somewhat enjoyable hopefully. Uh so yeah, let me know what you think. Uh questions, comments, concerns, suggestions, whatever. Feel free to leave them below. Otherwise, I will see you all in the next video. Oh, also uh

1:12:32 if you want to buy this, I’ll put a link in the description. I think it’s sort of like a referral. It’s like a tracking link. I don’t there is no affiliate marketing like I don’t get anything. I’m not even sure you save money by um by using my link, but it is like a link that I think that they’re just trying to use for um tracking purposes. So, like like to know like who referred you basically. So, uh I’ll put a link in the description if you want to check out this robot. Um I think it’s pretty cool. I actually have a few other robots on the way. I’ve got a Sourcey. I forget what they’re call if it’s actually

1:13:03 literally just called Sorsy. I’ve got one of those and then a near I might have the name wrong on that. And then I did also get one of the little open ducks uh from HuggyFace. I thought that was a cool one. Um and I think that’s all the robots I have on the way. But yeah, I’m definitely um got quite a few robots that uh hopefully we will see on the channel very soon. Anyway, okay, for real, that’s all. I’ll see you guys next time.