Why Developers Are Obsessed with Budget LLMs (Jev, Luna, & more) | Merge Conflict ep. 534

Author: Merge Conflict Podcast · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Why Developers Are Obsessed with Budget LLMs (Jev, Luna, & more) | Merge Conflict ep. 534 · Channel: Merge Conflict Podcast · 39:58 · Watch on YouTube

Topics: AI agents, Software engineering with AI, AI content generation, Model inference and optimization, AI models

Summary

James and Frank discuss how lower-cost models such as Luna and Soul are changing developers’ choices, and James describes building a Wheel of Fortune-style app entirely from his phone with cloud agents. They compare model prices and capabilities, then debate why generated interfaces often need iterative refinement, especially around information flow and user experience. The main topic is Jev, a fast, inexpensive language model designed for classification and choosing among predefined answers rather than generating text. Frank explains how it works and where it might be useful, including routing requests to the right model.

Key points

James built Wheel of Wisdom, an offline-capable progressive web app with 600 puzzles, entirely from his phone using the GitHub mobile app and cloud agents.
The hosts discuss new model releases and price changes, including Luna, Soul, and Claude Opus 5.5, and emphasize choosing a model that fits the task.
Luna Max produced a polished Oslo day-tour website and deployed it in about six minutes for 18 cents, according to James.
Frank advocates treating generated interfaces as rough drafts and refining them through iteration rather than expecting perfect UI on the first attempt.
The hosts distinguish visual styling from UX, stressing that information flow, prioritization, and interaction patterns are often harder for models than generating attractive CSS.
Model benchmarks are increasingly evaluated by both success rate and cost; the hosts caution that logarithmic price charts can obscure large cost differences.
Jev is a language model built for classification: it selects from predefined options and can return confidence scores rather than generating open-ended text.
Jev can cheaply classify one input across many questions by reusing the prompt-processing stage and producing a limited set of answers.
Potential uses include checking generated content, assigning categories, moderating or filtering content, and selecting which model should handle a request.

Chapters

Start	Chapter	Summary
00:00	Opening and travel update	James and Frank greet listeners and talk about James’s travels through Europe.
02:09	Building a game from a phone	James introduces Wheel of Wisdom, a travel-friendly game he built and deployed entirely from his phone.
05:15	New models and shifting prices	The hosts discuss recent model releases, pricing changes, and the pace of updates.
09:05	Luna Max in a real-world demo	James describes using Luna Max to create and deploy an interactive Oslo day-tour website quickly and cheaply.
11:50	Iterating on generated interfaces	Frank explains why model-generated UI should be treated as a first draft and improved over multiple iterations.
14:29	Design, styling, and user experience	The hosts separate visual styling from UX and discuss the models’ weaknesses in information flow and interaction design.
17:13	Comparing model capabilities and costs	They compare newer Luna and Soul releases with Opus and consider how to weigh price, performance, and conversational quality.
23:48	Model routing and the arrival of Jev	James shares a model-routing workflow before the hosts introduce Jev as a different kind of language model.
26:26	Jev as a classifier	The hosts explain how Jev classifies inputs, chooses among predefined options, and supports workflows that check generated content.
33:25	Why classification can be efficient	Frank describes Jev’s place among transformer models and discusses its speed, cost, and possible applications.
37:17	Use cases and closing thoughts	The hosts reflect on Jev’s potential, including model routing, and wrap up the episode.

Quotes

I coded this 100% from the GitHub mobile app. — 04:12

These models are never going to create the perfect UI for you — 12:23

What we care about is dollar bills. — 21:05

Its output is a category. — 25:54

choose a model for me. — 39:24

Concepts and entities

Luna, Luna Max, Soul, Claude Opus 5.5, Jev, HydraFusion, Impeccable, uiMD, Wheel of Wisdom, James Montemagno, Frank Krueger, Merge Conflict Podcast

Full transcript (original language)

00:00 Welcome back everyone to Merge Conflict, your weekly developer podcast about all things in the world of software development and new LLMs or not LLMs, who knows, new models, new things. I’m one of your hosts live from Vienna, Austria, James Monttoagno, and with me as always, the man on an island, Frank Cerr. How’s it going, buddy? >> Oh, it’s going well. But where in the world is James Monttoagno this time? Because this is not the normal background. You are showing me something different. I like it though. It almost looks like Are you in a cubicle? Are you in a cube farm? It it really does kind of look like I’m in a cubicle. So, uh, yes, I was at NDC Oslo. I recorded the

00:35 last one from Oslo if I I’m pretty sure I did at least, right? >> Or London. I don’t know, wherever I was. >> I was in London and then I was in Oslo and then I was in Germany. Now I’m in Austria. We went to Budapest today and then we’ll be back to London and then I’ll be home in Oregon uh by Sunday night at midnight and then fresh to work at 7 a.m. Very exciting. [laughter] So, Uh >> well, I’m glad you get to be our foreign correspondent yet again. I can’t wait to hear about what’s happening in Vienna.

01:05 Uh I’m I’m sure this is just a travel podcast now as far as I know. >> You know, uh Vienna is very beautiful. We’re just talking about very very beautiful. Budapest also very beautiful. We were in Insbrook, then we’re in Leo Gang, we’re in Salsburg. All very, very pretty. Vienna just we were walking around even though we had already been here once and like man so just beautiful. Yeah, everything is so regal and big and just beautiful and >> speed of an empire. [laughter] Yeah. >> Uh it’s uh it’s it’s it’s crazy. Um if you’ve never been to Vienna, I definitely re recommend. It’s very very

01:37 lovely. Uh and yeah, so excited to be here on the road. We took this this week off between NDC Osel and Heather as an event here in Vienna. So that’s why we’re here now. But we went did a bunch of sauna stuff in between. And we took in 5 days five aerial tramways lifts up to mountains. Five of them. >> Fino. >> Fino. [laughter] Yeah, >> that’s your favorite topic. I can’t wait to see what AI is going to generate about these ones this time. [laughter] >> It was good. I’ll tell you this. Okay. So, before we get into some new models, so there’s a few things that

02:09 >> Okay. >> I I think that I wanted to talk about because I just we had a little bit of fun on this trip. We took um we drove a bunch. We took some trains. Uh we took a bunch of buses >> and one thing that we forgot to bring was a Kather had a has a ebook, you know, uh and I have my Switch too, but we sort of forgot to bring something that we could do together. We saw some people playing cards and stuff. >> Uh and one thing Heather and I both read like we like Family Feud, we also like Wheel of Fortune. Heather’s really good

02:40 at it. So I built uh a two two or three player Wheel of Fortune game. And it’s called wheel of wisdom. I don’t want to get >> uh it’s at wheel of wisdom. >> Hallucinated trivia. Sorry. Say the URL again, please. >> Wheel wheel ofwisdom.app. >> Wow. >> Um it’s a full progressive web app. You can download it offline. Works. There’s 600 puzzles. Stores your history. Stores your games. It’s got a bunch of different categories. Uh it’s got trips

03:12 and mysteries and all sorts of stuff. And there’s going to be a um I’m I’m just getting ready right now to start a a full fresh build uh for an iOS application too. It’s going to be a Net Maui hybrid app. So I’m going to reuse this. >> Oh yeah, cool. >> HTML and JavaScript. It turned out really good. I built this baby with Baby Luna uh that we’re going to talk a little bit about this week and a bunch of other models too. I used a bunch of models on it based on the task at hand, but it’s really straightforward. It’s just HTML and JavaScript. Nothing fancy,

03:42 but it’s just like I built it from my phone, Frank. Like this was the thing I wanted to talk about. I didn’t even I have my computer with me. I’m here at the beautiful Moxy Hotel. Um you never stayed at a Moxy. It’s like very funky and you know exactly what you’re getting, which is a tiny room for not a big price. >> Um and they’re everywhere. It worked out for us for a location we needed to be. And but as we were going via bus, via whatever I didn’t have my laptop and I coded this 100% from the GitHub mobile app. >> Um, pretty cool.

04:12 >> I even bought the domain name, set up the DNS, everything 100% from my phone, >> uh, using just the cloud, you know, agents basically, which is is absolutely ridiculous. And, uh, we’ve been playing it on every single thing that we did. So, I just want to kind of throw that out there. >> Fantastic. I honestly I think you’re just feeding into your uh domain registration addiction. I think you’re just looking for excuses to buy new domains, but um this is pretty cool. I haven’t actually made that leap of doing it all from a phone yet. Um so once again, you’re ahead of me. I’ll give it

04:45 a shot. I’ll write some trivial game on as a web app on the phone. [laughter] And and then this is this is a fun trivial game because already there’s dollar signs at the top. So I can imagine you two trading money already. I I hope there’s gambling rules and drinking rules with this game. >> Oh, yeah. >> Um, but it’s pretty fun. I I’m excited to hear what your Baby Luna experiences are because um I don’t know if you’ve been getting the news over there in the Europe, but um over here in the USA uh

05:15 we’re not making models anymore. We’re slowing down. We’re we’re slowing progress down. No, no more model releases. And by no more model releases, we mean both major model providers are going to release a whole new suite of models [laughter] in the last week. I don’t know if you’ve gotten that news, but uh we’re we’re both slowing down and speeding up over here. How is Europe doing? >> Uh Europe, the interesting part about Europe is there’s actually more AI kind of everywhere than you would you would think. So definitely like there’s a few

05:47 small towns we go on the Oregon coast where we like are on the Facebook. was the only time we go to Facebook because there’s usually each community has like a Facebook feed and almost all the graphics for like the the restaurants are all AI generated for 100%. here. What’s really fascinating is they’re really into AI video like uh stuff >> scary. Okay. >> Ad advertisements and they all say AI in the bottom right corner, but there there’s a lot of them uh quite a bit in train stations and things like that. Um which I haven’t really seen in the states. maybe in San Francisco or something >> uh more I haven’t really seen it in

06:20 Seattle but there seems to be more of that around and and definitely more seeing uh but I’m not seeing like big ads obviously even here in Vienna for like claude and you know open AI and chat GPT and all this stuff so I don’t really know how that’s going necessarily but I will say even though I’m here in Europe my Twitter feed is still filled of a whole bunch of stuff and funnily enough Frank um Both Burke and I have been tweeting a bunch about Luna and I tweeted a day a day a day before GPT6 Luna came out. I said Luna Max.

06:55 >> Yeah. >> And I swear I swear to God I had no idea that this model was coming out uh at all. Um but GBT6 Luna and GPT6 Soul. No more Terra. Terara’s gone apparently. Um I guess >> who needs Earth? Earth overrated. >> RIP. [laughter] That’s what I hear. >> Those both landed at like half the price or whatever. >> Yeah, it’s really weird because both companies are playing the game of like our token counts are changing dramatically and we’re modifying our

07:28 prices. So, it’s just it’s I I take it as the money people just messing around so that people can get roughly the same amount of work done for roughly the same price. Technically, they lowered their prices, but if you look at how token hungry, especially on Macs, some of these models can be, it’s made up for with those new trainings. Um, it’s a little bit all over the place. Honestly, it’s so hard to keep up with this stuff now. I’m surprised we’re even trying to keep up. Thankfully, I have you. You keep me in the flow. Um, but also,

07:59 I I noticed with um GPT, like they flat out say like come middle of October, we’re throwing out the old models. So like update or GTFO, you know, you got to keep up with us. And so I used to make an argument like, oh, we we should keep with the old models for a long time. You get to know them. You get to know their um pros and their cons, but no, that’s not the world we live in. We’re moving in a fast-paced world where these models change every month or two.

08:30 Um, yeah. And you you I think it’s our podcast, but Luna’s really caught on uh for good reason. It’s cheap, [laughter] especially Luna Max is performing just as well as like I forget the exact numbers, but GPT 56, GPT 6, uh, soul at like lower medium thinking level. Luna Max is crushing it. So, you’re such a trends setter, James. Like Luna Luna’s out there. I I haven’t been using it too much. I’m an idiot. I’ve been using Terra. What a dummy over here. But um

09:05 seems like the rest of the world’s caught on to your little game. Cheap Luna for everyone. >> Cheap Luna. And like you know, you have to there’s trade-offs, right? When you put on a an extra higher or max reasoning, uh while it is crazy fast, it is going to be chit cchaty a lot more. It’s going to be burning more tokens. That being said, I did a demo at NDC London where I built an interactive I’ve done this a bunch of times. I did this at um Copilot Day, but I built I built and deployed in real time a full

09:38 interactive Oslo day tour which brought in a bunch of uh coffee shops, sites, and saunas. And then you could plan your day. It do a whole map and it was like it was an absolutely it used impeccable. used this beautiful website. I set it to Luna. This was 56 Luna at the time, Max. Um, and I said then, you know, publish the GitHub repo, set up the GitHub pages, verify everything, like click around, do the thing. It did all of it. It cooked the whole thing for about six minutes.

10:12 You that’s including the CI/CD pipeline. Six or six or seven minutes, >> 18 cents. >> Yeah. [laughter] >> And it was it was >> gorgeous. In fact, let me see if I can find it for you because I think you know if you go to my GitHub and I’ll I’ll post the link here for you. But if you go there and just type in uh Pi Oslo, I bet that’ll pop up. I don’t know what what is it called? Let me see what I call it here. It’s called uh we can all see what Baby Luna’s outputting for CSS. I I have CSS opinions because I have been fighting um

10:46 some web development with it and it it loves to bounce between coordinate systems. Like sometimes it uses M’s, sometimes relative M’s, sometimes px. It seems like these models just um >> they love CSS. They love generating CSS and they love being inconsistent with CSS. So I’m excited to see uh little baby Luna’s CSS. >> And this one honestly is just HTML, JavaScript CSS. It’s just like here it is. And I’ve run this demo a bunch of times and sometimes it does pick a framework. Some and this was on

11:16 autopilot. It asked no questions, did no things. >> And obviously I had very little input, but the whole goal is like here it is. You know, it’s not perfect. There’s a little bit of overlapping text here and there. But it has a grid, has a coordinate system, it has like, you know, all the different places. You scroll down, you have like different, you know, different beautiful pictures. You know, it’s, you know, it’s pretty good. You can copy to a link. you can like filter. I mean, for a oneshot, no big deal. Like, it’s a pretty nice looking app. >> Well, you know, I I I know it’s like it even did mobile pretty well. Um, it has,

11:50 like you said, a little defects around mobile, but what I’ve really learned, and I’ve I’ve had to change my whole attitude this last week. I was getting frustrated with models creating bad UI and bad UX, as I always am. That’s my perennial topic on this show. But I had a recent epiphany. I’m like, these models are never going to create the perfect UI for you because I have opinions and all that stuff. So, I just need to accept that it’s going to do a rough draft first and then we’ll refine that draft and then we’ll refine and

12:23 we’ll refine and we’ll refine. So, I used to get upset and I’m like, you big dummy. Why are you making that text bold? That text is obviously secondary. It doesn’t matter. Hide that away. Make this other thing bold. you totally forgot to do this thing I told you. I used to get frustrated and then I had a moment I’m like what’s happening here Frank you’re getting frustrated at a bunch of bunch of matrix multiplies like it is not helping the situation by you getting emotionally involved with this model producing bad code. So, I I had to

12:54 do a forceful mental shift this week of it’s going to do a crappy job the first time, and I’m just going to joyfully point out all its mistakes, [laughter] ideally with um dictation, so I don’t have to type. I’m curious how you were um interfacing with your phone if you were doing dictation or not. Um and then just, you know, if it takes 10 iterations, it takes 10 iterations. That’s just what it’s going to be. >> Um so, oneshotting’s cool. It definitely makes for good tweets, oneshotting and

13:24 all that stuff, but I’ve come around to using a slightly dumber model is okay because we’re not doing rocket science here. A lot of time it’s just human stuff that I want to change about these. And that’s why I’ve been interested in the smaller models like the Lunas and the Teras, even though Terra is not cool anymore. Um, but I I totally get where you’re coming from of using these smaller models because if I’m going to be iterating, I might as well get quick iterations and not be spending a fortune on it. >> Yeah, it’s hard. Like uh OpenAI was doing so many promotions with Soul that

13:56 like Soul was at some point cheaper than Terra, which is kind of crazy. And I am a big Terara fan. It was like one of my it was basically my default model for a long time until I sort of really started experimenting with Luna Extra High and Max. Um, but I agree with you on the UI bit. The thing that I’ve been doing the most is like taking especially in my mobile apps taking uh screenshots and then I mark it up and I say this this and this and this you know this here >> and that’s really helpful. And then the one thing I’ve been really interested in is like I use Impeccable on the web. It’s kind of built into the C-Pilot app.

14:29 It’s but you can al just install it. It was just an npm package impeccable and it has a bunch of design guidelines and things but it also helps you put design uh best practices into sort of impeccable folder and then with the impeccable skills it’s like a whole bunch of skills that are like I’m doing design let me check and let me generate it’s kind of keeping itself up to date kind of like your thing that you were talking about I think you had the styles >> UIMD yeah uiMD yeah >> it’s a difference and I I want to be clear like I don’t like style and design and user experience being mixed

15:04 together. >> That’s a good point. >> You know, data flow, information flow, what inputs go where, all that. That’s user experience. That’s UX. That is how does the app feel. Uh Apple used to call this human interface guidelines. Then it became UI, but now we don’t say UI anymore. We say UX to separate it from I don’t care what color it is. I really don’t. Um you know, maybe when I’m bored, I’ll work on the design and style. Like that’s why I don’t care too much about impeccable because to me CSS styling is completely secondary. I would if I could convince these models that

15:37 they all refuse to. I would say don’t generate any CSS. You are not allowed to do any CSS. No styling. Let’s work purely on information flow and all that stuff first and then we’ll get around to styling. I just want to point that out because you’re you’re mixing the two and it just it’s a little peeve of mine. Instant terrible at UX. >> You are correct. Frank Krueger, your fort. Um, you know, we talked about many times on this podcast, thank you for calling me out and correcting me. Let’s get this merge conflict back into the correct uh commit state here, is that I was just having this conversation with

16:10 Heather about AI and design. And then I said, you know, the thing that I talk about with Frank all the time is like the actual user experience or the taste. Like there is a design taste, but there’s also a flow taste as well in the user experience of how people are touching interactive. Does it fly out? Does it do this? Is it coming up from the bottom? What’s the swipe interaction with? What’s the button press? What’s where is the placement? Where’s the like what’s how how are we moving around the application? Um, that’s a really important, even more important to be honest with you. And and an AI can see a

16:41 thing and it can also obviously like press buttons, but it’s not going to have that sense of is that a satisfying movement? Touching the mic. [laughter] Yeah. Um yeah, and and and I find that that’s it’s the information flow and such that they tend to get wrong. All these models are honestly good enough at design. They they love their um purple to blue gradients, as I like to say. They’re they’re really good at generating all those gradients and all that. And it’s fine. I I let them play on that all day.

17:13 What I find is they’re just bad at information flow and prioritizing items and all that stuff. But like I said, um I’ve I’ve learned to not even be frustrated with them anymore. I just sit back and regard that as um yeah, iterative. We’ll we’ll work on the information flow one bit at a time. >> Yeah. Uh well, these new models are out. There’s obviously the the Luna, the the five Luna and also the five soul. They are half the price of the previous. And also, I will point out that uh the default before for Luna was 200

17:48 uh K on the uh context window. Now it’s 272. It’s It’s gone up there. So, it’s matching the other ones. >> Yeah. >> But we also, like you said, got another We have more models, but we did get another model from our friends over at Anthropic. >> Uh uh Claude 5.5. I don’t know where that came from. The Claude Opus 5.5 is >> Yeah, it’s almost a happy birthday. Like, can do you remember Opus 45? I think that’s when we all lost our jobs because that was the first model that actually did a good enough job where that was the big transformation last

18:21 year I think from definitely for me it was like oh here’s here’s some code I have in front of me I don’t feel like typing please modify the code to do this to here’s an app I want just write the stupid app for me >> and that was the big epiphany with Opus 45 so I think the five was them doing a little bit of marketing and pushing and getting Now, um this uh 5.5 anniversary model is how I’m thinking of it. Like we had to show that we’ve actually improved over Opus 45 because as we all know, Opus 5 is garbage. Just [laughter] put

18:56 that put that thing down. Delete it. We don’t need that thing. Um so hopefully 5.5 is their um their swan son. No, that’s the wrong analogy. Um their apology letter to all of us for a bad Opus because we’ve all been rocking Fable. Fable’s been great, but Opus 5 has been trash. Um, so I I’m I’m excited to try maybe OP Opus 55 now. Again, >> I definitely haven’t had that experience with Opus 5. I feel like when I’m doing UI work, it does still do a a pretty good job of of different things. That being said, it’s still really expensive,

19:30 right? It’s definitely the most I mean, this is the cheapest Opus yet. And obviously it’s, you know, 25% cheaper than the other one, which is, you know, 50% of of Fable, but still really expensive model at the end of the day. Especially when you’re thinking of, for example, >> you know, Luna or Soul, which is going to come in, Soul is going to come in uh let me see five soul here. Yeah, be cheaper, right, than than it uh or I guess maybe it’s the same price. It’s very No, it’s cheaper. judge cuz it

20:01 really depends on how many tokens they use and how >> it >> our pricing models are absolutely ridiculous right now. >> If you were to use the default context on six soul, it’s half the price of five. >> Um and that’s not including like you know the cash other stuff and stuff like that. But yeah, so there’s a bunch of things in there. Now that being said, um like you got to remember also Fable is still a model that doesn’t have that. I mean, I think they’re now doing it with 51. It doesn’t have that zero data

20:32 retention policy, which is also of interest. You’re kind of doing I think it now does in 5.1, >> but I do think you’re right. Like, I think when they do the the five, they’re trying to make a statement, right, that okay, we had five, four, five.1 or whatever. This is like the thing, right, >> that we’re doing, which is important. Um, >> but the question is like, yeah, how often you need these like crazy super heavy models, right? That’s the real question at the end of the day, especially if you’re going to go on a higher reasoning. I mean, those tokens gota be crazy. Yeah. And I like that we we’ve gotten to a place where the benchmarks and the performance graphs that people are

21:05 publishing, the x-axis has become dollars now, which I really appreciate because it used to be other arbitrary things like tokens or time and all that. And in the end, it kind of doesn’t matter. What we care about is dollar bills. Um, how much does this test cost? And I like that the benchmarks are actually judging themselves like what was our our success rate and how much did it cost? what was our success rate and how much did it cost? And I think it’s taken us a year, but we finally settled on that’s the graph everyone’s interested in. Because when you make that graph, that’s when you see your

21:38 beloved baby Luna far on the left side away from everyone else being super cheap but still going about halfway up that graph vertically. And if you want to get the other vertical distance, then you got to move to the far right of the graph [laughter] where everything becomes very expensive. So there’s actually a weird kind of empty zone a little bit in the middle territory. And also everyone please pay attention to whether that xaxis is logarithmic or linear. >> A lot of people cheat and you can get really confused with how much something is costing because a lot of times they

22:11 show a logarithmic one and that makes the very expensive models look like they’re only a little bit more expensive than Luna when in fact they’re 10 times 20 times more expensive and all that. But I I like that that’s where we’ve gotten as an industry. Like here here’s here’s how much it’s going to cost for you to get 50% on this benchmark. >> Yeah. No, it makes sense. I still think, you know, pick the right model for job. You know, I still use a lot of MI code 1.1 Flash at the highest for bug fixes specifically. I still use a lot of Luna.

22:41 I still do some deep dives into Soul and into to Opus on occasion back and forth. Um, and but >> maybe I’m a weird programmer, but I still like to chat with these things. So, I still judge them a lot on their ability to explain and talk. So, their conversational ability. And so, when I was making fun of Opus, a lot of it was because of that. It was just a really bad writer. It would just it would introduce words you don’t know. It would throw away concepts. It just bad English, basically. It was just bad at

23:13 being a chatbot, which is kind of what the basic skill should be. So, I’ve I’ve learned if I want to actually have a good conversation, I need to be talking to soul roughly something like that. Um, some of these smarter models, they might win on the benchmarks, but when you’re trying to have it explain to you how a bilateral filter works, [laughter] it’s better sometimes to have the slightly more medium models that are just better at explaining things and speaking English, basically. And and I also want to point out, don’t sleep on these model routing um things that we

23:48 talked about recent like hydrofusion. I just I think it’s out in VS Code Insiders now. You can use it there. I’m not sure if it’s in the app yet, but I’ll point out that I took this wheel of wisdom. It’s not like it’s the most complicated thing in the world, but I wrote this big prompt. I talked to the the CLI. I turned on Hydra Fusion. said, “Hey.” And I pointed at the the Don Maui hybrid web [snorts] view docs and I said, “Hey, listen. Here’s what I want you to do. I want you to do blah blah blah blah this whole thing with the shop blah blah blah.” It went and it cooked, dude. It cooked for like 20 minutes, 25

24:21 minutes. Came back, one shot of the whole thing, like tested it, did it. It’s like I ran it, I verified it, I put it in the mobile camera, did all blah blah blah. $4 the whole thing. Yeah, >> like it was it was it was amazing and it was rubber ducking back and forth and like that’s a thing where I’m like I’m doing a big a bigger thing, right? Let me do something that’s going to really >> duke itself out between these different models. And and I thought it was so cool. But frank’s a new model that is it even a new a new model? I don’t have

24:51 access to this model, but I know that the I think that the internet may have gone I think the internet may have broke >> because while Baby Luna seems to be cheap and fast, apparently there’s the the cheapest and fastest thing in the entire world. >> Oh, >> it’s um Bev. Is it Bev? >> Bev [laughter] Jeb. >> Jev, which is a terrible name, but whatever. We’re all getting used to terrible

25:21 names. Um, yeah, it’s it’s you know, you know, the fastest way to do something is to not do it. Then it’s done instantly. [laughter] >> Checks out. >> So, it is a large language model. But, um, unlike all the ones we’ve just been talking about and we use every day and have proven themselves useful to us, we’re not using them because of marketing. We’re using them because they’re actually useful and we all like them. Jev doesn’t output text. It doesn’t output code. It doesn’t chat.

25:54 So, it’s kind of a funny, it’s still a large language model because you still input to it with unstructured natural language, but its output is not unstructured natural language. Its output is a category. Basically, [laughter] it’s hot or not for prompts. That’s what I call it. Um, it’s a new model out there and it took the the internet by storm. My god, there was it was the only thing you could read about there for a couple days. And it is fascinating because it’s a new modality for these

26:26 large language models. There’s a lot of things we can do with these models other than turn them into chat bots. We found a lot of success with chatbots and then agents. But someone thought, hey, what if I train an LLM to do something else? What if I train it to be a really good classifier? What about a really good general purpose classifier? And what if we make it kind of fast and train it really well to do that? And that broke the internet, this classifier. >> Yeah, I watched the um CJ over on the Syntax FM YouTube and podcast break it

26:59 down. And the thing that he kept saying over and over again was classifier, classifier, classifier, classifier, classifier. He’s like, you know, was good at classifying. He has some really good um demos, you know, you know, he he was like, “Hey, imagine you’re in a word processor and you’re trying to analyze if this is, you know, a certain tone, a certain if it’s AI generator is this, and you can like write these rule structures.” He also used it in their podcast metadata generator, which everyone’s got one apparently currently syntax too, but they were saying like they take their their recording, it goes

27:29 through a a transcription model, then it goes into LM to write all this stuff. And he said the biggest thing is like what you can do with Jev is basically write rules like it can answer questions like you give it the the data and you give it questions to answer and it can basically do a yes or no or like a categorization type of thing which allows it to check check itself right so given hey the LLM output this it can then classify yes was it accurate was it not accurate XYZ and and and and then and then if it it goes down and is

28:02 classifies correct corness or needs human review, then they they routed it in a certain way, which seems interesting. Uh or they were using it. I saw a bunch of demos for like self-drivless cars and like these decision trees and all this other stuff. I don’t know. Um someone did like an air airplane tracking price thing or something like that. I don’t know. But I I couldn’t turn off the internet and I still didn’t get it, I guess, uh in general, but because I didn’t try it. I didn’t have my computer. I didn’t even know I don’t even know how do you use it? How do you What do I got to do? Do I got to go buy something? So, no, it it’s it’s actually pretty easy to use. Um,

28:34 like I said, it’s input is still natural language. So, you can give it all the information in the world. So, you can, you know, tell it a story or ask it a question or something like that. But what you need to do is specify its output schema kind of. It’s not even a schema, but basically given a question or something, some information I want, here are the potential answers for that question. So you can’t have it like calculate a number. But if there’s 20 possibilities and you wanted to pick from one of those 20 possibilities, you just enumerate those 20 possibilities and then it will pick one of those

29:09 possibilities and give you a confidence score. But that’s a trivial thing for output because all these networks are probabilistic. They all have probability scores. They just don’t show us the probability score. >> Yeah. [laughter] >> But they all have it. That’s how that’s how they work. Um, and it it’s clever because let’s say I give it a whole bunch of text and I have 20 questions. Well, we all know that there’s basically two parts to using these LLM. First, we do what’s annoyingly called the prefill where it’s taking all the context and feeding all that information into the

29:42 LLM. And then the next half in a normal one is the decoding stage where it starts generating more text and all that stuff. Well, it turns out when you’re doing that input thing, it’s already doing predictions. And so it’s already predicting the next token. That’s how all these things work. And so Jee is just kind of clever in that it’s always kind of predicting just the next token is going to be one of those answers. It’s a discrete set. It’s

30:12 one out of two answers. That’s your true false kind of stuff. It’s one out of 20. If you say, “Here’s 20 potential answers to this question. pick which one of these is best. Um, and it can be a rating even. So like you know um it’s just not just hot or not, it’s how hot. So is it cold, medium, hot, you know? So it could be three levels like that. >> And because all these models are just always outputting the next token. This one is just always outputting the next best guess. And if you have 20 questions for it, you don’t have to do that

30:45 prefill step 20 times. it can just kind of fork the model at that last little token and then um very cheaply answer the 20 questions vaguely in parallel. Um so that’s why everyone likes it because of that inexpensive nature of it. All you’re paying for is those input tokens which as we all know are kind of the cheaper ones for reasons. It’s doesn’t actually make that much sense but that’s just how the world has evolved. The markets have evolved. But the nice thing

31:17 is if I have 20 questions for one set of inputs, it’s pretty dang cheap because it’s only really generating 20-ish tokens then as output tokens and it can do all those in parallel. So it’s also fast. So that’s like at a low level how it’s working. Um but pretty cool. I mean it’s just a clever way to use these models that we’ve had for a long time. So, I wouldn’t, for example, like take, let’s say I wrote a blog post. I wouldn’t take this blog post and then I wouldn’t say, “Hey, go give me

31:50 five spicy titles that you would come up with.” But if I said, “Hey, my blog has 50 different categories. Given this text, go figure out what categories it falls into.” It could do that. >> Yes, absolutely. Or you can go for the hybrid approach. Have one network Luna generate 20 different titles for the blog post and then have this pick between them because maybe it’s a bigger smarter model. Maybe it has better

32:21 judgment. Maybe it has better taste because it’s been refined to have better judgment, better taste, but it’s not generative in the normal sense. So you could do a hybrid trick then. >> And then you could write specific questions about how you want your titles to be. Is it clickbaity? Is it blah blah blah? Is it what? and it can answer those yes or no and that could go into this algorithm to get the now that’s kind of cool and I think that’s what CJ maybe was saying where he’s like okay this is where the LLM is generating their podcast metadata and then we use this to like do the and they come up

32:51 with the ideas that’s really neat actually >> uh >> yeah because it’s not spending all of its time generating text they can make the model really big and powerful and smart and tuned to this capability of choosing between multiple options really well. Whereas the current models are tuned with with the reinforcement learning, the human feedback reinforcement learning that we all use, they’re tuned to be chat bots and agents and all that. They’re not tuned to make decisions necessarily. They’re they’re obviously making plenty

33:25 of decisions, but they’re not tuned to that. This one can be tuned to that exact problem. And so it’s just smarter. It’s it’s just a more efficient use of these models and you’re guaranteed the correct schema output because it’s not like we’re parsing JSON in the end even though it’s all presented as JSON. We’re not. It’s really just generating one option out of X number of options. >> Efficiency. Yeah, it’s you know when when transformers were first invented with the paper um attention is all you need. The LLMs had were two big models.

33:58 There was the encoder and the decoder. And then a couple years later, we decided, you know, we can simplify this whole kitten kaboodleoodle to just the decoder. So now that’s all we use today. That’s happened with GPT. So we took the transformer model, we threw out a good half of it because we said, you know what, that part’s actually just extra complication. We don’t actually need that. And look at all this amazing stuff we can do with just the decoder side. I like Jev because it’s saying, yeah, that’s cool, too, but you know, we don’t actually need to be generative all day long. we can chop off um the the

34:33 recurrent part of this whole thing and just output something an answer. And so we’re just we’re ref we’re finding it’s still the same thing. It’s still a transformer. It’s still an LLM. It’s using all the same architectural principles. People have made junky versions of this already. It’s easy any any idea can train a network to do what this is doing. Hard to make a good network obviously. Um, but I like it because it’s just another version of the same thing. Like this architecture, this transformer architecture that was invented has way more potential than just what we’re doing with it today.

35:07 Chat bots became a thing. First predictive text became a thing. That was a little boring. Then we got chat bots. Those were great. Then we turned chat bots into agents. Fantastic. And now this is just like, hey, here’s another thing we can do with these LLMs and it’s efficient. We’re not paying for all these output tokens. Yeah, that’s smart. Yeah, that that makes a lot of sense. Have you used it for anything yet? Have you tried it out? >> I’ve definitely I got access to it thankfully. Um my little brain wasn’t able to come up with any great scenarios [laughter] for it. So, I was asking it

35:37 the dumbest questions. Of course, I fed it a bunch of stock market data and I’m like, “Hey, which one of these stocks should I buy? Which one should I sell? Which one should I hold?” Do not use it for that. That’s a trap. Um but you can you know because you know buy, sell, um whatever, hold, you know, those are three options. You can ask any network for those. Um other people were doing a lot more clever things. But that that’s always my stress test. I I like to just see what it has to say. So I would say like um uh I would make up data. I’m like here’s the Microsoft stock is like crashing. Should I buy it? And it’s like

36:10 33% chance you should buy it. [laughter] Like okay, it’s crashing. But they’re like, yeah, but it’s Microsoft. It can’t say that because it’s not generative. But, you know, that’s what it’s thinking. It’s like it knows what Microsoft is. So, even even though I gave it false data of Microsoft crashing, it’s still like >> maybe you should still think about holding it. [laughter] >> Yeah. >> So, it was fun to just like do those kind of little stress tests on everything of it. But I think about other things like um people had it playing Doom, which is really hard. It’s a fast model, but it’s not that fast because it still has to take input tokens. So you still have to translate

36:43 um images and visual things and other um media into text because it’s predominantly a text model other than people are working on multimedia versions where you can give it a picture and have it answer you know multiple choice questions about a picture. Um it’s there’s a lot to play with it and again it’s just it’s just another use of a transformer. We’ve had classifier transformers in the past. um we’ve all used them especially in our early days when we were using much smaller transformers. So this is kind of

37:17 honestly just revisiting an older idea of these classifiers that we all used to use and just making hey for the last three years two three years we’ve really learned how to scale up these LLMs and make them huge and intelligent and we’ve learned this trick um human feedback reinforcement learning let’s apply that to classification also and I don’t want to be reductive by keep calling it a classifier but that is what it is fundamentally at a machine learning level. It’s just a big general purpose classifier. But the neat thing is it’s a

37:50 big smart model that’s been tuned to be good at classification and that’s why it’s useful. >> Specialization as I like to call it. >> Yeah. >> Exactly. Yeah. >> Well, I don’t know. Did it deserve to take over the internet by storm? No. That’s the bubble. [laughter] >> Yeah. >> What generative LLMs have proven themselves. They’re fun to chat with. Um they help us in our day-to-day work. They help you write weird apps while you’re on vacation. They’ve proven themselves. Yeah,

38:21 >> this one is an old idea given a modern twist. Um, and it took over the internet because we’re in a bubble. That’s fine. But it is it’s but that shouldn’t detract from it either. It is a good idea. It is a more efficient use of all these things. And it is nice to see a modern version of that. And maybe because of the the cost and the very specific use case, there are applications today that might be doing some things like think of WordPress, right? out that could be be able to infuse a bunch of little things and give you all these recommendations and all these little classifications for your

38:52 blogs or for your YouTube for example like your videos and all these other things that are doing I think I saw someone they they wrote a plugin I think for Twitter that was like a classifier of all the tweets and it would automatically hide the tweets or whatever like in real time that are maybe biased or or spicy or whatever. So stuff like that. But I could imagine a different use case for even some of the apps I’m talking about, the podcast generator and some of these other things that maybe I’m using an LLM like you’re saying, but I’m also using this to classify um in correlations. That might be a good uh trial run to do. So very cool.

39:24 >> Yeah. And you brought up probably the most useful use case that is probably going to be used for at least in the immediate future is if you’re using something like open router and you have a choice between 12 different models to use to actually answer the question, I need a classifier. choose a model for me. You know, why why pay for an expensive model to do that classification? Might as well pay for a cheap Jev uh to choose which model should actually do the dirty work. >> Yeah. I’ll be interested if Jeb takes over the router world. So, very >> Yeah, basically. Yeah. All right. Well, I’m going to go hang out with my wife. Thank you so much, Frank, for giving me

39:58 the lowdown on Jeb. I did see it all over the feed, but I said by the time I get back, there’ll be something new. But Frank’s going to tell me all about it. And then I got a bunch of new models to talk about. So, I was like, “Wow, he’s so crazy.” Never stops, man. >> Um, >> all right. Well, go play with all his new toys and set that Luna to max and have just give it a rip. It’s it’s it’s a good time. It’s so cheap. It’s just >> it’s it’s awesome. Um, all right. That’s going to do it for the six merch conflict. So, until next week, I’m James Watsono >> and I’m Frank Krueger. Thanks for watching and listening. >> Peace.