$100M ARR Founder: The Secret Behind Great AI Apps

Author: Greg Isenberg · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: $100M ARR Founder: The Secret Behind Great AI Apps · Channel: Greg Isenberg · 48:57 · Watch on YouTube

Topics: AI agents, Enterprise AI applications, AI evaluation and benchmarks, Software engineering with AI, Multimodal models

Summary

Gamma co-founder John Noronha explains an agent harness as the tools and workflows that let an AI agent gather context, use models, inspect its work, and deliver a finished result. He describes how Gamma uses many models and calls to create and visually check presentations, while arguing that founders can differentiate through domain-specific tools, taste, proprietary data, and a clear point of view. The conversation also covers AI smell, subjective evaluation, experimentation, model selection, and emerging opportunities in vertical AI products. Noronha’s closing advice is to choose a niche you genuinely care about, build for its users, and stay flexible about the product as you learn.

Key points

An agent harness is the set of tools, context, and processes an agent uses to complete work, inspect its output, and respond to user preferences.
Gamma’s presentation workflow uses multiple models for context gathering, questions, outlines, template selection, generation, and visual quality checks.
Distinct agent harnesses are most valuable when work requires subjective judgment and users care about the specifics of the result.
Potential competitive edges include a distinctive vision, proprietary data, and integrating AI into real-world services or specialized domains.
AI smell is the recognizable sameness of AI-generated work; Gamma aims to offer varied creative styles rather than one signature look.
Objective evals are useful for measurable requirements, but subjective quality often calls for human taste, user feedback, and large-scale A/B tests.
Gamma frequently experiments with models and product flows; removing steps did not always improve satisfaction, engagement, or paid conversion.
Founders can start with existing models and focus on usage and harness design, then consider fine-tuning or specialized models for cost, personalization, or hard tasks.
Vertical AI opportunities may serve consumers, prosumers, or professionals; durable products can emerge by starting in a niche and expanding to adjacent needs.
Noronha advises founders to choose work they can believe in and use themselves, and to be stubborn about the vision while flexible about product details.

Chapters

Start	Chapter	Summary
00:00	Introducing Gamma and agent harnesses	John Noronha previews a discussion of practical agent building and Gamma’s role as a creative agent harness.
01:34	What an agent harness is	Noronha traces Gamma’s shift from preset LLM workflows to an agent that uses tools to create visual communication.
04:07	Inside Gamma’s generation workflow	He walks through the sandbox, context gathering, user questions, model calls, presentation generation, and visual quality checks.
08:13	Finding opportunities for vertical harnesses	They discuss where specialized harnesses can help, especially in domains with subjective decisions and strong user preferences.
13:26	AI smell, evals, and taste	Noronha explains how AI-generated work can feel generic and why subjective creative quality is difficult to capture with fixed evals.
18:06	Measuring value and finding users	They discuss signals such as editing, sharing, payment, and retention, as well as the distinction between consumer and professional AI spending.
20:09	Personal agents and building on models	Noronha describes a vision for a Slack agent that learns from behavior and explains why startups can begin with third-party models.
23:12	Models, prompts, and experimentation	The conversation covers closed versus open models, context engineering, prompt use, and how Gamma tests new models.
28:22	Product friction and marketing engineering	A test found that some review steps improved satisfaction, prompting a discussion of experimentation and increasingly technical marketing roles.
32:30	Agents, data, and managing work	They discuss automations, the role of marketing engineers, and the challenge of supervising layers of agents.
36:04	Building an experimentation muscle	Noronha argues that learning from many tests takes practice and that founders should expect to work through disappointing results.
38:37	Underexplored AI image opportunities	Noronha sees untapped uses for image models and explains how specialized harnesses could address practical needs such as furniture shopping.
43:16	Vertical products and adjacent niches	They distinguish consumer, prosumer, and professional opportunities and discuss how a broad business can grow from focused niches.
45:19	Startup ideas and founder advice	Noronha considers creative technologies and games before advising founders to choose a niche they care deeply about.

Quotes

I think maybe what harnesses illustrate is that actually the wrapping is most of the work. — 03:36

One type of edge is what you just said, which is you have a distinct vision of the world and you are just moving faster towards it — 11:21

Ultimately, what founders can bring to their domain is taste. — 17:05

Even though it was slower, the people that got through felt more invested in their output and did more. — 28:22

Be stubborn on the vision, flexible on the details. — 48:57

Concepts and entities

agent harness, Gamma, John Noronha, Claude Code, Anthropic, OpenAI, AI smell, evals, AB testing, Optimizely, Slack, Notion, Linear, Salesforce, Zapier, n8n, OpenClaw, Hermes, modelos de imagem, marketing engineer, vibe coding

Full transcript (original language)

00:00 Today we’re talking with John Noronha, co-founder and chief product officer of Gamma, and we’re going to talk about agent harnesses, all the opportunities around it, vertical agent harnesses. I can’t wait to dive in, and I want to give a special shout-out and thank you to Anthropic, the team behind Claude, for sponsoring today’s episode. >> [music] >> John, thank you so much for coming on the podcast. By the end of this episode,

00:31 what are people going to learn? >> Thanks for having me, Greg. I’m really excited to be here. Today we’re going to dive deep on what it means to actually build an agent in practice, including demystifying what is an agent harness, and what that looks like for us at Gamma, which is a creative agent harness, an agent for visual communication. >> And not just you’re being humble. Gamma’s, I mean, it’s it’s grown extremely quick. I think it’s hit 100 million ARR. You’ve just relaunched the platform, huge huge release, and there’s a lot you’re learning. And what I hope to get out of this podcast is really just

01:03 little insights around opportunities in agent harnesses, what you’re learning in real time, and other opportunities that exist, startup ideas, and stuff like that. So, John, can you commit that by the end of this episode I’m going to learn some of those things? >> I’m in. Let’s do it. Let’s make it happen. >> So, I want to start off by, like, back here you can see, what is an agent harness? Cuz I think a lot of people, you know, scroll X, they see agent harness, but they don’t really understand what it is. So, can you clearly explain what it is?

01:34 >> Absolutely. Let’s break it down to the two words, agent and harness, and I’ll use maybe Gamma’s journey to illustrate both. So, for everyone’s context, Gamma is a tool primarily for making presentations, also documents, websites, social graphics, that sort of thing. We’ve actually been around for several years, long before agents were ever a thing. We started the company in 2020, really took off in 2023, and early on we were just making these one-off LLM calls to do things like write a single slide or make the outline for a deck. And we very manually strung them together into these preset workflows to do things like

02:04 generate a presentation with 10 slides. The big revolution that’s happened in the last year or two is this agent revolution where the most powerful AI products now are not stringing together these individual LLM calls by hand, but they’re actually having one higher-level loop of an LLM run that will do some kind of work, inspect it, use tools, and then turn that into something over a whole series of executed calls. And so we actually went through this whole journey over the course of this year of rebuilding our product to be an agent. And building your product to be an agent

02:36 involves building a harness. So the harness is all the tools that agent uses to execute things in its environment, inspect its own work, and deliver something finished to the user, including soliciting from the user what their actual preferences are. So probably the most famous agent harness is something like Claude Code. Claude Code is a harness for programming. So Claude Code teaches the agent how to go through all the files on your system and see what’s there, teaches it how to like run tests, um and even prompts it with certain uh rules like go into plan mode

03:06 in certain cases, or run these tests after completing execution, or do this stuff on GitHub. In our case, Gamma is not a coding agent, it’s a creative agent. So it needs to do things like understand your art style and creative preferences, help you craft a story, and we care a lot about visual design. We’re doing things like generating images and laying them out, so we need to know how they look and make sure they look right. So our harness takes all of those things and turns them into tools that an agent can use. And I can talk through what that looks like. >> Yeah, I mean, please do. And I think uh

03:36 also I’m I’m curious if you agree or disagree with me. I’ve been saying agent harnesses are the new GPT wrappers or new wrappers, right? [laughter] Um is that is that just a good tweet or is it uh is there some truth to it? >> I I there’s truth to it, although I always uh disputed this concept of the GPT wrapper originally because it sort of implies that 90% of the work is the LLM, and then there’s just this thin layer on top, which is the wrapping back and forth. I think maybe what harnesses illustrate is that actually the wrapping is most of the work. Um

04:07 the you actually have all these LLMs, which are quite similar in their core capabilities, not to say they don’t have some variation, but generally all the magic of turning them into a product is building the harness around them. And even the model companies like OpenAI and Anthropic are putting so much of their work into harness development as well. So they are themselves doing wrapping of these things to achieve different capabilities. >> Could you walk me through what happens like in 30 seconds after someone types a prompt on Gamma? Like how many models does it touch like realistically, you know, cuz I think a lot of people are

04:37 are are not sure how how that how that actually works. >> Let me walk you through it. Let’s go through a sample prompt. So very typical use case for Gamma, someone will paste in a bunch of messy notes or a document or PDF and say help me turn this into a 20-slide presentation. So the moment you put that prompt in, um a whole bunch of things actually happen under the hood. The first thing is we actually spin up an entire computer in the cloud. This is what they call an agent sandbox. This is what the OpenAI ones broke out of famously in the last few months. So we actually spin up this entire sandbox where the agent gets all these files representing the user’s

05:08 prompt, the file they brought in, also other context about them. Um we then prompt the agent to go look through all that file and their prompt and run certain tasks. And for us actually one of the first tasks is usually to figure out what other information we need from the user. Um and so typically we will actually pop up a form and ask a few questions. Uh questions like who’s this actually for? You know, is this like an educational presentation for school? Is this for uh adults? Is this for your work thing? We will actually call out often to different connectors. So we’ve built integrations into tools like Notion or

05:38 Slack or Linear where we can actually pull in all this relevant context. Just already in that first maybe 10 seconds, we’ve now made calls using probably three or four different models, including expensive and cheap ones. We’ll talk about actually how we manage costs, as well. Just to sort of source information and ask a few questions. We’ve run it through UI around it with some generative UI to actually show you what those choices are. Then we’re going to actually write an outline for your deck. That’s usually several more LLM calls, and we will style a template for you. So we have a

06:10 library of templates with different visual styles. We’re going to use vision models to try to figure out which one’s a good fit for you. We’re going to do some cool search embedding things to sort of look at the space of thousands of templates and find the right one, and even customize it if you said, “Yeah, but I want it red or blue.” or whatever it is. Then we’re going to once again show that to you and let you edit it and inspect it. We haven’t even started making the deck yet, and already we’ve probably made something like 30 or 40 different LLM calls to maybe four or five different models doing different tasks. Now you’re

06:40 going to hit that generate button, and we are going to build up a presentation for you before your eyes. And there’s actually a lot of pieces involved in doing that. We have to write the words on the slide. We have to lay out all the things. In our case, we generate a lot of imagery as well, whether decorative or more substantive. We’re often making diagrams or infographics or charts. So actually all of those are now more calls that are going out to different models, often in parallel. There’s a huge performance consideration because if we did these one by one, it’s going to be quite slow and you’re going to basically leave before we ever finish. So we do a

07:12 lot of work to make it all pretty speedy. And then in the space of about a minute, you’re going to actually see that whole presentation streaming in front of you and generate. And now we’ve made probably close to 100 different model calls, and I would say probably 10 to 15 different AI models have been used somewhere along the way. And we’re still not done because you probably won’t stick with the first draft. In fact, we don’t stick with the first draft. After we generate that first presentation, we’re now going to have a AI vision model look at every single slide and look for problems that

07:42 I can find. Misaligned text, the person’s head got cut off in the image, there’s weird contrast, something’s not padded right. It’s actually amazing how many things can go wrong in something like a simple presentation that a human eye can detect right away. Or sometimes the human eye can’t detect because it takes a designer’s eye. So, not everyone can, but we can train models that can catch that. So, we go through this whole round of visual QA. We actually will do multiple rounds of fixing things based on what we find in that QA and then redo them. Um and then after all those rounds of fixing, we will finally serve it up to

08:13 you. And now at this point, we’ve probably used 20 different models making like >> [snorts] >> maybe 200 different calls along the way to deliver what looks to you like one finished output. >> So, what you’ve just described to me is like a workflow in a particular niche. In this case, it’s, you know, creative output. >> Totally. >> You started with presentations. You’ve now moved, you know, it’s essentially a creative suite. >> Yep. >> You know, where do you see opportunities to create agent harnesses outside, you know, if

08:45 you were starting a company tomorrow? Where do you see an opportunity to create a vertical agent harness? What’s a What’s an example of a niche that, you know, you see under explored? >> That’s an interesting one. I mean, you know, just leaning on my experience, I think where we see a lot of value of making our own distinct agent harness is when that problem looks very different from just plain old coding. In our case, what makes it different is that it’s very subjective. There’s a lot of judgment calls to make along the way where two different people will disagree about what good looks like, which means you want to have the human in the loop.

09:17 And also where the person using it actually really cares about the specifics of the output. And so, I would think about maybe those two traits as something that go into like, wow, these are a good case for making your own agent harness. So, just thinking aloud, some examples are like, I know a lot of people are making games with Cloud Code. And I’m really bullish on this idea of like the personal video game where you almost like five coded it yourself. Um coding agents are already really good at making video games, but I think they’re often not good at actually sourcing the

09:47 players’ opinions on where it should go and where it should evolve. And if you think about, you know, things like Roblox or Minecraft, there’s all these things where people are actually shaping the world for themselves. And usually it’s like non-technical people who are not coders themselves. I wonder if there’s a harness to be built there uh where in fact the whole harness runs in like a 3D world or something. >> Mhm. >> I think people should really think broadly and try to, you know, in general my philosophy of business is try to be different. Try to do something that looks nothing like what everyone else is making. And for us that’s been a challenge because we try to be

10:18 different, but then others, you know, copy us or do the same thing and we have to keep looking for more and more different. >> I think that’s, you know, it’s not even just for you guys. It’s anyone who’s building anything in software today. As soon as you catch lightning in a bottle, >> Totally. >> uh people are copying you, but I think the thing that people I think the reality is they’re copying where you were six, you know, three months ago, six months ago, right? So you you as the founder have to be like, okay, where do I what’s my strong point of view? What’s my thesis

10:50 on where the world is going? >> Yeah. >> And how do I essentially create like a a product studio where, um you know, every two weeks, you know, I’m building a feature zero to one, exploring a you know, uh a new piece of this puzzle. >> I think that’s totally right. And I think for anyone thinking about building their own agent harness, the really important question to ask is what’s my edge going to be? And there’s a couple different types of edge you can have. One type of edge is what you just said, which is you have a distinct vision of

11:21 the world and you are just moving faster towards it and it’s not the one that others are focused on. Um I think another edge is you have some kind of proprietary data. You’ve got some information that other people don’t. Maybe because you already have a successful platform or something. I think I’d totally different edge too is that it’s not just software because software is getting very crowded and it’s a scary market to be in. But, you know, there’s there’s hardware, there’s like physical services, there’s all sorts of real life businesses to be made that incorporate AI in interesting ways. >> How how do you, you know, physical services? Like, can you elaborate on on

11:54 what you mean? >> Yeah, I mean, to be clear, I’m a software guy, so I don’t know anything about this. But, if I were not a software guy, I’d be thinking a lot about how can I integrate AI into some kind of real life experience that people are having where there’s actual some kind of human exchange of value. Whether it’s like, you know, a person giving you a massage or a restaurant that’s serving you a meal or whatever it is. I think there’s so many times of like real businesses that are not quite reinvented with AI yet. Um, I don’t know that they all need their own harness, although

12:25 perhaps some of them do. Like, another category where I’m seeing a lot of harness going on is like legal tech, for example. I suspect medical tech is like a big one that has a lot of interesting like data lock-in and regulatory stuff going on. >> Yeah, I think legal was one of those where it was like, obviously this makes sense. >> Yeah, yeah. >> Um, accounting obviously makes sense. I think uh, the thing I would be thinking about is like would I be building a legal tech AI AI agent harness today? Probably not. >> Yeah.

12:55 >> Um, just because it feels so crowded. >> Totally. >> So, I like what, you know, when you’re talking about service businesses, even boring businesses in general, like agent harnesses for HVAC companies or, you know, um, if someone wants to build an agent harness startup, how um, in a niche, how do they approach uh, designing designing it in the sense of like you need the key insight, you need the niche. >> One really simple answer about what an agent harness is is an agent harness is

13:26 just a bunch of tools that an agent uses to complete a task. And so, I think the question to ask is what are the unique tools that either I already have or I can build that are not already out there. And ideally, I think as a founder you should always ask the question of what am I uniquely suited to do based on my life experience, where I am in the world, the people I know, that sort of thing. Um >> And how how do you think about AI smell? >> Yeah, totally. >> Uh you know, cuz I think a good a good agent harness is going to

13:57 kill that smell as much as possible, and I know you have a strong opinion on that. >> I do. It’s I’ve been using this term AI smell um all year. I’ve been talking about deodorizing, spraying it off, all that sort of thing. Maybe some people don’t just don’t know the term. AI smell is what we use to refer to this fact that when people make work with AI, you can often tell they used AI. I think it shows up most in writing, where when someone writes you an email made with Claude, you’re just like, you don’t normally use that word. That’s not what you sound like. Um and and I notice it a lot with AI images now, where I just like walk around in San

14:28 Francisco and every cafe, every restaurant has a poster where I can just smell like that’s Nano Banana, that’s GPT image. I do this stuff all day, so I can like spot them a mile away. And um I think more and more people are starting to recognize that AI smell as it diffuses just through the economy and normal people. So, for us at Gamma, it was actually a huge goal of this launch. We had kind of been a victim of our own success as one of the early AI tools, where when nobody used Gamma, if you used it, your stuff would look really cool and stand out. But once everybody started using it, everything started to

14:59 look like Gamma, and people didn’t want to be recognized as using AI. And so, kind of our whole goal of this launch we just pulled off was we need to rapidly expand our sort of uh creative landscape. Um The truth is there’s still going to be AI smells, but there can be a thousand AI smells instead of one monolithic one, and we can help people jump to different little pockets. And we needed This is another form of a harness, by the way. We needed to create our own automated system for constantly cycling through different creative looks. We actually use the analogy of fashion. Fashion has a smell, too.

15:30 You’re like, I know that H&M shirt you’re wearing. Like, I have the same one. Uh and you know, with fashion, there’s kind of this dynamic where people are cycling through different looks. You know, right now we’re into like really big wide jeans, but soon it’s going to be skinny jeans again. And so, we realized we had to actually ride that cycle and give people the tools to adapt to it. >> So, is there an eval for this looks generic? >> You know, uh eval’s such an interesting topic because I think eval’s work really well for things that are objective. >> Actually, break if you don’t mind, break down what an eval is.

16:02 >> Yeah, totally. So, um an eval just means a You could think of it almost like a software test. It’s a hard-coded question that you ask in a way of judging it. So, a simple example in our world of Gamma is we often want the agent to ask questions when you do something ambiguous. So, we’ll give it an eval like, um when the user says, “Make a deck about tigers,” the agent should ask something like, “Is this for kids or for adults?” Because there’s very different kinds of tiger presentations you could make about that. Um and so, we typically use evals to

16:34 test out new props or new models. And usually you score them. And you can put a bunch of evals together into a benchmark where you say, “Oh, it turns out model B scores 3% higher than model A.” Every time the labs come out with a new model, they’re like, “Oh, Opus 7 crushes like these 15 evals.” The problem for us in a very creative domain is that evals have always been hard for various subjective taste-oriented things, like, “Does this look cool?” And even more so, very personal ones, like, “Is this me? Does this sound like something I would say?” There’s no one objective eval for this.

17:05 And so, you know, we’ve tried to create some evals that are proxies for this, like, “Does it follow instructions? Does it For example, if you provide your brand guideline, does it stick to it?” That’s an objective question that we do test ourselves on. But we’ve never limited ourselves to just evals. And I think this is actually an interesting insight too of how you can compete in this world. Ultimately, what founders can bring to their domain is taste. They have their own sense of what looks good. They’re actually talking to their users and those users have one. And the less verifiable a domain you’re in, like say fashion, the

17:36 more I think there’s room for these unique perspectives. I think when it comes to clothing design, there should be 100 different harnesses. I think, you know, Nike should have a different one than like Adidas because they have a different perspective on taste. Also, for us, because we have a super diverse user base who are all different, we actually rely a lot more on AB testing. We will actually run two models or two prompts at very big scale and measure across that wide user base what’s actually performing better because that captures subjectivity. >> And what’s what’s the metric that matters most there? >> You know, we have a bunch of different

18:06 ones. To give you a few examples, we look at things like after you generate, how much time do you spend editing what you made? Do you end up sharing it or exporting it? Are there signals that show you’re investing time? We also have a freemium product, so people will use our product for free and then if they like it, they’ll upgrade to paying. That’s kind of my favorite signal cuz it’s hard to fake. Generally, people only pay if they’re really seeing value out of a tool. So, we use that one as like a north star. Sometimes we’ll even look at like downstream retention. Like are people still coming back to the product a week later or a month later? It’s

18:37 obviously slower to measure, but it’s also one of those hard to fake signals. >> The chart that’s going viral this week is I think an A16Z chart that says I think only 2% >> of people are spending on AI tools. What’s your opinion on that? >> Yeah, it doesn’t surprise me. I think in general subscriptions are always a bit of a niche thing towards people that are doing some kind of valuable work. I don’t think consumers like subscription. I as consumers don’t like subscribing for things, you know, I’m still always looking to jump on somebody’s Netflix

19:07 account or whatever it is. Um >> No, I will not share my password with you. >> Are you sure? What if I had to explain what an agent harness is. Uh but I think it’s different for work. I think for work it’s a totally different story where, you know, work subscriptions are what has driven essentially the entire SaaS boom of the last like 10 to 15 years in software industry. And those things will keep being successful. And I think that same chart says that when you actually zoom in to professionals, the numbers are rising like crazy in terms of who’s paying for AI. So, that’s certainly our main focus and I think most people’s main focus in AI is how to find those

19:38 those prosumers, the professional consumers, and how to break into B2B where there’s a totally different kind of spending going on. >> If someone wanted to build the Gamma for X, what would what would you pick? Like if someone listening to this is like, “Wow, like you know, I’m inspired by what Gamma is building. >> Yeah. >> Um but I I think that like that, you know, basically what you just described, but like for a different niche. Any come to mind? >> It’s such a personal thing because if, you know, if it was my thing, I’d be

20:09 building it already. I think you need to do the thing that I don’t know anything about, whether it’s like Pokémon cards or fashion or sneakers or whatever it is. Like find your domain where you know what’s going on and build it. >> And and you personally, is there like a harness that you wish existed that doesn’t exist that you’d use tomorrow if it if it came out? >> You know, um I’m really intrigued by this whole sort of like personal agent space. And I haven’t gotten to use all the latest ones, but what I feel uh nobody’s totally cracked is like work

20:39 collaboration. I think if I could build a harness for anything, it would be Slack. Um and I’m a little disappointed that Slack themselves have not built the harness for Slack. >> What would that look like in your mind? >> So, there’s just this absolute flood of communication for anybody whose company uses Slack or Microsoft Teams, of course. Um there’s so many messages going through. Um you’re constantly trying to stay on top of like, “Well, which ones do I actually have to do something about?” Um I’ve seen them nibbling around the edges of this of like you can sum summarize 30 Slack messages into one. But I want something that just like runs it all

21:10 truly for me and it learns me. It learns how I would reply before I reply. I want to be able to open any Slack thread someone’s pinged me on and have the first draft drafted for me. >> And it learns you not by answering like a quiz or onboarding it by actually like your actions. >> By observing me. >> Yeah, by observing. >> I have sent approximately a billion Slack messages. That should be all you need to know how I’m going to reply. >> Right. >> Even in emails doesn’t feel totally cracked. I’ve seen Gmail has the little things that are like, “Oh, you know, like yes, I’d love to go to the dentist on Thursday.” Which is not how I’d ever

21:40 reply in my life. But uh I think there is so much more to just like represent me in a professional setting authentically to me. Like maybe that’s what I’m really craving and it’s a little bit of what Gamma does is I want an agent that can learn my voice. >> Mhm. >> And of course that’s hard because it’s personal and this is why the agents haven’t done it. It’s really hard to make agents personal. >> So, you know, at Gamma and you know, you’re you’re building something on top of other people’s models. Mhm. Um what’s your advice for founders building

22:11 on other people’s models in general? >> I think there’s no shame in building on other people’s models. We, you know, we have done this from the very start often bouncing between Anthropic, OpenAI, Google models. And so much of the discourse has always been like, “No, you got to train your own stuff. You got to fine-tune it because you need the moat.” I have always felt that that’s um unrealistic for early-stage startups in particular. I mean, first of all, the money involved in training your own models is really hard. But also, the models out there keep on getting better. Uh I rarely would want to be competing against the

22:41 models that are out there. I think though it does change as you get further along in the journey because you start to amass real usage. And I mean, let’s be honest, just amassing real usage is the main battle people should be focusing on. By all means, use the models that are out there to get that usage in the first place. Once you do though, you can start to do really interesting stuff. We’ve actually had great success fine-tuning models to bring our costs down and in some cases even to start to do more personalization for people. We also have started to identify specialized tasks that, you know, the big models are bad at. Like I mentioned earlier, this whole process of

23:12 QAing a slide and telling if it’s good. We’ve actually not had great results using any of the models out there for it. So, we are actually doing some of our own research to build models for those specialized tasks. I think people should absolutely aspire to that, but by all means use the great models that are out there. >> what about thought, you know, current What are your What are your What are your thoughts on frontier models or cloud models versus open source local? >> Yeah. Um I think, you know, cloud models are underrated. Again, I think open source gets a lot of hype and excitement, but follow the money. It’s

23:43 all going to the closed source models because they’re really darn good. So much work has been done for you. And the great thing for all of us is there is intense competition in the world of closed models. I think in a world where, you know, like Anthropic could just purely run away with it, this would be actually a different story because we wouldn’t be able to choose between like great OpenAI models, great Gemini models, but I I feel like all these companies are just a few months behind each other and for us in the application layer, that’s the best thing ever. It keeps the heat on them to keep doing better and better. >> You’ve once said that

24:14 you you were surprised with how shallow your AI stack was. Is that still true today? >> You know, I would say that our AI stack is broad but shallow. What I mean by that is we actually use a ton of different models, but we often use them in fairly simple ways. Like we get so far with prompt engineering, with providing and maybe your listeners have heard the term context engineering increasingly of just like providing the right context to the models. I really view that as kind of the whole game here of just it comes back to harness

24:45 building. Can you get the right context from the user? Can you express the AI’s output to the user in the right way? And can you distill all that into things that make the LLM’s job really easy? When you make the LLM’s job really easy, you can use any off-the-shelf model and get good results. >> There’s a big debate prompts or fine-tuning. You have a a strong opinion there? >> Yeah, I think for us it’s 98% prompts is where we get the value out of. Cuz prompting is easy and it’s portable. So like and by the way, nobody’s writing prompts by hand anymore. Like Claude’s writing your prompt for you and so it’s

25:15 really easy. And so I think most problems can be solved with a prompt including breaking up a prompt into different pieces. And fine-tuning to me is more of an optimization on top when you want to do things like control costs or eke out that extra two or three percent of quality. >> When a new model drops, how quickly are you implementing it? Are you kind of like waiting and seeing or how how do you think about it? >> I know usually within the hour. If it’s if if it’s a model that I have reason to believe is going to be good. So you know, not every model’s interesting, but

25:47 I would say for us like any model from Anthropic or OpenAI is interesting generally if they’re dropping it. And I think I mentioned we have all these different jobs in our harness. Some of those jobs we use a like you know, Fable or Astra level model for. Some of those jobs we use a little Haiku Gemini Flash level model for. Some we use a little baby GPT Ludon model for. And so anytime any model comes in this class that can help us fill out that sort of Pareto frontier, I want to try it and see how it does. And generally because our use case is so subjective, yes we’ll run

26:17 some evals. But I just want to run it on a sample of users. And we’re very lucky to have a big user base where I can start a test in an hour and get data two hours later on how it’s >> doing. Um so one of the things I learned cuz I’m a co-founder of a design agency called LCA and we actually worked on a project together at Gamma. And one of the things I learned working with you guys was you guys run a lot of experiments. >> Yeah. >> I’m curious about um experiment you maybe one experiment that the result surprised you. >> Mhm. >> Um because you guys are just

26:48 it feels like you guys are running a lot a lot at a time. >> We run a huge number of experiments. Any given time we probably have 10 or 20 running. Um and we’re cycling through Many of the experiments last less than a day. >> So walk me through like the process of an experiment cuz I think it’s really really interesting. >> Well, I’ll walk you through one that I just started in the Uber ride over to talk to you. I just fired off eight different experiments because uh Claude Haiku 5.5 came out yesterday. We were really excited to get to use it because you know, I mentioned that we have a lot

27:19 of these tasks that don’t need that like super fable level model. They’re quick and cheap like picking a template from a library or sometimes like writing an outline based on something. And you know, we’ve been looking forward to new small models we could use to test this. So uh this morning I fired off eight different tests for it and right after we got off this podcast, I’m going to check the results and probably cancel four of them and see how they did. Um and we’re testing it for things like making micro edits on a slide or you know, like revising an outline, whatever it is. In terms of the kinds of things we learn from these tests, um

27:52 often the great thing that we learn is that we can use a model that is half the cost of another model and get the exact same results. But sometimes we actually get uh the opposite finding. We find like, “Wow, actually the big beefy expensive model is necessary to achieve a certain task.” Um sometimes we even discover things like good friction. Like we actually just ran an experiment a few days ago of a fast mode of our product. Um and fast mode just means skip through some of those steps that I told you about earlier. Like showing you an outline of

28:22 your presentation or asking you questions up front. And the hypothesis was of course if you remove friction in the process, then people will get through it faster and get to that sort of magical value moment more quickly. I was I was really rooting for this test. I wanted to see it win, but it did not win for us actually. Um it turns out those speed bumps we had along the way, like seeing your outline and reviewing it, you know, choosing uh the kind of high-level arc of your story, answering those questions led to much higher satisfaction ratings. Even though it was slower, the people that got through felt

28:52 more invested in their output and did more. And they actually converted to paying at a higher rate. Um But I think what’s also cool is a test is never just one answer or another. You can like dig a little bit deeper. So, we actually shipped a version of fast mode for brand new users on mobile because, you know, on mobile you don’t want to be typing a bunch of answers to a question. >> I just right before this I posted an X article about marketing engineers. >> Mhm. >> Um my thesis is that uh marketing’s changed. If you think about the Mad Men Mad Men era, it was like, you know, you

29:24 did one big commercial a year and in the ’50s and ’60s and ’70s. Then it became um in the ’90s and 2000s the digital marketer, maybe a few more uh experiments. Then we had um the growth hacker era. A few more experiments per week. Um I think in 2022 I would say AI-assisted marketer, uh more experiments. And you know, I think we’re now entering the age of the marketing engineer where you’re you’re you know, as a marketer you’re running dozens of experiments per day.

29:56 How are you able to do that? You’re using AI agents to go and do that for you. I’m curious your P POV on that. Do you think that marketing has changed and like where do you think it where do you see it going? >> It’s funny because before I worked at Game, I worked at a company called Optimizely. This is where my love of AB testing came from. We were AB testing as a service. And a lot of our vision was about personalization. This is idea that, you know, as it gets easier to change your website, we are going to create a hundred little micro variations of it and everyone will get served a slightly different version and that will

30:27 optimize conversions in all these interesting ways. Surprisingly to me, that vision I would say largely has not come true out on the internet. Um I would say websites are still mostly one size fits all. Uh so are mobile apps by and large, um, except for these like specialized areas which are like hyper personalized and succeeding. When I say specialized areas, I mean things like your TikTok feed, your Instagram reels. Like we know that companies are spending billions of dollars to eke out every last, you know, dollar from you in those surfaces.

30:58 >> heard that McDonald’s is testing personalized pricing. You hear about this? >> Wow, that sounds like a backlash waiting to happen. >> That’s why I saw it. >> Yeah, yeah. [laughter] Um, but I think maybe that McDonald’s example illustrates why this isn’t happening. First of all, I think there’s a real societal allergy to this kind of thing. In the same way that if you found out that we both bought the same airline ticket, but they charged me five bucks more cuz I was in San Francisco. >> Well, I think in general like even surge pricing like I feel like when I I had surge pricing last night uh, coming home from dinner and I was like I I was like uh, you know, like

31:29 yeah. It just it’s not a good feeling. >> Yeah, yeah. And, you know, I actually think in this age of hyper-efficient, you know, like performance marketing, actually what’s working is companies that really doggedly invest in their brand. This like Apple mindset of like just protect the brand at all costs. And so I think that trend is going to counter some of what you’re describing of everybody just going wild with their marketing agents. That said, I still buy the thesis that like the marketing engineer is rising. And I just love that term marketing engineer because it sounds like a contradiction, you know,

31:59 much like GTM engineer sounded like a contradiction. These are supposed to be different departments. But I have loved seeing how everybody’s skill sets are just blending. Uh, we’re seeing this with Gamma actually where people are more and more using Gamma not to just make one presentation here or there, but creating these really cool automations where they’re doing things like every time a new deal opportunity is created in Salesforce, let’s automatically create a pitch deck based on the call notes that have happened. >> How do you set that up? >> We’ve actually been building kind of connectors where you can automate more

32:30 of it. Um, so you can just like plug into Salesforce. But I would say there’s still a fair amount of plumbing going on. Like people are using maybe a tool N8N or Zapier or something to trigger it all together and wire it. Um that is a good example of a thing where it’s still the domain of specialists, I would say. There’s like these really smart people or agencies that know how to help a business like wire things up. I think there is still a business opportunity in taking these practices that are done by the top 1 or 2% of companies who are really savvy about engineering it and just turning it into

33:01 a super easy thing that everyone does. Another example of this, to go back to the personal agents, is like what’s happening now with tools like Muse and Instinct. You could do some amazing stuff with OpenClaw or Hermes if you could get your Linux box set up in the right place and avoid getting hacked. Personally, I skipped out on that whole trend. I was like, “I’m a busy dude. I don’t have time to set up my OpenClaw. I’m going to wait a few months for them to just make it easy for me.” And now I got Muse and Instinct blowing up my phone because they made it easy for me. >> Right. So, you don’t have a Mac Mini running uh >> I never got my Mac Mini. I’m ashamed to

33:31 say as an AI founder. >> [gasps] >> And you know, going back to marketing engineering, in your mind, what does a top 1% marketing engineer look like? >> Um well, one thing that comes to mind is they’re really a marketing engineer data scientist. So, they are like ultra data-driven and grounded in what’s performing. And this is something that’s gotten easier than ever. Like absolutely Clod is my data scientist. And so, like I would say now everybody on our team is constantly armed with like very fine-grained analysis of like failure

34:02 modes and opportunities. They’re looking at traffic on our site. They’re finding gaps. They’re you know, like I look at our you know, the person we have on our team who’s closest to a marketing engineer and he is just constantly thinking of new ways to do SEO, ad creative at just an unbelievable scale for one person. You know, they’re like they they’ve gone from being like individual creatives to people who are running this army of agents. Um and they are just in this high-level supervisory role. >> Yeah, I I it’s basically a department. >> It’s a department, yeah.

34:32 >> it’s like we’re we’re getting to the place that if you’re a top 1% marketing engineer, your surface area of what you’re working on isn’t just uh well, the surface area is huge. You’re looking at SEO, GEO, you’re looking at uh social content, you’re looking at paid ads, you’re looking at um pretty much like all creative brand, you know. >> Absolutely. >> It’s I think that’s what’s interesting to me because I think that the people that figure out how to become a top 1% marketing

35:04 engineer have an unfair advantage around getting their product seen. >> Well, and the other 98% are quickly becoming dinosaurs. If you can’t keep up with >> [laughter] >> the people in the market that are doing this. I’ve kind of had this idea that all of us over the last maybe like year or two, at least all of us who are using these tools kind of got this rapid series of promotions. We went from individual contributor to manager to director to like VP. We now all have these like layers and layers of agents under us in a pyramid. And

35:34 it turns out that job’s actually kind of hard. Like um uh there’s a reason that like few people do it because you are sort of constantly barraged by information. You’re having to make so many decisions. Uh you kind of are like overwhelmed and confused by all of it. You need tools to even make sense of it all. You also need people to help you make sense of it all. You kind of need your your assistant and you need your chief of staff and you need all of these things. And so all of us are having to rapidly upskill to essentially like how to manage a thousand people. And building

36:04 that skill is one of the most important things we can all do. Also, I think building tools to help the average person become a like VP of their own life is still under explored. Going back to my Slack idea. >> I also think uh you know, I I I experiments in general, it feels like building a startup today is or building a business in general is it’s like you have it’s more it feels like almost a little more science than art. >> Yeah. >> Now in the sense like your the ones who

36:34 are going to win are the ones that have the experimentation muscle and just trying a bunch of different things. That’s why like I love talking to you because it it feels like that experimentation muscle is I mean that’s your like your thing. Um what’s one thing at Optimizely that you learned that every AI founder should should know you know would be good for them to know. >> Mm that’s a great one. You know one thing that comes to mind

37:05 for me I think from years of like helping other companies do experimentation was that so much of it just comes to like how many app bats you have. Um uh it’s all just sort of a numbers game just like sales just like dating just like kind of anything else in life. You got to really try a lot of different stuff versus expecting that you’ll run one big you know test that magically unlocks your business. Um I like how you called it a muscle because I do think it is. First of all because it’s trainable you know. Yeah maybe some people are born with more muscle than others but if you put in the

37:36 work that muscle’s going to grow. >> I also think like your first experiment’s not going to work. >> Totally. >> Just like if you’re trying to get fit and you’re trying to build muscle >> Yeah. >> you have to just show up to the gym every day and you have to put in the work and then two months later then then you might see a six pack right but you can’t expect it on day one. I think uh that’s the hard part about being uh an experimenter is like there’s like a trough of sorrow that you have to go through. Um and I think what a lot of people end up

38:06 doing is they’re like you know what I’m just going to go with my gut. >> Absolutely. >> Um and >> By the way the same is true of things like posting on social media writing a blog sending cold email outbound. There’s a trough of sorrow in all of it and I think all success in life comes from swimming through some kind of rasa sara. >> [sighs and gasps] >> What’s What’s something that’s under hyped right now that you’re like, you know what, people are just not talking enough about this. I was talking with um

38:37 a friend of mine about voice AI. Um that felt like under hyped. >> Yeah, yeah. >> Um right now people are talking about personal agents a lot. Anything that comes to mind? >> Maybe it’s a lame answer, but I actually think AI image models are under hyped. It’s a lame answer because we’ve been talking about image models for years. Ever since like stable diffusion came out in 2022, people have been making like cool stuff. But I think um ironically, as they got good, people stopped talking about them. As they stopped messing up in so many ways and just became sort of like part of the

39:08 fabric, we sort of like forgot that we have this unbelievable technology that can make anything. And I think if you look at the way people are using like chat GPT images and a few other it’s kind of just the same three or four use cases over and over. You know, they’re they’re making like maybe the the menu for their small business. They’re making like a logo for their dad’s side project. There’s like a handful of things they’re doing. But these models are capable of so much. And we have a

39:39 really cool vantage point to this at Gamma. We didn’t talk as much about images, but we use probably like 10 different image models as well for different tasks. And the pace of innovation there has been so much. And just the sort of like unexplored capabilities. For us it’s things like making beautiful diagrams or infographics to go in a presentation. Making cool little textures that can be the background of a presentation. Making custom icons. Like all these things are possible now. And most people just don’t realize it’s there. And I think you can pay attention

40:10 to almost like any domain that’s creative and find ways that you could be using these things. And especially the average person just has no idea. You know, even one that I think is like probably many AI native people are doing, but most people don’t realize you can do is using AI image models for like clothing and like home decor. You know, like you you found a new couch you want to buy on Amazon, you can actually just chat GPT to say what would that look like in my house um and then say like go find me a few others and visualize those, too. Like so many of these things are just waiting to

40:40 be more deeply productized. >> Do you think there’s an opportunity to build vertical products there? Like just focused on you know, interior design, for example, um or do you think the the big model companies are just going to own that use case, so it’s not even worth trying? >> I think there is so much room for these vertical use cases and it’s all about understanding the specific needs of the people that do them. Like let’s just take this furniture one, for example. I actually just bought a couch recently, so I

41:11 >> Congratulations. >> Thank you. >> [laughter] >> It’s really comfy. And I went through this and it was so interesting um like trying to do this. And it turns out the AI image models are amazing at showing me what the couch will look like, but they’re not actually showing me exactly how it’s going to measure, right? And there’s a huge difference between, let’s say, a 68-in couch and a 58-in couch if your wall’s a certain size. So like could you actually show me the units in the display? That’s a great example of a harness, right? Um the models are actually capable of a lot of these things. One way they’re capable is you could imagine doing a thing where you

41:41 render the couch in my room, but then you use some good old-fashioned programming to draw a line across in pixels and measure how long it is and see if it actually came out of the right size and prompt the model to tweak it to get exactly right. I don’t see the labs doing that anytime soon. >> Or prompt, “Hey, like this couch isn’t the right couch for you.” >> Yeah, totally. >> hey, but here’s uh another couch uh from Restoration Hardware that I think would be perfect for these reasons. >> Totally. And dude, what I learned about couch shopping is there’s hundreds of websites that sell couches and you have

42:12 to click through a lot of little buttons on all of them. I ended up on Google Shopping. It was not a pleasant experience. Maybe somebody’s already built a great product for this and I missed it, but I haven’t seen it yet. >> I think there’s been examples. I think levels.io has like a >> Yeah, he has like an interior, I think it’s called. >> Yeah. Yeah. Yeah. >> But I think there’s still room. You know what I mean? Like I think uh especially as the models are getting better, the image models are getting better. Um the the concept of a harness is getting more and more mature. There’s just there’s more here. I think uh

42:46 I think it’s safe to say that the also the consumer experience today around couch shopping or furniture shopping isn’t optimized. >> Totally. >> Um and what’s cool about that specific use case is it’s high ticket. >> Totally. >> Um so uh and there’s margin, right? Like if if for example, like I don’t think it’s crazy to do a deal with Restoration Hardware where they’re like, “We’ll give you 5% commission if you recommend our product.” >> couches, yeah. Absolutely. I think the other version of this, when

43:16 you think about like what will survive competition with the model companies, um and we’re certainly facing this question constantly at Gamma. There is the pure consumer version, which to use the couch analogy is like help me buy a couch. Um but then there’s the like prosumer version, which is um hey, I am an interior designer. Maybe I’m a part-time interior designer, but I also have like two kids at home and that’s what I do, but I I you know, I I do it by the hour. Those people need a tool and they will seek out the best. And if it helps them do their job better, they

43:47 will happily pay a subscription for it. Then there’s also the true pros. There’s Restoration Hardware itself has people who work there. What tool are they using? I bet it’s not just ChatGPT to do all of this. I think there is actually an opportunity to build companies for those. And I do suspect that highest tier of professional, the the bigger companies, is where you can really build something durable. >> Yeah, and I I feel like it’s under explored. >> Yeah, it is. >> cuz the the narrative I think is there’s there’s not that much opportunity um creating vertical vertical SaaS or

44:17 vertical AI products. >> Yeah. >> Um which is unfortunate cuz I think that um there is. >> And it’s a big world out there, >> [snorts] >> you know. That’s maybe one other thing we’ve learned from the Gamma journey. Um our user base is very broad cuz if you just think about how many people on Earth make presentations, it’s a lot of different kinds of uh people. Um for us uh less than a quarter of our revenue is in the United States. Um it’s spread across many different job roles from like consultants to teachers to marketers to sales people to lawyers. I

44:48 didn’t know lawyers make presentations. Turns out they do. >> Mhm. >> And you know, the more you can tap into with little niches, you know, like in many ways a big company is just a thousand little niches piled on top of each other. So, if you can find something that cuts across, and you don’t have to start broad, you can start with just one, but then look for the adjacencies you can tackle. >> 100% and you actually don’t know where you’re going to where it’s going to lead, right? Like you didn’t realize like, “Hey, lawyers are using are creating presentations.” and then you end up who knows, you end up creating something specific for lawyers. And then

45:19 all of a sudden that becomes a huge part of your business. And that’s why I’m always like get started. Like put something out there. Like put something out there. You got to just find [clears throat] out. That goes back to experimentation. Experimentation doesn’t just mean run an AB test with like 10,000 users in each arm. It usually just means put up a landing page, get a product someone can try, which has never been easier. >> If you I know you’re not leaving Gamma, but like let’s say you were to leave Gamma tomorrow, what what’s something what what what’s a company you would start to build? >> Oh, that’s a good question, man. You

45:50 know, it’s I I just love circling the space of like enabling people’s creativity. I think there’s other kinds of like creativity that like intrigue me though. Like um as crowded as the space is, I’m so intrigued by just like vibe coding as I practice and just the way I had this experience maybe as somebody who knows a lot of other tech people and lives in San Francisco earlier this year where every time I would catch up with a friend I hadn’t seen in three or four months, I’d be like what are you up to and they’d be like check out my app I made.

46:21 >> [laughter] >> It sort of took the world by storm. I want to know maybe what’s the next vibe coding or what’s the next frontier of vibe coding. As people can make anything, um where will that take them? And so like within that I’m I’m certainly intrigued by video models finally getting good. I’m intrigued by voice models getting good. Like what’s the version of uh some kind of thing you could call and it would do something for you. Games are so interesting to me. I just think like the video game industry is ripe for some kind of disruption.

46:52 But I haven’t cracked it yet. I don’t know. >> I think on the video side it does feel like we’re hitting like the vibe coding moment for video. >> Yeah. >> Uh Opus 5.5 has is obviously >> some cool stuff. >> Makes some cool stuff. The yeah, so I think the question is in a world where anyone can create video, what are the types of videos they can create? And then I’m also interested it’s like as a founder I’m like, okay, in a world where anyone can create video, what are some startup ideas that I can create >> Totally. >> on top of that. And that goes back to agent harnesses. >> Yeah, absolutely.

47:23 >> Everything goes back to agent >> Everything goes back to the harness, baby. >> Um dude, thank you so much for coming on the podcast. Is there anything you want to leave people with? One thing? >> Uh I would just always encourage people who are thinking of starting a business to know that it is a long and sometimes lonely journey. And because of that, you really have to love the thing you’re working on. Um one of my kind of rules for myself before we started Gamma and it would stick around if I ever did another startup is it has to be something that I could dog food,

47:56 that I could be a user of and that I could believe in. And to be honest, that actually rules out a whole lot of boring businesses, but it was good for my soul. I just needed to have something that I wanted to spend hours every day in. And so, I always encourage people to dig deep and think about like what is your passion and the thing that um only you know, the thing that you want to spend eight hours a day on that when you look around your friends don’t want to spend eight hours a day on and zero in on that as your North Star. >> that’s the important point too around like the thing that you want to spend

48:26 eight hours a day on that other people don’t. >> Totally. >> you might be really into sports or something, but a lot of other people are into sports and that might be very competitive. I think like if you’re if you’re choosing a startup idea or choosing a business, I always say like um marry the niche, date the product. >> I love that. >> Like fall in love with who you’re trying to serve. >> Yeah, yeah. >> And because we live in an age where you you know, you’re going to be iterating on the product so many times and it’s going to look so different, but if you love the the niche,

48:57 um and that’s why I always encourage people to uh build media, build a following in that niche cuz then you can sell them a bunch of different products. >> an audience. I think that’s totally right. You know, Jeff Bezos has a very similar quote where he says, “Be stubborn on the vision, flexible on the details.” >> Mhm. Well, I’m glad I’m in good company with Jeff Bezos. >> are. >> [laughter] >> Thank you. I appreciate you and I >> Thanks for having me. This is great. >> It was awesome.