Build a real-time voice AI agent with Google ADK and Gemini Live API
Read the Portuguese translation →
Author: Google Cloud Tech · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Build a real-time voice AI agent with Google ADK and Gemini Live API · Channel: Google Cloud Tech · 09:24 · Watch on YouTube
Topics: AI agents, AI models, Multimodal models
Summary
Section titled “Summary”Annie Wang demonstrates how to build a real-time voice agent with Google ADK and Gemini Live, using a radio DJ that can converse, take music requests, and be interrupted. She explains why live voice works better with a persistent, two-way audio connection than with a sequential speech-to-text, model, and text-to-speech pipeline. The video introduces ADK’s agent, runner, and session, then shows how a live request queue and event stream coordinate audio in both directions. It also emphasizes keeping the session in memory to avoid delays and distinguishes continuous audio input with send_realtime from complete inputs with send_content.
Key points
Section titled “Key points”- Live voice uses a persistent, two-way connection rather than a sequence of speech-to-text, model, and text-to-speech steps.
- The browser connects to a backend over a WebSocket, which carries audio in both directions.
- An ADK agent is configured with a model, instructions, and tools; tools can be added as Python functions.
- The runner manages the live call and returns events that the application can respond to.
- Keeping the session in memory avoids network round trips that can delay audio.
- A live request queue decouples microphone input from events returning to the browser.
- Use send_realtime for continuous streams such as microphone audio, and send_content for complete inputs such as text or a photo.
- Send microphone audio even during silence so the model can detect when the user has finished speaking.
- The application can play voice chunks, show captions, handle tool requests, and stop audio when an interruption occurs.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | Demo and overview | Annie introduces a conversational radio DJ and outlines the video’s architecture, ADK building blocks, and live audio loop. |
| 01:32 | Architecture and ADK components | The video explains the persistent two-way connection and introduces the agent, runner, and in-memory session. |
| 04:13 | The live audio loop | A queue decouples audio flows, while send_realtime handles continuous audio and send_content handles complete inputs. |
| 07:22 | Tracing a request and key takeaways | The video traces a music request through the system, explains interruption handling and the importance of in-memory sessions, and recaps the implementation. |
Quotes
Section titled “Quotes”Don’t think of it as a walkie-talkie, but a phone call. — 01:32
live voice is one open line with strings, not five steps in a row. — 02:37
Always send, even silence. — 05:46
If this is continuously happening, use send_realtime. — 06:17
one queue, two directions, neither waits. — 06:48
Concepts and entities
Section titled “Concepts and entities”Google ADK, Gemini Live API, WebSocket, send_realtime, send_content, runner.run_live, Annie Wang, Google Cloud Tech
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 ANNIE WANG: Hi. I’m Annie. Today, we’re building a live voice agent with Google ADK, one you can actually talk to, one you can interrupt mid-sentence, and one you can ask to do things. Here’s the agent we’re building. This is a radio DJ that can hold a real conversation and take requests. Hey, Mira. MIRA: Hey, welcome. What’s on your mind tonight? ANNIE WANG: Can you play something dream pop? MIRA: Dream pop? Of course. Just a moment.
00:30 Let me find something for you. ANNIE WANG: By the end of this video, you will understand how to build it with ADK. We will explain it in three parts. Part 1, the architecture, why real-time voice needs a different design. Part 2, the ADK building blocks. Part 3, the live loop, how audio flows both directions at once without blocking. That part is where live voice agent either feel natural
01:02 or fall apart. So before we get into ADK, let’s quickly reset the architecture. In the last episode, we talked about why live voice is different. The common way to build voice is a pipeline, speech to text, model, text to speech. Each step waits for the last one, and that works for chat. But in voice, even two seconds of silence feels broken. Live voice needs a different shape.
01:32 Don’t think of it as a walkie-talkie, but a phone call. Both sides needs connected. Audio flows both ways, and you can interrupt naturally. That is a shape Gemini Live gives us, one persistent, two-way connection, audio in, audio out, streaming the whole time. So here’s the full diagram. On the left, the browser. That gives us the mic, the speaker, and the music player. In the middle, a small backend.
02:04 The browser talks to it over a WebSocket that WebSocket stays open. Audio moves both ways. Inside the backend, we have the ADK pieces, a queue, a runner, an agent, and a session. On the right, Gemini Live. This diagram shows the full flow, browser to backend, backend to Gemini Live, and events flowing back. So part 1 takeaway, live voice is one open line with strings,
02:37 not five steps in a row. Now let’s look at the ADK pieces inside the backend. First, the agent. The mental model is simple. An agent is three things, a model, a personality, and abilities. In ADK, you define an agent in one place. You give it a model, you give it instructions, and you give it tools. That is the whole agent definition. If you want a new ability, just add a Python function
03:10 to the tool list. That is the pattern. The agent is defined by configuration. The abilities are just functions, and we will go deep on those tools in the next episode. Next, the runner. An agent definition is not an agent running. The runner runs her. It owns a live call from open to close, and everything that happens comes back through the runner as events. Hold that word. Events are the key to the live loop.
03:42 Then we have the session. The session holds a conversation state. For live voice, we keep it in memory, and that way, the audio path does not wait on a network call. So the setup is describe the agent, hand it to the runner, keep the session in memory, and tell the call what we want. Answering audio uses voice, gives us captions, done. So part 2 takeaway, describe the agent.
04:13 Hand it to a runner. Keep the session in memory. So now the hard part. How does the audio move both ways at once without colliding? Here is a problem. The browser sends microphone audio every few milliseconds. Gemini sends voice back just as fast. Those two flows are happening at the same time, and if one side stalls, it cannot block another. That is a whole challenge. So picture a sushi conveyor belt.
04:44 The shelf keeps putting plates on one end. You keep taking plates off the other end. Neither waits, and that is the point. The browser is sending audio chunks. The model is consuming them. In ADK, that belt is a live request queue. It decouples the two directions. Your server becomes two parallel jobs, upstream, browser to queue, downstream, events back to the browser,
05:15 completely decoupled. That is the belt. Now, what goes on the belt? There are two ways to send input. First, send_realtime. You can use this for continuous streams, like microphone audio. Think of it like holding down the Talk button. There is no clean message boundary. The audio just flows, and Gemini’s voice detection decides when you have stopped talking.
05:46 Here’s the part everyone gets wrong easily. Always send, even silence. The model needs the quiet to hear where your sentence ends. And second, send_content. You can use this for discrete, complete things, a text message, a photo, something where the user has clearly hit Send. Think of it like hitting Send in the Chat app. This is done. Respond now.
06:17 So rule of thumb, if this is continuously happening, use send_realtime. If this is complete, use send_content. For our mic loop, every chunk from the WebSocket goes straight onto the queue with send_realtime. That is sending. Now, the other direction. One call starts a pipeline, runner.run_live. You point it at the queue, and now the full duplex session
06:48 is live. What comes back is a stream of events. A voice chunk? Play it. A caption? Show it. A tool request? Start a music. An interrupted event? The user talked over her, stop the audio instantly. That is why events matter. The runner turns the live call into a stream of things your app can react to. So part 3 takeaway, one queue, two directions, neither waits.
07:22 That is what makes the agent feel live. Now let’s trace one sentence through the machine. You say, play something dream pop. Your mic becomes audio chunks. Those chunks write the WebSocket to the backend. The upstream job puts them on the live request queue with send_realtime. The runner feeds them to Gemini Live. Gemini’s voice detection hears when you finish,
07:52 then events come back down. Her voice goes through your speaker. The caption goes to the screen. The play command goes to the music player. Then you talk over her. An interrupted event arrives. Her audio stops. One sentence, this is the entire machine, one implementation detail that really matters. Keep the session in memory for live voice. The hosted session service writes an event for each audio chunk.
08:22 That means a network round trip every few milliseconds, and her voice came four seconds behind. In memory, the call feels instant. So here is what you learn. First, the architecture. Live voice is one open line with streams in both directions, not a five-step pipeline. Second, the ADK building blocks, describing an agent, hand it to a runner. Keep the session in memory.
08:54 Third, the live loop. Use one queue. Send continuous audio with send_realtime. Send complete input with send_content, and react to events as they flow back. That is a live voice agent with ADK. That’s it. Don’t forget to leave your thoughts in the comments. In the next episode, we give this agent abilities to calling, not just talking, actually controlling the app.
09:24 You speak. The multimodal agent decides what action to take. Your code runs the action, and the conversation keeps going. All right, I will see you in the next episode. Bye. [MUSIC PLAYING]