Give your voice AI agent real abilities (tool calling)

Author: Google Cloud Tech · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Give your voice AI agent real abilities (tool calling) · Channel: Google Cloud Tech · 09:23 · Watch on YouTube

Topics: AI agents, AI developer tools

Summary

The video explains how tool calling gives a Gemini Live agent the ability to take actions during a conversation without interrupting its flow. It outlines the declare, decide, execute, return loop and compares implementing tools with the raw Google GenAI SDK and ADK. It then presents two rules for smooth live interactions: tools should return immediately, and policy checks should happen in a callback before execution. The examples use music controls and quiet-hours rules to show how the pattern works.

Key points

A tool is a described action that the model can request, while the application’s code executes it.
The tool-calling loop consists of declare, decide, execute, and return.
Tool descriptions help the model choose the right action; unclear descriptions can lead to incorrect tool choices.
With the raw SDK, the developer handles the tool-call loop; ADK can handle it using described Python functions.
Adding an ADK capability can be as simple as adding a function and placing it in the tools list.
The backend runs tools and can send events to the browser to update the app’s UI.
Live tools should return immediately so the agent’s voice does not pause while waiting.
ADK’s before_tool_callback can observe calls, block them by returning a result, or rewrite their arguments.
Policy checks, such as enforcing quiet hours, can prevent a tool from running while allowing the agent to respond.

Chapters

Start	Chapter	Summary
00:00	Live tool calling and the action loop	The video introduces mid-conversation tool use and explains how the model requests actions that application code executes through a declare, decide, execute, and return loop.
03:09	Implementing tools with the SDK and ADK	The video compares manually managing tool calls with the raw SDK to defining Python functions and letting ADK handle the loop.
05:13	Keeping live conversations responsive	The first rule is to make tools return immediately, while the second is to use a callback to observe, block, or rewrite tool calls before execution.
08:22	Key takeaways and what comes next	The video recaps the tool loop, implementation options, and live-agent rules, then previews controlling a browser by voice.

Quotes

The model cannot run anything. It asks. The code acts. — 02:05

The model chooses tools by reading the descriptions. — 02:36

In a voice conversation, silence does not feel like loading. It feels broken. — 05:46

Keep tools instant. Put policy in the callback. — 08:22

Concepts and entities

Gemini Live, tool calling, Google GenAI, ADK, before_tool_callback

Full transcript (original language)

00:00 ANNIE: Hi, I’m Annie. Today you will learn how to give a Gemini Live agent abilities, tool calling, but not the request response version you have seen a hundred times. This is the live version. The agent acts mid-conversation and the conversation does not break. Here is what that looks like. Hey, Mira, can you play something dreampop? MIRA: That sounds perfect. [MUSIC PLAYING] ANNIE: Skip this one please.

00:30 MIRA: Sure. Let’s find something else. Skipping now. ANNIE: Hey, Mira, can you play something loud? MIRA: A little louder. OK, let’s try this. See if this changes the feeling in the room. Do you like where it’s going? Oh, I’m sorry. It looks like we can’t be quite so loud right now. Quiet hours. Maybe something a bit gentler, just to sort of float with? I can try that. ANNIE: We will learn this in three parts.

01:01 Part one, the tool calling loop. How abilities actually work under the hood. Part two, the same ability written two ways– the raw SDK and the ADK way. Part three, the two rules that decide whether a live agent feels live or broken. By the end of this episode, you will have everything you need to know to build your own multi-modal agent. Let’s start with the loop. On its own, the model can produce one thing–

01:32 words. It cannot start music. It cannot skip a track. It cannot pause a player. So you give it a menu of abilities. Think of this as actions your app can take. For a radio DJ, the actions might be play a track, skip, pause. Each action is a function in your code. It has a name, it has a description, and it tells the model when to use it. Now, mid-conversation, a loop runs.

02:05 Four steps– declare, decide, execute, return. Declare, you send a menu of tools with a session config. Decide, the user says “play something dreampop.” Instead of only answering the words, the model emit a tool call called play_playlist, mood dreampop. That is the heart of tool calling. The model cannot run anything. It asks. The code acts.

02:36 Execute, your code runs the function. Return, your code sends the result back. Then the model keeps talking like nothing happened. While the code is onscreen, notice the description line. That sentence matters. The model chooses tools by reading the descriptions. So if your model calls the wrong tool, do not start by blaming the model. Start by fixing the description. So part one takeaway, an ability is a described action

03:09 plus this loop. Declare, decide, execute, return. The model decides, your code acts. Now let’s look at the same ability two ways. First, the raw SDK way. With Google GenAI, the tool menu is a dictionary. You declare the function, you catch the tool call, you dispatch it, you send the response back. The four arrows are all yours. Now the ADK way.

03:39 The same ability is just a Python function with a docstring inside a tool list with our four functions. The names are on screen. That is the whole thing. ADK reads the schema from the function signature. It uses the docstring as the description. It catches the call. It runs the function. It returns the result. In the raw SDK, we write the loop ourselves. In ADK, the framework handles it.

04:12 That means adding a new ability is simple. Add one function. Let’s do it. Onscreen we add set_sleep_timer. It takes minutes. The docstring says, “stop the music after a delay.” Then we drop it into the tools list. Can you play something dreampop? MIRA: Hm. Dreampop. [MUSIC PLAYING] ANNIE: Skip this one, please.

04:42 MIRA: Sure Let’s find something else. Skipping now. [DIFFERENT MUSIC] ANNIE: Can you stop the music in 30 minutes. MIRA: OK. 30 minutes and it fades away. ANNIE: One more connection to the last episode. The model does not control the UI directly. It asks for a tool. Your backend runs it. Then the backend sends an event to the browser. That event tells the app what changed. Start the music, skip the track, update the screen. Same live architecture.

05:13 So part two takeaway– adding a capability is just adding one described function. But in a live conversation, the hard part is not adding the tool. It is keeping the conversation smooth after the tool is called. So next, two rules for making the live tool calling feel smooth. Rule number one, tools are synchronous. Her voice waits for you. In a live session, when the agent calls a tool, her voice stops, and it stays stopped until your code returns.

05:46 Our music tools return almost instantly, so you never hear the pause. But if a tool is slow, you get silence. And in a voice conversation, silence does not feel like loading. It feels broken. So the rule is, acknowledge. Do not wait. Fire the command. Return OK immediately. Let the real work happen somewhere else. Every tool for a live agent has to pass one test. Does it return instantly?

06:16 If yes, the conversation stays alive. If no, the agent feels stuck. And if the work is generally slow, that needs a different pattern. That is where the next episode will go deeper. Rule number two, there is a checkpoint between the model’s decision and your execution. In ADK that checkpoint is before_tool_callback. It sees every two calls before the two rounds. You can do three things there.

06:47 Observe, block, rewrite. Observe means you log every tool call in one place. Block means you return your own result. The tool never runs. Rewrite means you adjust the arguments. Then you let the call continue. Let me show you what block means. Imagine your app has a quiet hours rule. After midnight, the agent should not start loud music. So if the model tries to call “play playlist,”

07:17 the callback catches it first. Instead of running the tool, we return our own result. It is quiet hours, suggest something gentler. The playlist tool never runs and the agent can still respond in character. Hey, Mira, can you play something loud? MIRA: Hm, a little louder? OK, let’s try this. See if this changes the feeling in the room. Do you like where it’s going? Oh, I’m sorry. It looks like we can’t be quite so loud right now.

07:49 Quiet hours. Maybe something a bit gentler just to sort of float with? I can try that. ANNIE: So in the next episode, the agent controls a real browser. It clicks. It types. It waits for pages to load. So this checkpoint becomes much more important. It helps us handle slow actions without freezing the conversation. And it gives us one place to inspect the action before it runs– before it clicks, before it types, before it touches the real world.

08:22 So part three takeaway. Keep tools instant. Put policy in the callback. Here is what you learned. First the loop. Declare, decide, execute, return. The model decides. Your code acts. Second, two ways to write tools. Raw SDK gives you full control over the loop. ADK turns each tool into a described Python function. Third, the two rules for live agents.

08:52 Keep every tool instant and put your policy in the callback. That is how you give a live voice agent abilities without breaking the conversation. That’s it. Don’t forget to leave your thoughts in the comments. As I mentioned before, in the next episode, we will take this one step further. You will learn how to control a real browser by voice. The hard part is not just calling tools, it is keeping the conversation fast while the browser action runs in the background.

09:23 All right, I will see you in the next episode. Bye. [MUSIC PLAYING]