This Is What Voice Coding Was Supposed to Be (Qwen Audio Agent)

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: This Is What Voice Coding Was Supposed to Be (Qwen Audio Agent) · Channel: Better Stack · 09:19 · Watch on YouTube

Topics: AI agents, Software engineering with AI, AI developer tools, Local and open-source models

Summary

Qwen Audio Agent is presented as a voice-coding runtime designed to keep a conversation active while a separate coding agent handles longer tasks. The voice agent can answer quick questions, provide task updates, and accept interruptions while the coding agent works in the background, with results returned to the conversation when ready. The video explains its two-part architecture and notes that the runtime is open source, but its best-sounding voice path depends on a paid Alibaba cloud API; a local option exists with limitations. The reviewer sees its parallel-work approach as promising, while flagging the Chinese-first setup and lack of published latency numbers.

Key points

Dictation replaces typing but does not remove the wait for coding work to finish.
The runtime supports full-duplex conversation, allowing users to interrupt and continue speaking while work proceeds.
A lightweight voice agent handles conversation and quick requests, while a separate coding agent handles files, tools, and longer tasks.
Coding work runs asynchronously over the Agent Client Protocol (ACP), and results return to the live conversation.
The voice agent can answer unrelated questions and provide status updates without stopping the coding task.
Interruption handling is implemented as a state machine that cancels or clears in-flight audio and transcripts.
The runtime is open source, but its default high-quality voice uses Alibaba’s paid cloud API.
A fully local mode is available via a Hugging Face speech-to-text pipeline with an MLX backend on Apple silicon, though setup and voice-profile options are limited.
The reviewer found no published latency measurements and describes the project as Chinese-first.
The video’s central takeaway is that useful voice coding depends on concurrency, not just better transcription.

Chapters

Start	Chapter	Summary
00:00	The problem with voice coding	The video introduces Qwen Audio Agent as a runtime that keeps the conversation going while a coding agent works in the background.
02:34	Trying parallel voice and coding tasks	The demonstration shows the voice agent answering simple questions and handling new requests while Claude Code refactors a repository.
04:39	How the two-agent setup works	The video explains how the voice layer and coding agent communicate over ACP, and how interruptions are managed.
06:45	Limitations and local options	The reviewer discusses the paid cloud voice path, the available local mode, the Chinese-first experience, and the lack of published latency figures.
08:49	Who might find it useful	The video recommends the approach for people who want to keep talking during long coding tasks and concludes that voice coding is a concurrency problem.

Quotes

Keeping the conversation alive while the actual work happens. — 00:00

Full duplex means something different. It means both sides can talk at the same time. — 01:02

So the voice layer never blocks on the slow thing. — 05:10

That’s a concurrency problem, not a speech problem. — 08:49

Concepts and entities

Qwen Audio Agent, Claude Code, ACP, full-duplex, Alibaba DashScope, MLX, OpenAI real-time API, Better Stack

Full transcript (original language)

00:00 Voice coding falls apart the moment the agent has to do actual work. Refactor this file, run the build, and then silence. The agent disappears and we’re left staring at the mic icon, waiting for it to come back. Quen audio agent works differently. Claude code can be refactoring the background while you keep talking to the voice agent. And when the task finishes, it can interrupt you and tell you it’s done. And here’s the weird part. Quen audio agent isn’t a model. It ships zero model weights. It’s a runtime built around one thing most

00:31 voice coding setups completely miss. Keeping the conversation alive while the actual work happens. Now let me set up why that matters because I think most people have the wrong idea about voice encoding. The usual setup is dictation. Whisper listens, transcribes, pastes text into your editor. That’s useful, but look at what actually it replaces. It replaces the whole typing part. You sit there,

01:02 you still wait. You’ve swapped the fingers for your mouth and changed really nothing. It does speed things up, though. Full duplex means something different. It means both sides can talk at the same time. You can cut in mid-sentence, and more importantly, the thing on the other end can keep the conversation alive while work happens somewhere else. The second half is the whole reason this exists. If you enjoy coding tools that speed up your workflow, be sure to subscribe. We have videos coming out all the time. All right, now let me show you why this is different. I’m going to make it talk for

01:33 way too long on purpose. Not going to lie, the setup was actually really frustrating. It was all in Chinese, so I had to go to the Alibaba cloud English site, set the desktop UI for frontend and backend worker with the Alibaba API key, and then add some anthropic API credits for the backend. I did have to translate some of that. Once it’s all synced, though, it does work pretty good. I can just speak without pressing anything. It’s always listening unless I turn off the mic. So, here we go. I’m going to give you a code repo that I

02:04 need you to work on. It starts explaining that. Okay. And then halfway through, I’m just going to go here. Stop. Too long. I get it. >> Understood. >> And it cuts off just like that. It it listened to this. Now it’s all gone. I can refresh from here. It doesn’t finish the sentence because I cut it off. There’s no extra second of audio fighting me after I start talking here. The runtime actually kills the old turn and moves on. I’ll paste in the path to the repo that we are going to work on here.

02:34 Now watch what happens with something simple. What’s 19 * 24? Don’t touch the repo. I got an immediate answer though because clawed code never needed to see that. The voice agent can handle the easy stuff itself, which is great because I don’t want every random question turning into a big task or racking up my API credits for anthropic. But now I’m going to give it something that actually takes a bit more time. The code repo that I gave it is just TypeScript practice problems. I could have chose something harder, but let’s just keep it straightforward for now.

03:05 I’m going to say here, refactor record problem TS. Make sure it works correctly. Now watch this. The task leaves the voice agent. Claude code picks it up and this work card appears. It’s the proof that Claude actually has the job now. And normally this is where a voice coding setup could fall apart. You’ve asked it to do something real. So now you have to wait. Except here I don’t have to. I can say also can you go make me a normal JS file with an async function to handle HTTP

03:37 requests? It answers and does both while they’re both running side by side. Claude is still working. Now I’m going to fire out a random question. Also, what is the weather like in Dubai? Okay, I can read it from here. Thanks. Another answer. Same conversation. And I can even ask about the task that is being worked on without having to wait for it. Hey, how’s the refactor going? Don’t wait for it to finish, just the

04:07 status. Now it can tell me Claude is editing, running tests, or still working. There are basically two things happening at once. I’m talking to one agent, another agent is doing the work, and neither one has to stop because the other one is busy, which means while Claude finishes, I can just keep going. The coding task finished, the result came back into the live conversation, and I didn’t have to check anything. I haven’t seen an open- source desktop runtime handle that this cleanly. So underneath all this, the

04:39 setup is actually pretty simple. On one side, we’ve got the voice agent. Okay, great. That handles conversation, interruptions, status, cancelling, a lot of the fast stuff. On the other side, clogged code is connected over ACP. That’s the side touching files, running tools, and doing the slow work. Now, mechanically, it’s really two lanes here. And once you see that, this design is a bit more obvious. There’s a lightweight front-end voice agent whose only job is to hold the conversation. It answers the easy questions immediately.

05:10 What’s the status? Cancel that. Never mind. Then there is your real coding agent on the back end and they talk over the agent client protocol, the ACP. Real work gets handed over as an async task and runs on its own. When it finishes, the result gets injected back into the live conversation. So the voice layer never blocks on the slow thing. And because it’s a protocol rather than an integration, the backend is swappable. Clawed Code, Codeex, Open Code, and a bunch of others. Now, one detail I want

05:41 to flag because it shows someone was actually paying attention to this interruption isn’t handled as an event here. It’s a state machine. Detect user speech, mark an interruption, admit an interpreted signal, actively surpass any audio and transcripts still in flight, then start the new term. Their own design note says the engineering quality of interruption determines the reputation. And they’re right. The thing that makes a voice assistant feel broken isn’t slow responses. It’s interrupting it and still hearing another second and a half of the former sentence. That kind

06:13 of gap is actually rather annoying and a big gap. This has a slight one sometimes. Now, where does all this sit next to what we already know? Well, the OpenAI real-time API is the layer underneath this. It’s the thing that does speech in speech out. This runs on top and is provider pluggable. Pycat and LiveKit agents are frameworks. They give you the parts and you assemble the pipeline. This is assembled and pointed at a specific job. And to whisper to LLM to TTS pipeline you wrote yourself is

06:45 the thing that a lot of us have, a lot of us build out. Now, let’s get real here for a second because there are two things here that the readme will not lead with. First, the runtime is open source. The voice is not. The code is Apache 2.0, but the default path, the best sounding one, roots through Alibaba’s Dash scope with a paid API key. You get some free credits, but it’s still paid. The optimized real-time voice is cloud API only. There is no self-hosted version of the good audio generation. So, open- source voice agent

07:16 is only partially true here. There is a real escape hatch, and I’ll give them credit for documenting it. a fully local mode using a hugging face speechtoext pipeline with an MLX backend on Apple silicone running on MPS. In full local mode, you need no cloud key at all. It’s a second Python install and the tune profile in the docs is Chinese only. Okay, so translate it. But the path exists and it’s pointed at max specifically. Second thing, and this one matters more, there are no published

07:46 latency numbers anywhere on any hardware I went looking. Okay, now a final quick points here. It’s Chinese first. The wake word is this, which translates to this. The main design article explaining the full duplex architecture is Chinese only. The tuned voice profile is Chinese. The model underneath claims a 100 plus languages for recognition and a few dozen for speech. So, I had to translate the page to understand the gist of it when I was signing up for all this. In version 2.0 know is already listed in development with the

08:18 architecture being rewritten and the stars are running ahead of the usage. 23 2400 stars here but around 4,000 mpm downloads a month and on the latest release 92 downloaded the Mac version build against 502 on Windows. Now, if you already work inside Cloud Code or Codeex, which I assume most of you all do, and you want it to stop being chained to the keyboard while long tasks run, this is the only thing I found doing the parallel task plus line conversation together, or at least the one that does it really well. Get the

08:49 DMG. It’s a signed universal build. No CUDA, no Python, nothing to compile. It takes about 10, maybe 15 minutes to get everything integrated. If you want a fully open, fully local voice stack, this isn’t it. But the MLX path is a real starting point and it’s more than most projects offer. And the idea worth taking away whatever happens to this particular repo is the thing that makes voice usable for real work isn’t better transcription. Transcription got solved. It’s whether the system can keep talking to you while it’s busy. That’s a concurrency problem, not a speech

09:19 problem. And almost nobody is treating it like one. I’m Josh from Better Stack. If you enjoy coding tools like this, be sure to subscribe. We’ll see you in another video. >> [music]