Great at coding isn’t always great at voice! Meet Voice Standard Bench.

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Great at coding isn’t always great at voice! Meet Voice Standard Bench. · Channel: AssemblyAI · 03:55 · Watch on YouTube

Topics: AI evaluation and benchmarks, Model inference and optimization, Multimodal models, Open source and software ecosystems

Summary

AssemblyAI introduces Voice Standard Bench, a benchmark designed to compare language models on voice-output tasks rather than coding, math, or reasoning. The video demonstrates comparing models in the LLM gateway and explains how performance, cost, and speed can vary by task and audio length. The benchmark covers use cases such as summarization, text correction, and dictation cleanup, and users are encouraged to test it with their own audio. The open-source benchmark and LLM gateway provide access to 50 models through AssemblyAI’s platform and APIs.

Key points

Models that perform well at coding are not necessarily best for voice outputs.
The AssemblyAI dashboard’s LLM gateway lets users compare and access 50 LLMs.
The demonstration compares Qwen 3.5 4B fast and Gemini 2.5 Flashlight on transcript summarization and task extraction.
The benchmark focuses on voice-output use cases, including speech-to-text and voice agents.
For each task, the benchmark measures performance, cost, and speed.
The best-performing, lowest-cost, and fastest models can differ depending on audio length.
The benchmark includes summarization, text correction, and dictation cleanup.
Users are encouraged to test their own audio because accents, vocabulary, and needs vary.
The benchmark and its GitHub repository are open source.
The LLM gateway offers direct access to models through AssemblyAI’s developer platform and APIs.

Chapters

Start	Chapter	Summary
00:00	Introducing the voice benchmark	The video explains why coding ability does not necessarily predict voice-output quality and introduces AssemblyAI’s benchmark.
00:32	Comparing models in the LLM gateway	A transcript is used to compare Qwen and Gemini on summarization and extracting tasks.
01:36	What the benchmark measures	The benchmark tracks performance, cost, and speed across voice tasks, with results that can change based on audio length.
02:45	Testing your audio and using the gateway	The speaker recommends testing with your own audio and describes the open-source benchmark and direct model access through the LLM gateway.

Quotes

models that are great for coding aren’t necessarily great for voice outputs. — 00:00

Every task, we actually measured the performance, the cost, as well as the speed. — 01:36

we recommend using your audio to test on our benchmark because our benchmark is open source. — 02:45

So we can’t wait to see what results you get with your own voice data on our LLM gateway. — 03:19

Concepts and entities

Voice Standard Benchmark, AssemblyAI, LLM gateway, Qwen3.5 4B, Gemini 2.5 Flashlight, open source

Full transcript (original language)

00:00 Hi, models that are great for coding aren’t necessarily great for voice outputs. Let me give you an example. Let’s say you just got done with your 1 hour meeting. Now, you want to take out some tasks and then you want to maybe change it to British spellings or maybe you want to just summarize. How do you figure out which model is best for that voice output? Today, we have an answer. For that, we are today introducing our voice standard benchmark. To get started, let’s go to our AssemblyAI dashboard and then LLM

00:32 gateway. And here, you can compare 50 LLMs and use our 50 LLMs directly. So, let’s say I select a transcript. Here, you could upload a transcript if needed. I have uploaded one already. Let’s pick this one. This is all about our Universal 3.6 Pro launch. And this transcript, I’m going to click continue and then I could choose on one side Qwen 3.5 4B fast and on the other hand, Gemini 2.5 Flashlight. And then, let’s say summarize the transcript. And you

01:04 notice right here, Qwen 3.5 4B is extremely fast, but Gemini 2.5 Flashlight is very precise. You could ask a question of your type. Let’s say, “Give me task for me from that audio clip.” And then, you see again that Qwen is extremely fast, [music] but the output maybe you want is better from Gemini. It gave you seven tasks. Now, let me tell you why our benchmark is important because think about it, typical benchmarks measure coding,

01:36 mathematics, or reasoning, but our benchmark is solely on voice outputs and something you really need out of your voice agents or speech-to-text and all other use cases. Every task, we actually measured the performance, the cost, as well as the speed. So, for example, if you have like, you know, information instruction, let’s go to one of this task. And let’s say the length of your audio was 2 minutes or 30 seconds. And in this case, the best performance comes out to be from GPT 5.6. Lowest cost from Nemo Tron and fastest,

02:12 again, what I showed you, Gwen 3.5 4B. But things change when you increase the length of audio to an hour, then bigger models perform way better. For example, GPT 6.1 sold perform way better. And the lowest cost was still Nemo Tron, but fastest speed came out of from Gemini 3.5 flash. And similarly, you can check the results for summarization, text correction, dictation cleanup, and all your use cases. But our transcriptions are not yours. You might have different accent, different English needs. Your vocabulary might be different. Your

02:45 needs might be different. So we recommend using your audio to test on our benchmark because our benchmark is open source. And you can analyze You can analyze directly on our LLM gateway or directly through our GitHub repo, which is open source as well. And most importantly, why LLM gateway? Think about 50 models that we give access to you on our portal directly because of which you can actually get results faster. Think about, let’s say you have an output. You would On a typical scenario, you will transfer that data to

03:19 your actual server, then route to a particular LLM. Why don’t you save that cost and direct to LLM on our endpoint through our LLM gateway directly so you save time and latency and get access to 50 powerful models that are best for voice outputs directly on AssemblyAI’s developer platform as well as through our APIs. So we can’t wait to see what results you get with your own voice data on our LLM gateway. Let us know what models are best for your use case. Thank you. >> [music]

03:55 >> Mhm.