Voice Standard Bench - which LLM is best for transcripts?

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Voice Standard Bench - which LLM is best for transcripts? · Channel: AssemblyAI · 01:03 · Watch on YouTube

Topics: AI evaluation and benchmarks, AI models, Multimodal models, Open source and software ecosystems

Summary

The video introduces Voice Standard Bench, a benchmark designed to evaluate how well LLMs perform in voice-product tasks such as transcription cleanup and summarization. The creators tested models on transcripts of different lengths and compared quality, speed, and cost, using an independent LLM to score the answers. They highlight that GPT-6 Luna with reasoning turned off nearly matches the best open models at every length for about one cent per hour of audio. The benchmark is open source, and users can run it on their own audio and switch models through LLM Gateway.

Key points

Existing leaderboards do not specifically test LLMs for voice-product tasks.
Voice Standard Bench evaluates models on transcripts ranging from 30-second voicemails to hour-long meetings.
Tested tasks include information extraction, summarization, and general text cleanup.
An independent LLM scores each model’s answers.
The benchmark compares quality, speed, and cost side by side.
GPT-6 Luna with reasoning turned off nearly matches the best open models at every length.
The video says this model costs about one cent per hour of audio.
The benchmark is open source and can be run on users’ own audio.

Chapters

Start	Chapter	Summary
00:00	Why benchmark LLMs for voice tasks?	The video explains the need for a benchmark focused on voice products and introduces Voice Standard Bench.
00:33	Results and cost comparison	The presenters describe the scoring approach and highlight GPT-6 Luna’s performance and low cost.
01:03	Open-source benchmark	Viewers are invited to test the benchmark on their own audio and share what they find.

Quotes

That’s why we built Voice Standard Bench. — 00:00

we put quality, speed, and cost side by side. — 00:33

you might be overspending on your LLMs. — 00:33

The bench is open source, so you can run it on your own audio — 01:03

Concepts and entities

Voice Standard Benchmark, LLM gateway, GPT-6 Luna, AssemblyAI

Full transcript (original language)

00:00 Which LLM should you choose to run your transcripts on? We have leaderboards for coding, math, and general trivia, but none of them really test for what an LLM should do in a voice product, like a meeting transcriber or dictation app. That’s why we built Voice Standard Bench. We took every model on LLM Gateway and gave them the same transcripts, from a 30-second voicemail to an hour-long meeting, And we tested it on various different jobs like information extraction, summarization, and general text cleanup. What we then did

00:33 was had an independent LLM score every answer, and we put quality, speed, and cost side by side. My favorite finding was that GPT-6 Luna with reasoning turned off nearly matches the best open models at every length for about a cent per hour of audio. That means if you’re running one of these frontier models on every call, you might be overspending on your LLMs. But don’t just take our word for it.

01:03 The bench is open source, so you can run it on your own audio and even switch models on LLM Gateway by just changing one line. The link is down in the description. So good luck with your tests and let us know what you find.