Kimi K3.1 Invented the Benchmark Scores
Read the Portuguese translation →
Author: Matthew Berman · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Kimi K3.1 Invented the Benchmark Scores · Channel: Matthew Berman · 00:00 · Watch on YouTube
Topics: AI evaluation and benchmarks, AI content generation, AI models
Summary
Section titled “Summary”Matthew Berman asks Kimiko K3.1 Pro to create a graph showing its benchmark scores. The image presents broad categories including coding, reasoning, mathematics, and agentic tasks. When asked whether the numbers are accurate, the model says they are invented placeholders.
Key points
Section titled “Key points”- The prompt asked Kimiko K3.1 Pro to create an image of a graph showing its benchmarks.
- The generated graph included broad categories rather than specific benchmark names.
- The categories shown were coding, reasoning, mathematics, and agentic tasks.
- Berman questioned whether the numbers in the graph were accurate.
- The model said the numbers were invented placeholders.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | Benchmark graph prompt | Berman describes asking Kimiko K3.1 Pro to create a graph showing its benchmark scores. |
| 00:00 | Broad categories | The generated image groups benchmarks into coding, reasoning, mathematics, and agentic tasks. |
| 00:00 | Numbers are placeholders | When asked whether the numbers were accurate, the model says they were invented placeholders. |
Quotes
Section titled “Quotes”Those numbers don’t look accurate. — 00:00
Create an image of a graph showing off the benchmarks of Kimiko K3.1 Pro. — 00:00
Are those numbers actually accurate? — 00:00
No, they are invented placeholders. — 00:00
Concepts and entities
Section titled “Concepts and entities”Kimi K3.1 Pro, benchmarks, Matthew Berman
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 Those numbers don’t look accurate. Create an image of a graph showing off the benchmarks of Kimiko K3.1 Pro. This is what it created. And the first thing I notice is it just has these broad categories: coding, reasoning, mathematics, agentic tasks. [music] So, I asked it. I said, “Are those numbers actually accurate?” No, they are invented placeholders.