The Self-Driving Eval Trick No AI Benchmark Beats

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: The Self-Driving Eval Trick No AI Benchmark Beats · Channel: LangChain · 00:33 · Watch on YouTube

Topics: AI evaluation and benchmarks, Multimodal models, Robotics and autonomous systems

Summary

The speaker says their approach to evaluations was shaped by working at a self-driving company. Although the team tracked metrics for its vision models, a large grid of video clips was the key gate for deploying a model in a car. They watched clips together, took notes, and discussed what the model got right or wrong. The speaker argues that looking directly at hundreds or even thousands of examples is the best evaluation.

Key points

The speaker’s perspective on evaluations comes from experience in self-driving.
At the company, the car was the product, and vision models helped it drive.
The team used metrics and other evaluations for its models.
A large grid of video clips served as the gate for putting a model on a car.
The video contained short clips of about seven seconds each.
The team watched clips together, took notes, and discussed model behavior.
The speaker says there is no better evaluation than reviewing hundreds or thousands of examples.

Chapters

Start	Chapter	Summary
00:00	Evals in self-driving	The speaker introduces how working at a self-driving company shaped their thinking about model evaluations.
00:33	Reviewing examples together	The team watched video examples, discussed what the model got right and wrong, and the speaker calls this the best kind of evaluation.

Quotes

a lot of my thesis on evals comes from my time working in self-driving. — 00:00

the gate to get a model on the car was this big grid. — 00:00

we sat and we watched like 30 minutes, an hour of video — 00:33

there’s just no better eval than looking at 100 examples or 1,000 examples. — 00:33

Concepts and entities

evals, self-driving, vision models, DQA

Full transcript (original language)

00:00 So a lot of my thesis on evals comes from my time working in self-driving. At a self-driving company, the product is the car that drives and like the thing that makes it drive is the vision models, right? Of course we had evals, of course there were metrics that we looked at and that was very important. But the gate to get a model on the car was this big grid. I was this this DQA video, this like hour and a half or two-hour video that had, you know, seven-second clips. We sat down as a team and we had a projector in the living room equivalent of the office and

00:33 we sat and we watched like 30 minutes, an hour of video and we took notes and talked about it and said, you know, oh, like it actually gets this right there and this wrong there and there’s just no better eval than looking at 100 examples or 1,000 examples.