I Tested the Video AI That "Never Forgets" (It Scored 2/10)

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: I Tested the Video AI That “Never Forgets” (It Scored 2/10) · Channel: Prompt Engineer 48 · 08:14 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI content generation, AI infrastructure and hardware, Multimodal models

Summary

The video tests Delta Streamer, a research system designed to retain video details by recording changes and replaying original frames when needed. On a scripted five-minute desk video, a text-summary pipeline answered all 10 requested details correctly, while Delta Streamer answered only two. The reviewer traces the failures to the system often noticing the hand but missing the objects being moved, so the memory and later answers inherited those misses. The test has limitations, but the reviewer concludes that the system is currently better suited to research than product use and that a simple caption-and-summary workflow performed better in this test.

Key points

Delta Streamer records frame descriptions and changes, stores them in a layered memory, and can revisit original frames to answer questions.
The paper reports a score of 66.07 on OVE O bench, but the repository says these figures are not from a new run of the released code.
The test used a scripted five-minute top-down desk video featuring six objects and eight slow movements.
A summary pipeline based on 31 frames sampled every 10 seconds answered all 10 details correctly.
Delta Streamer scored two out of 10: zero out of three on Question A, one out of three on Question B, and one out of four on Question C.
The logs often recorded the hand appearing and disappearing but failed to record the object movement itself.
The reviewer notes limitations including a single artificial clip, one run per question, disabled tracking and audio, and thinking turned off.
Processing the full five-minute video took 41 minutes on an H100, while the summary pipeline took about 50 seconds.
The repository is research-oriented, has no releases, omits training code, and uses a non-commercial CC-BY-NC 4.0 license.

Chapters

Start	Chapter	Summary

Quotes

A plain text summary got 10 out of 10 details right. — 00:01

Delta Streamer, the model that is supposed to remember everything, got two — 00:01

10 out of 10 against two out of 10. — 05:39

If the memory is built from what the model noticed, the memory can only be as good as the noticing. — 06:42

On this test, it beat the Delta streamer by a wide margin, and it took about 50 seconds. — 07:43

Concepts and entities

Delta Streamer, Dense AI Quest Research, Nankai University, Chinese Academy of Sciences, Qwen3.6 27B, SAM 2.1, Whisper, Claude, RunPod, Hugging Face, H100, CC BY-NC 4.0

Full transcript (original language)

00:01 Same 5-minute video, same three questions. A plain text summary got 10 out of 10 details right. Delta Streamer, the model that is supposed to remember everything, got two, and its first answer was that nothing moved. Now, before you say I set it up wrong, I thought the same thing. So, I read the paper, I ran the real code on a rented H100, and I checked every single frame it looked at. All right, let’s go ahead and see what happened. So, what is Delta Streamer? This is the GitHub page. It is a research release from Nankai

00:31 University by Dense AI Quest Research and the Chinese Academy of Sciences, and the paper went up on arXiv on the 14th of October. The repo is very new. When I recorded this, it had 71 stars, zero forks, no releases, and two contributors. The readme says the training code is not included, and a full GPU regression has not been completed. The pitch is on the project page. Video keeps changing, memory should keep up. So, instead of describing each frame and throwing the frame away, it writes down what changed between frames. And when you ask a

01:01 question later, it can go back and look at the original frames again. On OVE O bench, the paper reports 66.07 overall. Gemini 3 Flash is at 64.02 on the same table, and humans are at 92.81. Mind you, those are the paper’s numbers. The repo says they are not from a new run of this release. Here’s how it works in four steps. One, it watches the video at one frame per second, and for every new frame, it writes a description plus a delta, which is just what changed since the last frame. Two, those go into a memory with three

01:32 layers. Working observations, episode summaries, and pinned facts, and every one of them links back to the original frame. Three, it does not see your question while it watches. When the question finally arrives, it makes a plan. What evidence do I need to answer this? Four, if the text memory does not have the detail, it goes back and replays the actual frames. That is the part that is supposed to beat a text summary. Under the hood, it needs Qwen 3.6 27B plus SAM 2.1 for tracking and Whisper for audio. But, the demo you can deploy turns

02:04 tracking and audio off, and I’ll come back to that. So, I wanted to test the one thing it promises. Remembering what happened earlier, I planned to film 5 minutes on my phone, but I wanted to know the exact answer to every question. So, I generated the clip with a script instead. It is a top-down desk with six objects: a red mug, a blue notebook, a yellow sticky note, a green plant, a black phone, and orange headphones. A hand moves one object at a time. Eight moves in 5 minutes with at least 25 seconds of nothing happening between moves, and there is a clock burned into

02:34 the corner. I wrote down the answer key before I ran anything. First, I tried the hosted Hugging Face demo. It only streams the first 12 seconds of a video, and the free GPU quota ran out after one answer. My laptop has an 8GB RTX 4060, and the 27B model needs far more than that, so I rented an H100 on RunPod. It took me three tries to get the model server up. First, a clash in the CUDA library path, then a missing build tool called Ninja that made the model crash on its first real question. The readme does not warn

03:05 you about either of those. The normal way to do this is a summary pipeline. Take a frame every 10 seconds, write a caption for each one, boil the captions down into a summary, and answer questions from that text only. I did exactly that with 31 frames. Claude captioned every frame without knowing the questions. And a second pass answered from the summary alone. Mind you, a strong captioner makes this baseline strong. So, this is a fair fight against a real workflow, not a weak one. Question A asked at 1 minute

03:35 40, “Which objects have been moved so far in order?” And when the right answer is three, the yellow sticky note at 35 seconds, the red mug at 60, and the black phone at 95. The summary pipeline said yellow rectangle first, then the red circle, then the black rectangle, each one within 10 seconds of the real time, three out of three. Delta Streamer said, and I’m reading it off the screen, “No objects have been moved.” And this is the part I checked. Look at the Delta log. At 35 seconds, it notices a pink circle, that is the hand.

04:06 At 38 seconds, it notices the hand is gone. And for the 3 seconds in between, it writes that all the shapes remain in identical positions. The sticky note had just crossed the table. Question B, asked at 3:20, “Where was the blue notebook before it was moved? Where is it now? And when did it move?” The answer key says, “Center of the desk, now left of center, moved at about 2:50.” The Summary Pipeline said, “Center before, upper left middle now, and moved between 2:50 and 3:02, three out of three.” Delta Streamer said the

04:36 notebook was at the center left and did not move at any point in the video. Its own log shows the hand arriving at 1:70 and leaving at 1:73, and nothing about the notebook in between, one out of three. And the one it got right is just where the notebook sits at the end. Question C, asked at the end, 5 minutes in, “How many times was the red mug moved? Is it back where it started? Which object was moved last? And how many times was that one moved?” The key says, “Twice, yes, the black phone, and twice.” The Summary Pipeline got all four. Red circle moved twice, back at the left.

05:08 Black rectangle last, moved twice. Four out of four. Delta Streamer said the mug was moved zero times, and that no object was moved last because no objects were moved at all. It did say the mug is at its starting position, which is true, but it said that because it believed the mug never moved, one out of four. Here is the scoreboard. Question A, the Summary got three of three. Delta Streamer got zero. Question B, three of three against one. Question C, four of four against one. 10 out of 10 against

05:39 two out of 10. And it is slow. At one frame per second, watching the first 100 seconds took 14 minutes on the H100. 200 seconds took 28, and a full 5 minutes took 41. Before we call it, here is what I cannot claim. This is one clip made of flat shapes and one run per question. The deployable demo runs with tracking and audio switched off, and the readme says those flags change the system that was evaluated. The model ran with thinking turned off, which is the readme’s own setting, and my moves are slow 3-second slides. So, every 1-second step is a

06:10 small change. To be fair, the example in the paper’s own figure is the color of a bus that passed. A detail that gets left out recovered by replaying the frames. That is a different test from mine. Mine asks what moved and when. Theirs asks what something looked like. Here’s what its memory looked like at the end of question C, 300 working observations, nine episode summaries, and one pinned fact, which is my question. Eight of the nine summaries mention a pink circle appearing or disappearing. Only one mentions an object moving, the green

06:42 circle. So, I’m not saying the paper’s numbers are wrong. I’m saying that in the delta log, the hand shows up at all eight moves, and an object moving shows up once, the green plant at 3:26. And the final answer ignored even that one. So, the memory, the plan, and the answer inherited the miss. That is the honest lesson of the design. If the memory is built from what the model noticed, the memory can only be as good as the noticing. And a replay step cannot help if nothing in memory says there is something to go back and check.

07:13 It took about 4 and 1/2 hours of H100 time to find that out, roughly $12 at the listed rate, and most of those hours were me fixing the setup. The summary pipeline took about 50 seconds. So, who should use it? If you do research on streaming video questions, the repo is worth reading, and the delta idea is clean. If you want to build a product, the license is CC-BY-NC 4.0, which is non-commercial. There are no releases, and the training code is not included. If you just want to know what happened in recording, caption it every 10

07:43 seconds and summarize it. On this test, it beat the Delta streamer by a wide margin, and it took about 50 seconds. My next test is a real thing. 5 minutes filmed on a phone, real objects, real hands, and faster moves since mine were slow slides. If you want to see that, let me know in the comments. If you’re looking for cheap and reliable GPUs, I use RunPod, and the link is in the description. So, in this video, we have seen what Delta streamer is, how it works, and where it missed in my test. If you want to try it yourself, the links are in the description. I’ll see you in another one. Check out the next

08:14 video.