onPanda with Qwen: Click a Token, Change the Model's Answer

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: onPanda with Qwen: Click a Token, Change the Model’s Answer · Channel: Fahd Mirza · 06:20 · Watch on YouTube

Topics: AI models, Local and open-source models, Open source and software ecosystems, AI safety

Summary

The video introduces onPanda, an open-source tool from Step One for annotating LLM outputs token by token. The presenter installs it locally with a Qwen model served through VLM, then shows how token colors indicate confidence and how alternative candidates can be selected. Choosing a different token changes the continuation and the model’s answer. The video also points viewers to a paper and a hosted version of the tool.

Key points

onPanda is presented as an open-source tool Step One uses internally to annotate LLM data.
Users can click a token, choose an alternative from the model’s candidates, and have the model continue from that choice.
Token colors indicate confidence: green means the model was confident, while red means it was unsure.
The presenter runs a Qwen model locally using VLM and configures onPanda to connect to the local server.
The demo prompts the model to write a two-line poem about the ocean.
Hovering over a token reveals other candidates the model considered.
Changing a token restarts the continuation and can change the model’s answer.
The presenter mentions a paper about the tool and a hosted version that can also be used on mobile.

Chapters

Start	Chapter	Summary
00:01	Introducing onPanda	The presenter introduces onPanda and explains its token-level approach to correcting model answers.
01:07	Local setup and installation	The presenter describes the local Qwen and VLM setup, creates a configuration preset, and launches onPanda.
02:38	Running a prompt and reading token confidence	The presenter sends an ocean-poem prompt and explains the token-by-token output and its confidence colors.
04:15	Correcting tokens and changing the continuation	The presenter explores alternative token candidates and demonstrates how selecting one changes the model’s continuation.
05:16	Paper and hosted version	The presenter points to a paper and demonstrates changing tokens in the hosted version, including on mobile.

Quotes

You can also see the probability of every token. — 00:35

Green means the model was confident and red means it was unsure. — 03:11

the model then continues from my choice. — 04:46

It even works on mobile as you can see. — 05:16

Concepts and entities

onPanda, Qwen, vLLM, Step One, correção em nível de token, probabilidade de token

Full transcript (original language)

00:01 There is so much to cover in AI. When it’s a slow week in model releases, you get to see tools running from everywhere. This week, one of the best tools I have seen so far and useful too is from step one which they have opensourced after 2 years of hard work and really hats off to them for that. I’m showing you in this video on Panda which is open sourced by step one the tool they use internally to annotate LLM data. So here is the use case. Normally

00:35 when a model gives you a bad answer, you rewrite the whole thing. Here you find the first wrong token, click a better one from the model’s top candidates and the model continues from your fix. You can also see the probability of every token. So it doubles as a way to look inside the model. It handles images, audio, video, and agent tools calls to. It’s MIT licensed and it’s already running at scale in production at step one. Let’s get it installed. And as you

01:07 can see on my terminal, I’m using Ubuntu with one GPU card. I already have picked up this Quen model in 4 billion just to showcase this demo. And in UV environment, I’m serving it with VLM. If you don’t know what VLM is, how to get it installed, just check my uh channel for that. Let me show you the VRAM consumption on my 48 GB VRAMm GPU. If you’re looking to rent a GPU on very good price, you can find the link to mass compute in video description.

01:38 Recently, I also have started this free weekly AI newsletter to recap all these tools and models which I’m unable to cover on the channel in addition to what I cover on the channel. So please subscribe to it. Now first thing we need to do in order to get this thing installed is to create this preset JSON file which primarily is telling the on Panda tool when we will install it to point to our local VLM server with the model and of course you can replace it

02:08 with your own if you like. Now that’s done. As you can see one thing you must have is node and npm. This is my version. Again, it’s a oneliner command to install. And then you can simply go ahead and run on panda on port 8080 with the config I just created. And you can see that now on panda is running on my local host at port 8080. Let me open that. And this is on panda running on my own

02:38 machine. The model behind it is 3.4 billion served locally with vlm. if you’re using any other model because normally at the start it just goes with its own default model but I’m going to just use my own. So just make sure that in control parameters towards the bottom you select this and then you can send uh set some of the hyperparameters. That’s done. Now let’s uh try to check it out. I’m going to give a user prompt here that write a two-line poem about the

03:11 ocean. Now let’s click on send and check out the response and just okay there you go. So you see on the left hand side is the raw text the model generated token by token the color under each token is its probability. Green means the model was confident and red means it was unsure. The right side is the rendered markdown. The line at the top shows the model name and the number of token. So you see it is still thinking going going and let’s

03:44 wait for it. So this is how it thinks. Now what this means actually is that it is trying to think and then it is trying to give us the correct answer. Let it finish and I will explain the output. I think it will make more sense then. So in the context of this tool, this is what it means. This whole result and that is a key idea of on panda. Normally you only see the finished text. Here you see every token the model choose with a

04:15 color showing its confidence. As I said earlier, green means it was sure, red means it was hesitating. And look at the structure words mostly green. Now look at the creative parts like the poem options more red because that is where the model was selecting between many possibilities. Those red spots are where I can uh step in and for example you you see these are the red spots and these are the probabilities. So if I hover over this token see um what else the

04:46 model considered and pick a different one. For example, I’ll just say let’s go with stick and it should just go with that probability. You see it changed and the whole output has started running again. The whole thinking pattern will change and the model then continues from my choice. So that is how you annotate the output and the model learns. Right? So I hope that you got the idea. In order to further expand on this, let me show you this diagram which is again

05:16 telling you the same thing which is a token level correction interface of on panda. They have also released this very good paper which I will drop the link in video’s description. And if you don’t want to get it installed locally, if you just want to play around with it, they also have a hosted version I believe on um from here. So just try it online. It even works on mobile as you can see. There you go. See? And if I just go on the word, you just change it to let’s say

05:49 something like breakdown. It will just change the whole thinking process and the answer will be done. For example, I just say okay, just go with confused. It has changed it. So, and then these are the green ones. You can also change them if you like. There you go. You see? So this is the whole idea behind this tool. That’s it. Let me know what do you think about this tool. I’m going to drop the link to it in video description. Again, if you want to help out the channel, please become a member

06:20 and uh please subscribe to this newsletter. Thank you for all the support.