Image Decision Models for RPA: Forms, Scans and Screenshots

Author: Sam Witteveen · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Image Decision Models for RPA: Forms, Scans and Screenshots · Channel: Sam Witteveen · 16:19 · Watch on YouTube

Topics: Task and workflow automation, AI evaluation and benchmarks, Multimodal models

Summary

The video demonstrates a form decision inspector that uses two open image decision models to answer questions about forms from PDFs or images, without first running OCR. The examples include checking whether fields are filled in, identifying form types, and detecting signatures or missing information. The models can disagree, and their accuracy varies by question format, so the presenter recommends testing them and using confidence thresholds to route uncertain results to a person. The demo also shows how to use conditional logic for follow-up messages and suggests fine-tuning as a possible next step.

Key points

RPA commonly automates fixed, repetitive steps, while decisions about what an image shows have often been left to people.
The inspector passes raw form images to two open image decision models and asks questions about their contents without using OCR.
Example checks include whether a form is signed, whether fields such as date of birth or address are filled in, and what kind of form it is.
The models return probabilities, including an unknown option, which can help avoid relying on uncertain predictions.
A confidence threshold can send low-confidence results to a human for review.
The models can perform differently depending on the question: in the demo, changing a signature check from true/false to a choice question improved the result.
Questions and model choices should be tested against the target use case rather than assuming one model works best for every task.
Conditional logic can generate follow-up messages, such as asking someone to sign a form, without using an LLM to write the message.
The presenter identifies fine-tuning with a LoRA adapter as a possible next step when suitable data is available.

Chapters

Start	Chapter	Summary
00:00	RPA steps and image-based decisions	The presenter explains how RPA handles routine steps while decisions about forms and images often require a person.
03:10	Why test image decision models for forms	The video introduces the challenge of deep-learning systems that require use-case-specific tuning and describes the form-inspection demo.
06:19	How the image inspector works	The presenter explains that the models receive raw images, answer typed questions, return probabilities, and can route uncertain results to a human.
07:50	Testing the models on a tax form	The demo checks a tax form and shows that the models can disagree, with performance varying between true-or-false and choice questions.
12:01	Question changes and a second form	The presenter adds questions and tests a form with a missing date of birth, highlighting the need to calibrate questions and models.
14:07	Confidence thresholds and next steps	The video wraps up the demo with confidence-based human review, conditional logic, and fine-tuning as a possible next step.

Quotes

I’m just passing in a raw image to the model — 06:50

It’s basically either sure that it’s something or it’s going to answer unknown — 07:20

They tend to be better at different kinds of questions. — 10:27

Don’t just go and think, “Oh, yeah, this model is the best model, therefore it’s gonna work for all my tasks, really well in here.” — 14:07

Concepts and entities

RPA, ImaJev, ImaJev 4B, Jev Omni, ImageBench, UiPath, OCR, fine-tuning, LoRa adapter

Full transcript (original language)

00:00 Okay, if you draw any business process as a flowchart, you get two main shapes. The rectangles are the steps. They’re things like copy this, click that, file this. And the diamonds are the decisions. Does this look right? Is this an invoice? Is this a payment? For years, RPA tools have been automating the rectangles. That’s robotic process automation, software bots that click through screens and copy data between systems the way that a person would, usually following fixed rules.

00:32 But the diamonds still often go to a person because a rule can’t look at a page and tell you whether it’s got a signature or not Now, people have had a lot of success at using deep learning to do this, but generally the challenge there has been that people have to use custom models for each particular company and for each particular business use case But it’s turning out that this is another place that a very specific kind of Jev model can really have a huge impact So what I’m gonna show you in this video is that I built an inspector

01:02 that basically looks at forms, whether they’re PDFs or images, and then runs them through two small open models that make decisions straight from images. these are literally OpenJev models And both the models are really high up on the ImaJev benchmark. In fact, the one that’s at the top here was built by one person in 15 days for just $1,200 of rented GPUs So I know I’ve made quite a number of videos about the Jev decision model concepts. And I’ve made videos of showing you how you can use it for agents and how you

01:35 can use it for text decision-making. In this video, I wanna focus purely on images and purely on the ImaJev decision models. Remember, this is something at the time that I’m recording this, that even the Jev model itself can’t handle images So in this case, we’ve really only got open models to try these things out all right, so what’s the problem that this is actually solving? If you’ve automated anything in business, you’ve hit some kind of version of this. You’ve got some kind of bot that can fill in a web form, can click and submit.

02:08 but often you don’t know if it actually worked until someone checks the actual screenshot. You’ve also got examples of where a customer uploads a photo for a refund claim. Does that photo actually show what they’re claiming in it? Well, someone has to look at it If you’re looking at invoices or contracts or forms you’ve often actually gotta go and look at that to see was it actually signed And every one of these is one of those diamonds in that flowchart. It’s a yes or no call that a person or a deep learning model will actually

02:39 make after looking at something. Now automating all of these kinds of things has been a huge industry. It’s got its own name, RPA, robotic process automation. if you look at some of the biggest companies in the space, you see ones like UiPath. And so I first came across UiPath in 2018. And at the time, what they were doing looked really impressive. They were certainly raising lots of money. And sure enough, by 2021, they had gone public with a valuation around $30, $40 billion.

03:10 Since then, though, it’s kinda collapsed. If you look at their stock price has dropped from the high 70s right down to sort of 13, $14 a share now. that market cap nowadays is under six and a half billion dollars. So the biggest question here has gotta be what went actually wrong here? and the thing that I found from talking to people who actually worked with UiPath was that while the promise of full automation was really awesome, the models never actually worked often like they were promised And to be fair,

03:41 I don’t think it was necessarily that UiPath was making a really bad product. It’s just that the type of deep learning that people were using for these kind of models really required a lot of fine-tuning for the specific use case. They just didn’t have, the general universal classifiers to be able to do a lot of these tasks, with a high level of accuracy so as I started to see people make these ImaJev models, I was really curious to see, is this something that could be applied to RPA?

04:12 So for the demo here, I’ve basically picked forms because pretty much every office has piles of forms that they’re either scanning in Or they’ve got electronic versions of as PDFs, et cetera. and what I was curious to test out was could we make a demo where you drop a PDF of a form on there, or even do it in batch mode Could you drop a PDF of a form and then have it run very quickly to generate answers back about that form? so for things like did the person fill out their date of birth properly?

04:45 Is the form actually signed? does it have an address for the person on it? Is it a tax form? Is it an invoice? all of these things are sort of stock standard RPA tasks, and they were all things that I was curious to see. can we take a model that hasn’t been fine-tuned for this specific use case and see how it actually handles this? all right, so this is ImageBench, a new benchmark for systems that make decisions from images. At the top there, you’ve got ImaJev 4B, And you can see that that one’s

05:15 definitely scoring really well here. Below that we’ve got Jev Omni, below that we’ve got NeoHorse and a number of other models in here. So interestingly, the person behind ImaJev actually wrote a Reddit post all about not only how they made it, but why they made it. so they’re a process consultant who worked on refunds, returns, and customer support. And his point was that in those process maps, almost every decision node ended up being made by a human because just the ambiguity is really hard to deal

05:46 with with code alone So I was curious to try out his model and some of the other models for my particular demo here. So I decided to actually make a demo with the first two top models here and see how it actually works out. Now, I know I’ve made a bunch of videos about Jev already, but those were mostly focused on text. the ImaJevs open up a whole set of new opportunities because a lot of the decisions in business aren’t sitting in text. They’re in screenshots, they’re in photos, they’re in scans, they’re in PDF files, and that’s what I really wanna dig into in this video So the whole

06:19 idea of these models is that they’re basically the diamond that gets the evidence, a screenshot, a photo, a page It can really just go through and answer a lot of different yes/no questions based on that image as the context. so in a normal Jev model, we basically pass in text as the state, and then we pass in our typed questions. Here we’re passing in the image as part of that state, but we’re still also passing in our type questions just like before now it’s important that I point out here, we’re not doing any kind of OCR here, right?

06:50 It’s not like I’m OCRing the form first and then passing that in. I’m just passing in a raw image to the model so the model itself is deciding what’s actually represented on that image. and then using it to basically answer questions like, is there a handwritten signature? Is a particular field empty? Is something filled out with a date there or not? And the cool thing here is that it gives us back a probability for every option. and ImaJev has also been trained to give you back the probability of unknown

07:20 So the good thing about having that in there is It just helps the model so that hopefully the model is not guessing. It’s basically either sure that it’s something or it’s going to answer unknown Now, with the responses that we get back, we still got the whole sort of same smart if statement that we had with the text models. It’s just now the smart if statements are things that you can actually see. and one of the cool things about having those if statements is that you can actually set a confidence score that if it’s returning below

07:50 that, then you can basically say, “Right, I don’t trust the model. I’m going to send it to either another model to either confirm the prediction or I’m going to actually send it to a human to look at . Okay, so if we jump into the demo here, this is basically the form decision inspector. So I’ll show you, just quickly what it can do. we can basically drag in a PDF, and just showing you what this is. This is basically a US tax form. And the text is pretty tiny in there, but you can see at least,

08:21 that, okay, it’s got a name, it’s got a bunch of details in there. And in this case, you can see that the signature is actually missing in here. The date, the person signing it is there, the date of birth is there, but signature in this case is missing So now if we come over to the actual questions, you can see we’ve got a bunch of questions in here. We’ve got, things like, is the date next to the the signature filled in, right? The postal address is filled in, date of birth is filled in. And then we’ve got, fields for missing, we’ve got fields for what

08:52 kind of form is this, in here. and then we’ve got some other things in there about things like, how legible is the handwriting, on this page? So I’m gonna run this. I can also change things like the DPI, so basically the resolution of the images, and what’s going into the actual model. I’m running two models here. I’m running the ImaJev 4B and the Jev Omni in here. Now I’m gonna show you, some of the issues you run into. So this goes through… It basically runs, and you can see, sure enough,

09:23 we can see that, this model has done a pretty good job, right? It’s basically said, “Hey, is it signed? No, it’s false. It’s not signed,” in this case. Now, up here we can basically set how confident does the model have to be before we actually accept the result or send it back to a human, there. Because it was basically lower than the 90% that I had set there, it’s gonna say that it should go to a human, ’cause I had that set up there. more importantly, this one is wrong, So let me come back to that in a second.

09:54 dated. They’re both in agreement. They’re both getting, it right there. Address, that’s looks right. we can see that this one, one said that the date of birth is there, one said that it’s not, in this case And we can see that this one, now when we ask, what is missing, the signature, is missing here, right? Yet up here, when we asked the signature is there true or false, it said that it was there. And this kind of highlights the problem that you are going to face with the different models at the moment, is that they tend to be better

10:27 at different kinds of questions. So you can see here, this model, when we ask it, what’s missing, it’s saying that nothing is missing, right? So that’s a choice question. That’s a classification question. The actual sort of true or false question in this case, it gets it right? We can see that the response, that it got back is going to be like that, and we can see the image if we want to there. in this case, it gets it wrong. Now this is where you have to be careful about what model you use, and not necessarily that one model is great and one model sucks.

11:00 They tend to be better at different things. So I’ll show you just quickly here with this exact same form. If we basically turn signature to be a choice, right? And I run it now You can see that this one is now saying, okay, that it’s typed name only. There’s no handwritten signature. And also this one now is basically saying typed name only, no signature. So it shows that especially in this model here, it’s probably better at the

11:31 classifications than it is at the true or false in relation to this task, right? I’m not gonna say that it’s better for every single task at this, but in the relation to this kind of task, I need to be calibrating for that, early on And we can see that both of them got it right for a tax form. both of them said that it was, either clear or readable, in there. In some ways, probably clear and readable are almost the, the same kind of thing, going through this. the other thing we can do too is we can actually have a follow-up email.

12:01 So this is not something that’s being sent to an LLM to write the email. This is literally just conditional logic. If the signature is not there, then we basically say, “Thanks for sending your…” and we fill in, tax form. “Before we can process it, could you please sign the form in the signature field?” so that’s not using a normal LLM in any way. But it’s important to understand that there’s no OCR going on here. This is literally just passing in an image, and then running it

12:33 through, for this kind of thing. And the cool thing here is that all of these, we can basically test, different, questions in here. So If I want to edit something, is missing could be the date of birth, I can just come in here and you can see that Each of these is basically just a string field that goes in there. and now I’ve got a new class, that it can predict as well. I’ll run it again. Obviously the date of birth was here for this one And you can see for this model, it wasn’t phased by me adding that at all. For this model, it was.

13:04 We’ve now got a little bit of things where it’s saying nothing missing to, okay, possibly the date of birth is missing So you do want to test these and work out, okay, one, what are the best kinds of questions for what you’re doing? And then also which model is the best one for those types of questions Okay, so if I give it a new form now, this is a form where the date of birth is actually missing. The signature is there, that you can see in there. So let’s try and run this one and see what we get And you see they’re running very quickly.

13:34 So you can see now we’re getting handwritten signature is there. both of, the models concur that’s the key thing there. If we go through and we get to date of birth, we can see now both of them are saying that date of birth is missing. This one’s less sure about it than this one in here And then again here when we ask what’s missing, this one can clearly see now that the date of birth is missing, and this model just doesn’t seem to be doing great at this type of question. So I’d probably think about, okay, am I gonna rephrase this for this model?

14:07 am I gonna only go for the results with this model here? But the key point that I want to show you here is that you need to be aware of this, right? Don’t just go and think, “Oh, yeah, this model is the best model, therefore it’s gonna work for all my tasks, really well in here.” So this is just a really simple sort of thing. You can see again, we can basically have it do a follow-up email, which is conditional logic. here it’s basically saying fill in your date of birth. It’s, worked that out really nicely And we can set this up to actually just use a sort of confidence score to decide

14:40 if we throw it back to a human or not. And that’s something that we could set. Now, with a little bit more work, we could actually set that confidence score to be different for, each of the questions that we’ve got in here. we can add in questions, quite easily, right? You can build something where you could put in a choice question, you could put in a scoring question, you could put in, a noul question in there So hopefully you can see here that this is something that is pretty easy to do with these kinds of models, right? You’re not having to train up a lot of models or do anything like that.

15:14 Now, that said, if people are interested, maybe I will make a video about actually fine-tuning some of the OpenJev models, ’cause that’s the next stage, that you take it to. If you’re starting to get, some data where you’ve actually got some decent data, you could actually take some of these models, whether it’s the image one or whether it’s one of the text ones, and actually fine-tune it with a little LoRa adapter in there to basically make it work for what it is that you’re trying to do. So anyway, hopefully that shows basically how these, Jev image models can be really

15:47 powerful for things like, RPA, for things like, very quick image classification, and dealing with large amounts of images in here We can run a whole bunch of different questions through it, and basically customize the questions and the model for our specific use case in here. so anyway, let me know in the comments if you’ve got any questions, about this hopefully again, this shows you a real world example of how this kind of thing can be used so that you’ve got a sense of what you can actually do with this. And as always, if you found the video useful, click like and subscribe, and

16:19 I will talk to you in the next video. Bye for now