Perplexity’s pplx-decider-v1.2: Is It Worth Using? We Tested It Live

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Perplexity’s pplx-decider-v1.2: Is It Worth Using? We Tested It Live · Channel: Fahd Mirza · 08:14 · Watch on YouTube

Topics: AI evaluation and benchmarks, AI models, Multimodal models

Summary

The video tests Perplexity’s pplx-decider-v1.2 on customer support triage, image understanding, document reading, sentiment, and language recognition. The model performs well on the initial support-routing example and accurately reads memory specifications from a GPU data sheet, but makes serious mistakes on seatbelt images and gives a questionable sentiment score. The presenter notes that the model card reports improvements over V1.1, especially on language and knowledge tasks, while tool-use results are close. Overall, he says he would not use the model in production because better text and vision options are available.

Key points

The opening customer message test returns Support at 99.8%, blocked at 98.4%, and a frustration score of 1.28 out of 3.
The presenter describes V1.2 as a 27-billion-parameter decision model on a Gwen 3.8 27B backbone that answers multiple questions about a request in one pass.
The model supports images up to 1.6 megapixels and is estimated to require around 28–29 GB of VRAM.
On two seatbelt image tests, the model incorrectly says the belt is worn correctly, including when asked to apply Indian road-safety law.
A GPU data sheet test is a strong result: the model correctly selects 24 GB and 48 GB memory options at 99.7%.
For a customer message with negative wording and an idiom, the model scores satisfaction at 11.2%; the presenter says the tone is mixed and advises checking sentiment-analysis results.
The model identifies a German paragraph as German, but the presenter is unsure whether it understood the paragraph’s meaning.
The presenter concludes that the model is not suitable for his production use, citing better alternatives for both text and vision.

Chapters

Start	Chapter	Summary
00:01	Customer-support decision test	The model triages a customer’s failed integration message and returns team, blocked-status, and frustration scores.
01:08	What Decider V1.2 offers	The presenter introduces the model’s claimed architecture, one-pass answers, image support, and hardware requirements.
02:14	Testing seatbelt vision and legal reasoning	The model incorrectly judges two seatbelt images, including a test that asks it to apply Indian road-safety law.
04:30	Benchmark claims and reading a data sheet	After reviewing reported comparisons with V1.1, the presenter tests and praises the model’s accurate reading of GPU memory specifications.
06:05	Sentiment, language recognition, and verdict	The model gives a questionable satisfaction score and recognizes German, but the presenter says it lacks a compelling reason for production use.

Quotes

That is a reasonable read of a polite but stuck customer — 00:35

by no stretch of imagination, you can call it a correct seatbelt. — 02:47

So, that’s a clean pass on reading a visual document — 05:33

you have to make sure you check and then double-check. — 06:37

I’m not going to use this model in production whatsoever. — 07:09

Concepts and entities

pplx-decider-v1.2, Perplexity, decision model, VRAM, análise de sentimento, Fahd Mirza

Full transcript (original language)

00:01 We wanted help deciding how to answer a real customer message. So, we handed it to an AI decision model and asked three questions. First, which team should take it? Second, is the customer blocked right now? Third, how frustrated are they on a four-step scale? The message itself is very simple and integration has been failing with a 401 error since the team rotated their API key and they’re asking for help. If you look at the answer what model has

00:35 replied, it says that support at 99.8% which is the right team for a technical integration error. It also said the customer is blocked at 98.4% which fits a broken integration. For frustration, it returned 1.28 out of three which lands between mildly annoyed and frustrated. That is a reasonable read of a polite but stuck customer and it is a score you can rank and act on and this is a whole script which it has

01:08 generated. The model which I have have used behind the scene is this new Perplexity’s Decider, the new version V1.2. This model is a decision model on the Gwen 3.8 27 billion backbone. The model information is quite clear about what changed from V1.1. The model reads the whole request in one pass so several questions about the same message are answered together and it now handles images up to 1.6 megapixels as well as

01:42 well as text and we are going to test that out shortly. It is a 27 billion parameter model that would require around 28 to 29 GB of VRAM. Perplexity also states that some of its gain came from training on official benchmark splits. Now, let’s test it out with an image. By the way, this is Fahad Mirza and I welcome you to the channel. Please also subscribe to this free weekly AI newsletter at fahadmirza.substack.com.

02:14 And now let’s do a vision test with this Decider. I’m going to use this AI generated image where we are sending the seatbelt photo of a passenger in a car to this you know, PPLX Decider model and asking it this question. Just one yes or no. Is the seatbelt worn correctly? This checks whether the model can read a real-world image and not just text. And this is the answer. It says the seatbelt is worn correctly at 65.9% which is a lean towards a yes rather

02:47 than a firm call. So, you can see the model has slipped there because um by no stretch of imagination, you can call it a correct seatbelt. At least here in Australia. Um so, I don’t think so that is a correct answer. Okay, next uh I’m just going to try to stretch the model a bit more. The same kind of photo, a passenger in a car, but now with a question that depends on Indian road safety law. We are checking whether the model can apply a legal rule

03:21 to what it sees, not just recognize a seatbelt and you can see the answer. Again, look at this AI generated image where you can very clearly tell the seatbelt is not worn correctly. So, model says the seatbelt is worn correctly at 89.2%. The belt does sit across the um chest and lap, so the visible position is right, but the model never showed it knows what Indian law actually requires. Plus, the seatbelt should have been over the shoulder, not under the shoulder. So,

03:55 which is a totally wrong um vision from the model. So, a 27 billion decision model should have been able to explain that, and this score doesn’t show it. So, I really expected more from a 27 billion parameter model. And we have checked heaps of these decision models, as you can imagine. We have compared them too on the channel. So, if you just go and search with decision or decision comparison, you would see we have been covering them a lot. And even a smaller under sub billion models have performed

04:30 fairly well, and we have also covered some of the vision ones where they have really excelled. Even if you look at some of the numbers which they have shared on their model card, where they have compared their older model with their new model, the new model comes out ahead overall, and the biggest gains as per this card is uh on the language and knowledge task, which I’m going to do next. The one area where it does not clearly win is tools, where the two ways of sending questions score very close. Um because, and this is quite important,

05:02 because tool use decisions are a common real-world use. On image, this one leads its older version, uh but you just saw that it was not doing very well. And now, let’s play to model’s strengths. So, I have given this model a real GPU data sheet, an image with the specs written in it, and asked which memory sizes the card comes in. Four options were on the table, including one wrong answer, as you can see on the left. So, this is

05:33 checking whether the model can read a fact straight off the page, not just recognize a graphics card. So, you can see in this answer, it has picked 24 gig and 48 gig at 99.7, which matches the data sheet exactly. It put almost no weight on the other three options. So, that’s a clean pass on reading a visual document and it is a first result in this video where the answer can be checked against the source image line by line and it has done quite well. And now let me check one of its another

06:05 strength. What we are testing here is a customer message full of negatives and an idiom as you can see on your screen. Like I I can’t say I’m not disappointed, that sort of stuff. So, we asked the model whether the customer is satisfied overall. The honest answer is mixed. So, the score should sit near the middle in my opinion. But you can see that the result is 11.2% which leans clearly towards not satisfied. The customer did sound disappointed, so the model may have read the negative words more than the overall

06:37 tone. So, that is where we need to be careful with AI and these are the occasions where if you are doing sentiment analysis, you have to make sure you check and then double-check. I’m not entirely sure if it is a multilingual model or not though it is based on Gwen. But anyway, you can see that I’m giving it a German paragraph on screen about happiness and contentment. I asked the model one yes or no question. Is this text written in German? It’s a very simple check that the model should be

07:09 able to recognize a language from an image and you can see that it has recognized the language, but I don’t think so. Um it has understood the paragraph because it didn’t say it. So, I think it is an okay answer, but I was really expecting it to go beyond our expectation and do a wow factor there. That wow factor is simply not there. So, if you ask me, I’m not going to use this model in production whatsoever.

07:42 Because there are a lot of better options available, not only in the text, but also in the vision modality. But anyway, let me know your thoughts. What do you think about this model? And especially I’m very keen to know because we have covered so many of them. Uh if you have selected any model for your real-world use case, if yes, please share your experience and which model are you using. That’s it. Let me know your thoughts. And if you want to help out the channel, please become a member from the first pinned comment or the

08:14 homepage. Thank you for all the support.