Can You Spot the $0.15 Legal AI? (GLM-Legal Test)

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Can You Spot the $0.15 Legal AI? (GLM-Legal Test) · Channel: Prompt Engineer 48 · 11:16 · Watch on YouTube

Topics: Enterprise AI applications, AI evaluation and benchmarks, AI models, Privacy and data protection

Summary

The video blind-tests GLM Legal against its base model, GLM 5.3 Flash, using three legal tasks, with Claude added to the final comparison. GLM Legal’s answers are more specific and connect documents into arguments, but the first two examples rely on private materials the presenter cannot independently verify. Public checks confirm several cited cross-border transfer rules, while an independent benchmark shows GLM Legal resolves about one task in five at a low listed cost. The presenter sees it as a tool for lawyer-reviewed first drafts, not a substitute for legal review.

Key points

GLM Legal is described as GLM 5.3 Flash with additional legal training and reinforcement learning; at the time of the video, access is through a beta application.
In the first blind test, GLM Legal’s answer was more specific and client-favorable, but its correctness depended on a contract that was not available.
In the second test, GLM Legal connected reports and other documents into a concealment theory; the underlying 40 documents were not public.
In the website privacy task, the answers were identified as Claude (A), GLM 5.3 Flash (B), and GLM Legal (C).
The presenter counted citations across the tests, but emphasizes that more citations do not mean an answer is more correct.
The presenter verified Singapore PDPA section 26, Malaysia PDPA section 129, and Indonesia PDP Law article 56 as cross-border transfer rules; article 55 and other citations remained unchecked.
On Harvey’s legal agent benchmark, GLM Legal ranked seventh of 79 with a 19.17% task resolution rate and a 92.5% criteria pass rate.
GLM Legal costs 15 cents per benchmark test; the presenter estimates about 78 cents per fully solved task, while noting that this comparison assumes a lawyer can identify successful runs.
The video reports limitations: benchmark rankings are close within the stated error bars, tasks took over an hour on average, and GLM Legal ranked 24th of 78 on the legal research benchmark.
The presenter recommends using it for lawyer-reviewed first drafts or checklist comparisons, not filing its work or handling client matters without review.

Chapters

Start	Chapter	Summary
00:00	Blind test setup and access limitations	The presenter introduces GLM Legal and GLM 5.3 Flash, explains the blind-test format, and notes that the model is not yet readily available to run independently.
01:36	Three blind tests	The video reveals the first two comparisons and then tests GLM Legal, its base model, and Claude on a regional website privacy task.
05:22	Citation checks and benchmark results	The presenter compares citation counts, verifies selected transfer-law references, and reviews GLM Legal’s benchmark ranking and cost.
08:33	Cost, limitations, and verdict	The presenter estimates cost per fully solved task, discusses benchmark caveats and evaluation limits, and concludes that lawyer review is essential.

Quotes

The legal model connects them into an argument. — 03:14

More citations doesn’t mean more correct. — 05:22

Unverified means unverified, and I’m not a lawyer. — 06:24

I would not file anything it writes — 10:45

Concepts and entities

GLM Legal, GLM-5.3-Flash, Claude, z.ai, Harvey legal agent benchmark, PDPA, PDP law

Full transcript (original language)

00:00 Here are two answers to the same contract question. A client is switching software providers and the task is to fix the termination and handover terms. One answer came from GLM legal, the new legal model from z.ai. The other came from GLM 5.3 flash, the model it was built on. You get 5 seconds to pick which one is the legal model. Keep that pick in your head. I’ll tell you the answer in a couple of minutes and then I’m going to make the test a lot harder. I’m prompt engineer and in

00:32 this video we blind test GLM legal and fact check what it says. All right, GLM legal came out on October 12th, 2026 and this is the announcement on the z.ai blog. It’s built on GLM 5.3 flash with extra legal training in the middle of the training run and then reinforcement learning across tens of thousands of legal task environments. Now here’s the part that decides what I can do in this video. At the bottom of the page it says join the GLM legal beta and you apply for API access. The desktop agent platform says you’ll soon be able to use

01:05 it. So I can’t run GLM legal on my own today. The Vals page listed as an open weight model and there’s a download button, but when I opened z org / GLM legal on hugging face just now, I got a 404 so I can’t tell you the weights are public. What I can do is three things. One, z.ai published three real examples where GLM legal and its base model answer the same task and we’re going to blind test those one by one. Two, I’ll check the legal citations it gives against public sources. And three, I’ll read the

01:36 independent benchmark on Vals. And mind you, in round three I’m adding a third contestant. That’s Claude, same task, fresh session, one shot, no tools. Okay, round one, the one from the start. Answer A says consider shortening the notice period and return client data and says reduce the notice from 60 to 30 days, remove the cure period for breaches of IP, confidentiality, or data security, and give up to 90 days of transition help. The legal model was B, and if you picked B, you probably picked it for the same

02:10 reason I did. A says consider shortening the notice period, B says from 60 to 30 days. Look at one more detail here. A limits the return to deliverables the client has paid for. B says whether or not paid for and points to section four, which already assigns the work product to the client. Now, I can’t check that because the blog doesn’t include the contract. So, B is more specific and more aggressive for the client and whether it’s right depends on a document I can’t open. Write that down because it comes back in round three. Round two,

02:41 the task is to read 40 documents from a software project dispute and find contradictions in the provider’s progress reports. Answer A sites a September 30th report that says on track, 60% complete, and sets it next to a September 8th warning and an internal snapshot of about 41%. Then it adds the CEO’s instruction to suppress the risk and keep invoicing, and the contract terms saying payment doesn’t waive rights. It calls that a concealment theory and says to test the remaining claim elements before drawing a final conclusion. Answer B finds the same two reports but says these documents may

03:14 support the client’s position and that you should review further. Pick one, you’ve got 5 seconds. The legal model was A. So, in both rounds, the tell is the same figures, document references, and section references. The base model points at the right documents. The legal model connects them into an argument. And again, I can’t check any of this because those 40 documents aren’t public. So, rounds one and two tell us what it writes like they don’t tell us if it’s

03:45 correct. Round three, this task needs no private documents, so I could run it myself. Plan privacy compliance for a website launching in Singapore, Malaysia, and Indonesia with tracking, email marketing, and overseas cloud storage. Three answers. One is GLM 5.3 flash, one is GLM legal, and one is Claude in a fresh session with the same task, no tools, and one try. Quick disclosure, my first run of Claude came back as a numbered list at 141 words, which would have given it away. So, I ran it once more and asked for one

04:15 paragraph of about 85 words, the same shape as the blogs. This is the second run. Answer A starts with mapping data flows and appointing a data protection officer, then a consent banner, opt-in email with do not call checks, vendor contracts with transfer references, and breach response. Answer B says to publish a privacy notice, review consent for cookies and analytics, and check the applicable laws in the three countries. Answer C labels its first block P0 recommended before launch with English and Indonesian notices, opt-in for non-essential analytics, a suppression list for unsubscribes, and a vendor map

04:50 with section references for each country. You get 10 seconds. Here’s the reveal. A was Claude, B was GLM 5.3 flash, C was GLM legal. If you got all three, tell me in the comments. I want to know if the specific versus generic tell still worked when two of the three answers were specific. Let me count what separates these answers. I counted citations, meaning

05:22 document references in section or article references, and I counted a range like articles 55 to 56 as one. Round one, the base model has zero and the legal model has one. Round two, four against six. Round three is where the tail breaks. The base model has zero. GLM legal has three. Claude has seven. So, if you picked A as the legal model because it had the most citations, that was the trap. A was Claude. More citations doesn’t mean more correct. It means more things for a lawyer to check, and that’s the next job. Now, the part

05:53 that matters for a legal model. Answer C sites three laws for overseas transfers. Singapore PDPA section 26, Malaysia PDPA section 129, and Indonesia’s PDP law articles 55 and 56. Claude’s answer sites the same three transfer rules with article 56 for Indonesia. I checked them against public sources. Singapore section 26 says the transfer limitation obligation, and the PDPC guidelines say that Malaysia section 129 is the cross-border transfer rule, and the 2024

06:24 amendment removed the old white list with a guideline published on April 29th, 2025. Indonesia article 56 is the cross-border transfer provision. Article 55 I could not confirm from what I found, so that one stays unchecked. So, section 26, section 129, and article 56 all check out as the overseas transfer rules. I’m not going to claim more than that. I did not verify the other citations like the data protection officer and breach ones in Claude’s answer, and I’m treating those the same way. Unverified means unverified, and I’m not a lawyer. Now, the benchmark. This is Harvey’s legal agent benchmark

06:58 run by Vales, updated October 12th. GLM Legal is number seven of 79 at 19.17%. That’s about one task in five. At first, that sounds low, but look at the criteria pass rate, which is 92.5%. Harvey counts a task as resolved only if every single criterion passes. So, it satisfies most of the checklist and still fails the task most of the time. Now, look at the chart in z.ai’s own blog on Harvey. It shows GLM Legal at 19.2 with Kimmy K3 at 12.9, Claude Fable

07:30 5 at 11.3, and GPT-6 Astra at 5.4. Those numbers match Vales, but that chart has five bars. Vales’ full leaderboard has six models above GLM Legal. On Legal Bench, the blog says parity, and its chart has GLM Legal at 86.2 against Claude Fable 5 at 88.6, and GPT-6 Astra at 87.1. So, that’s close within about 2 and 1/2 points, but it’s not ahead. Here’s the cost. GLM Legal costs 15 cents per test. The leader, Mu Spark

08:02 1.2, scores 25.42% at $2.09 a test. Gemini 4 Argon is at 19.58% for $10.55, and Claude Fable 5 is at 11.25% for $19.23. And Val’s notes it fell back to Claude Opus 4.8 on four tasks. Compare that with the base model GLM 5.3 Flash at 6.67%. The blog says plus 12.5 points, and the math checks out. Here’s a number that’s easy to miss. If

08:33 you divide the cost per test by the share of tasks resolved, you get the cost per fully solved task. For GLM Legal, that’s about 78 cents. For Mu Spark 1.2, it’s about $8.20. For Gemini 4 Argon, it’s about $54. And for Claude Fable 5, it’s about $171. Now, that’s simple division, and it has a catch. It assumes you can tell which run solved the task, and that takes a lawyer. So, treat it as a way to compare prices, not a promise. And for scale, at listed prices, 100 pages of contract at roughly 50,000

09:06 tokens in and 5,000 out costs about half a cent. The token count is my rough assumption. The prices are Val’s. The benchmark’s 15 cents is higher because it’s an agent working for over an hour on one task. Now, the honest negatives. Rank five, six, and seven are separated by less than the error bars, which are plus or minus about three points. It took 1 hour 12 minutes per task here, where the top three took around 15 to 25 minutes. And on Val’s legal research bench, it’s number 24 of 78 at 41.83%

09:39 so it’s cheap and decent and it isn’t the best. Z.ai’s blog lists four questions that lawyers ask about an output. Are all critical issues and edge cases addressed? Are the arguments well substantiated? And are the citations legally sound? Does it follow the matter brief and can the team use it without rework? Here’s what my test could and couldn’t cover. Critical issues and edge cases, I can’t judge that because I don’t have the contracts of the 40 documents. Citations, I checked the three transfer rules and they point at the right laws. Following the brief, all three examples stayed on the task but I

10:12 only have Z.ai’s excerpts. Usable without rework, the blog says the desktop agent will put edits into Word with tracked changes and I can’t run that yet. So what do I make of it? In Z.ai’s three examples, the legal model gives numbers and sites documents and the base model gives general advice. The citations I could check pointed at the right laws. On the independent benchmark, it passes about one full task in five at 15 cents. I’d use it to draft a first pass that a lawyer then reviews or to compare against your own

10:45 checklist. I would not file anything it writes and I would not hand it a client matter without review because four out of five full tasks still fail and you can’t use it today anyway because it’s a beta with an application form. This is a test of how a model writes, not legal advice. If you’re looking for cheap and reliable GPUs, I use RunPod and the link is in the description. So in this video we saw how GLM legal does in a blind test and on the benchmark in the comments, tell me what you picked in round three, A, B, or C and I’ll post

11:16 the tally in the next video. I’ll see you in another one.