🔍 Read the full analysis: Z.ai’s Legal AI Ranks Near The Top—What Its Benchmark Tasks Reveal on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal posted a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models. It passed 92.5% of individual criteria, but the benchmark counts a task as complete only when every required criterion passes. The score is independently listed by Vals; Z.ai’s claim that it is state of the art among open-source models is not supported by a named comparison set in the material provided.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s Harvey Legal Agent Benchmark leaderboard, with a 19.17% all-criteria pass rate. The independent leaderboard measures whether a model completes legal-work tasks to every requirement, making the result relevant to firms considering legal AI—but it does not mean the model got just 19% of the underlying legal analysis right.
Vals AI’s archived Version 1 leaderboard reports that GLM Legal passed every rubric criterion on 19.17% of tasks, at a listed cost of $0.15 per test. The same leaderboard gives it a 92.5% pass rate across individual criteria. Under the benchmark’s all-or-nothing task rule, a response can satisfy most criteria and still count as a failed task if it misses even one required item.
Z.ai says the result is a 19.2% all-criteria pass rate, an improvement of 12.5 percentage points over the base model, and describes its performance as state of the art among open-source models in the evaluation. The arithmetic for the improvement matches the leaderboard: GLM Legal’s 19.17% minus the base GLM-5.3-Flash score of 6.67% is 12.50 percentage points. But the source material does not identify the comparison set behind the open-source ranking, and it provides no published weights for the legal model.
Z.ai’s developer documentation describes GLM-5.3-FlashX-Legal as designed for document understanding, legal reasoning and task execution, including research, contract review and drafting. The company says the model received legal-domain mid-training and post-training, including reinforcement learning in tens of thousands of simulated legal-workflow environments. Its October 13 AutoClaw blog post opened a beta by application and said the model was built on GLM-5.3-Flash.
What the Score Means for Legal Work
The result gives buyers a measurable comparison point, but its two rates describe different things. The 19.17% task pass rate is a strict end-to-end measure: each task contains several requirements, and any missed criterion makes the whole task incomplete. The 92.5% criterion pass rate indicates that the model often met individual requirements, while not reliably satisfying every element of a full assignment.
That distinction matters in legal work, where a largely correct document may still be unsafe to use if it omits a deadline, misstates a dollar amount or cites a case incorrectly. Harvey’s benchmark illustrates the granularity: one change-of-control task has 57 criteria. A single failure can sink the task score even when the rest of the work is correct. Conversely, a high criterion rate should not be treated as proof that a document is ready for a client or court.
For law firms, the leaderboard is one input to procurement, not a substitute for review. Z.ai lists API prices of $0.37 per million input tokens and $1.25 per million output tokens, while Vals lists $0.15 per benchmark test. Those figures are different cost measures, and neither establishes the total expense of a deployed workflow, including data controls, human review and connected legal research services.
Popular right now
Sony BRAVIA XR8B 65-Inch 4K OLED Google TV (K-65XR8B)
Ninja Foodi Dual Zone Smart XL Air Fryer, 10 Qt, 6-in-1, Black (DZ550)
De'Longhi Rivelia Fully Automatic Espresso Machine with Milk Frother, White
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The public release contains 1,250 tasks across 24 legal practice areas. Each task pairs an instruction with materials from a client matter and asks an agent to produce work for review. More than 75,000 expert-written rubric criteria cover items such as facts, conclusions, citations, recommendations, deadlines, dollar amounts and formatting.
Harvey calls its scoring rule all-pass grading: a task is complete only if every criterion passes. The public leaderboard uses a private holdout set; Vals says its holdout contains 120 tasks and that internet access is disabled. Two language-model judges grade each submission, and the reported task score averages their pass rates. Vals’ archived leaderboard lists GLM Legal at 19.17%, while Z.ai rounds the figure to 19.2%.
Earlier benchmark results offer context but are not directly comparable. Harvey’s May 26 baseline reported a top all-pass rate of 7.1% among the frontier models it tested and said the models completed fewer than 10% of tasks end to end in aggregate. That earlier run used a different evaluation and should not be read as a direct head-to-head with the later Vals listing.
“On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model.”
— Z.ai, in its developer documentation
Limits of the Leaderboard Evidence
The leaderboard establishes a benchmark result under a specific scoring setup; it does not show how the model performs across every firm’s matters, jurisdictions or connected data systems. The score comes from 120 private holdout tasks, judged by two AI systems, rather than a broad sample of live legal work. The supplied source material does not report confidence intervals or human-review results for this model.
Z.ai’s description of the model as state of the art among open-source systems also remains difficult to assess from the available information. The company does not name a comparison set, and the material says the legal model’s weights have not been published. Its developer documentation does not identify the benchmark operator, though the October 13 blog post names Vals AI.
A separate four-task spot check described in the source material found citations to rulings that the reviewer could not locate. That small check is not a benchmark and cannot establish a general error rate. The same reviewer reported no invented rulings in OpenAI and Claude models in that check, while two Claude models cited a real ruling for the opposite of its holding. The source also says Z.ai’s pages describe outputs as drafts for legal professionals, but none of the pages reviewed carried a “not legal advice” notice. How the model handles client data, and what safeguards apply in particular deployments, are not specified here.
Questions Firms Should Ask Before Use
Z.ai’s AutoClaw beta was opened by application, and the company recommends connecting the model to MCP services for statutes, cases and company data. The next meaningful evidence will be further leaderboard results and independent evaluations that clarify the comparison set, reproducibility and performance across different kinds of legal tasks.
For firms evaluating the beta, practical checks include what client information is sent to Z.ai or connected services, where it is processed, what retention and access rules apply, and who verifies every citation and factual claim. The benchmark supports a narrow conclusion: GLM Legal performed near the top of this leaderboard while completing fewer than one in five tasks under its strict all-criteria rule. Whether it is dependable or cost-effective in a particular legal workflow remains to be tested by the firm using it.
Key Questions
What does GLM Legal’s 19.17% score measure?
It is the share of benchmark tasks on which the model passed every required rubric criterion, as reported on Vals AI’s archived leaderboard. Missing one criterion makes a task a fail, even if the response meets most others.
How can the model have a 92.5% criterion rate but a 19.17% task rate?
The figures use different units. The 92.5% rate concerns individual criteria; the 19.17% rate counts complete tasks, each of which can contain many criteria. A high rate on separate requirements can still produce a much lower rate of tasks with no misses at all.
Was the 12.5-percentage-point improvement independently verified?
The figures on Vals AI’s leaderboard support the arithmetic: 19.17% for GLM Legal versus 6.67% for the base model equals a 12.50-point difference. That comparison is tied to the listed benchmark results and does not establish performance in other settings.
Does the ranking prove the model is ready for legal practice?
No. The leaderboard is a bounded evaluation, not evidence that outputs can be used without review. The benchmark score does not resolve questions about citation accuracy in live work, client-data handling, jurisdictional coverage or a firm’s own review requirements.
Source: ThorstenMeyerAI.com
