🔍 Read the full analysis: Z.ai’s Legal AI And The Benchmark Puzzle: Few Passes, Near-Top Ranking on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s version of Harvey’s Legal Agent Benchmark, ranking seventh among 81 models in the archived leaderboard. The score counts a task as passed only when every rubric criterion is met; the same run recorded a 92.5% individual-criteria pass rate. The benchmark result does not establish reliability for legal work, and questions about citation accuracy, data handling and verification remain.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on an all-criteria test of legal-agent tasks, placing it seventh among 81 models on Vals AI’s archived leaderboard. The result is notable both for the model’s ranking and for what the score means: it completed every required part of roughly one in five tasks, under a strict grading rule.
Z.ai describes the model as optimized for legal document understanding, reasoning and task execution, including research, contract review and drafting. Its documentation says the system underwent legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of simulated legal-workflow task environments. The model is built on GLM-5.3-Flash, according to the company’s October 13 AutoClaw blog post, which opened a beta by application.
Z.ai reports an all-criteria pass rate of 19.2% on Harvey’s Legal Agent Benchmark and says the result is 12.5 percentage points above the base model. The archived Vals AI leaderboard gives the legal model a score of 19.17%, compared with 6.67% for GLM-5.3-Flash. That is a 12.50-percentage-point difference on the same leaderboard version. Vals lists the legal model’s cost at $0.15 per test; Z.ai’s pricing page separately lists $0.37 per million input tokens and $1.25 per million output tokens.
The benchmark’s all-or-nothing task score is not the same as its individual-criterion score. Vals reports that GLM Legal passed 92.5% of individual criteria in that run, but a task counted as complete only if every rubric criterion passed. Vals says its leaderboard score averages task pass rates assigned by two AI judges, GPT-5.5 and Claude Sonnet 4.6, on a private holdout set of 120 tasks. Internet access is disabled during testing.
A Strong Rank With Strict Limits
The result highlights a gap between partial accuracy and complete work products. A model may satisfy most individual requirements yet fail a whole task because one citation, deadline, recommendation or other required detail is wrong. For law firms, that distinction matters: benchmark placement alone does not show whether an output is safe to use without close review.
The reported 92.5% criteria pass rate offers a different view from the 19.17% task score, but it should not be read as a guarantee that a document is 92.5% correct or usable. The benchmark measures performance on its defined tasks and rubrics. It does not, by itself, establish the model’s reliability across live client matters, jurisdictions or firm-specific workflows.
Cost is another point of interest: Vals lists the test at $0.15 per task, lower than several higher-ranked entries in its archived table. That is a benchmark cost figure, not a complete measure of procurement expense. Firms would also need to evaluate integration, data controls, human review and the cost of correcting errors.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
Logitech MX Keys S Wireless Keyboard, Graphite
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as an open-source evaluation on May 6, 2026. The public set contains 1,250 tasks across 24 legal practice areas, assessed against more than 75,000 expert-written rubric criteria. Each task supplies an instruction and client-matter materials, then asks an agent to produce a work product for review.
Harvey calls its grading approach all-pass grading: a task is complete only when every criterion passes. The criteria can cover facts, legal conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting. Harvey has argued that identifying many risks in a deal report is not enough if the work product misses a material requirement.
The ranking cited here is Vals AI’s archived version of its leaderboard, last updated October 18, 2026. Z.ai’s October 13 blog post names Vals as the evaluator, while its model documentation does not identify the operator. Z.ai calls the result state-of-the-art among open-source models, but the supplied announcement does not specify the comparison set. The source material also says published weights for this legal model were not found, making the open-source comparison difficult to assess from the available information.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%.””
— Z.ai
Accuracy and Data Controls Unresolved
The benchmark establishes a result under one evaluation setup; it does not settle how the model performs across legal work outside that test. The source material describes a small, four-task spot check in which the writer said the model cited rulings that could not be found. That check was not a benchmark and cannot establish a general error rate. The same check found that two Claude models cited a real ruling for the opposite of what it held, illustrating that citation and interpretation risks are not unique to one system.
It is also unclear from the cited Z.ai pages who processes client data, under what rules, and what safeguards apply when the model is connected to external services. Z.ai recommends using it as an agent linked through MCP to legal sources and company data. The source material says the pages reviewed did not carry a “not legal advice” notice, while describing outputs as drafts for legal professionals. Details of data retention, deployment choices and the legal model’s published weights were not established by the materials provided.
The precise meaning of “state of the art among open-source models” also remains uncertain because Z.ai did not identify the comparison set in the cited claim. The archived leaderboard supports the stated rank and score for that version, but rankings can change as models are added or evaluations are updated.
Watch for Evaluation and Deployment Details
The next useful developments would be updated leaderboard results, a clearly defined comparison set for Z.ai’s open-source performance claim, and additional evaluation results that can be compared with the current holdout test. Z.ai’s AutoClaw post opened a beta by application; the source material does not specify a broader release date.
For potential legal-sector users, the outstanding practical checks are whether the product can be deployed under their data rules, how its citations and legal conclusions are verified, and what review steps are required before drafts are shared externally. The benchmark provides evidence of performance on a defined set of tasks, not a recommendation to rely on the model for legal advice. Legal AI outputs can be wrong or incomplete, so professional review remains necessary.
Key Questions
What did Z.ai’s legal AI score?
Vals AI’s archived leaderboard lists GLM-5.3-FlashX-Legal at 19.17% on the all-criteria task pass rate for its version of Harvey’s Legal Agent Benchmark. Z.ai rounded the result to 19.2%.
Why is the pass rate lower than 92.5%?
The 19.17% figure counts whole tasks and requires every criterion on a task to pass. The 92.5% figure is the share of individual criteria passed in the reported run. Missing one required criterion can fail an otherwise mostly correct task.
What does seventh place mean?
It means the model ranked seventh among 81 entries in Vals AI’s archived leaderboard version, based on that evaluation’s task pass rates. It does not show that the model is reliable for every legal workflow or that the ranking will remain unchanged.
Is the model ready to provide legal advice?
The cited materials describe outputs as drafts for legal professionals and do not establish that the model can safely provide legal advice without review. Legal outputs may contain errors or missing details; citations and conclusions require professional verification.
What is not yet known about using the model?
The materials do not fully establish client-data handling, retention rules, deployment safeguards or the comparison set behind Z.ai’s open-source performance claim. The beta was opened by application, with no broader release date given in the supplied information.
Source: ThorstenMeyerAI.com
