🔍 Read the full analysis: Testing Legal AI: 15 Questions Asked Of 42 Different Setups on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
ThorstenMeyerAI.com reports that its Lexassay test asked 42 AI setups 15 legal questions covering German, US and EU law. Its operator says legal-focused GLM-5.3 FlashX Legal scored 8 out of 100, while Claude Opus 5.5 led with 79; the site also disclosed that Claude agents graded answers and that its operator runs the test.
ThorstenMeyerAI.com has published results from Lexassay, a test that put 42 AI setups through 15 legal questions, reporting that the law-focused GLM-5.3 FlashX Legal scored 8 out of 100 while leading model Claude Opus 5.5 scored 79. The results matter because errors in legal deadlines or citations can mislead users, but the test is a limited spot check rather than a broad measure of real-world legal performance.
The test covered German, U.S. and European Union law, including deadline calculations, recent legal changes, court decisions and questions containing false assumptions. Lexassay says ten questions are public and five are hidden; the hidden-task scores are published without the questions. Models answered without internet access or external tools, and different reasoning-effort settings counted as separate setups.
According to the results, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The test operator says the leading model met every criterion on only eight of the ten public questions. Mistral Large 4 scored 10, while GLM-5.3 FlashX Legal scored 8 and received zero on the hidden set. Its base model scored 3, the report says.
The citation review found 2,644 citations assessed as real and 804 classified as invented or not supporting the claim made. Lexassay deducts points for invented or misattributed citations. The report says GLM-5.3 FlashX Legal produced 21 real citations, 25 misattributed citations and one invented citation. On two German-law tasks, it used its full 64,000-token thinking budget without returning an answer.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal Errors Can Change Outcomes
The test illustrates why a plausible-sounding legal answer is not the same as a reliable one. In the example given by the report, two German clients received civil judgments on the same date and wanted to appeal. Because Corpus Christi falls on different relevant calendars in Bavaria and Berlin, the reported appeal deadline differed by one day: May 28 in Munich and May 27 in Berlin. The report says one model gave Berlin the later date, while another placed the deadline for written appeal grounds a month too late.
Those examples are the test operator’s account of model responses, not evidence that a user actually missed a deadline. Still, the stakes are clear: under German civil procedure, the report says a late appeal is rejected as inadmissible. Citation errors pose a related risk, since a genuine law or ruling may be cited for a proposition it does not support. The findings support treating chatbot output as a starting point for research, not as legal advice or a substitute for checking primary sources.
Popular right now
Ninja Foodi Dual Zone Smart XL Air Fryer, 10 Qt, 6-in-1, Black (DZ550)
De'Longhi Rivelia Fully Automatic Espresso Machine with Milk Frother, White
TCL 65-Inch QM6K QLED Mini-LED 4K HDR Smart Google TV, 144Hz
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Answers
Lexassay compares each answer with a prepared gold answer containing three to six weighted criteria, according to the site. It checks citations against official databases and classifies them as real, misattributed, invented or not findable. A misattributed or invented citation costs a quarter of the task’s points; the method also deducts 5 percent when an answer lacks a legal-advice disclaimer or referral to a lawyer.
The publisher describes the exercise as a spot check, not a strict benchmark: each model and task receives one answer, so small score differences may reflect noise. The test measures what models produce without search or tools, rather than how a legal product might perform with supplied case files or professional review. The report notes that Vals AI’s legal-agent benchmark, which gives models documents, ranks some of the same systems differently: Mistral Large 4 is eighth there, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th.
There is also a disclosed conflict of interest. Thorsten Meyer AI runs Lexassay, and the answers were graded blind by several Claude agents made by Anthropic; a Claude model ranked first. The publisher says its method, public answers, gold answers and citation checks are available for inspection. The results and rankings are claims from the test publisher and have not been independently verified in the supplied material.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in its developer documentation by the report
Limits of the Published Results
The results do not establish how these systems perform across the full range of legal work. The test used one answer per setup and question, excluded internet search and tools, and included only 15 questions across three legal systems. The report itself says small score differences may be noise. It is not clear from the supplied material how the systems would perform with repeated trials, updated legal sources, document sets or lawyer supervision.
The publisher disclosed that its own organization operates the test and that Claude agents graded answers, while a Claude model placed first. Publishing the method and checks gives readers material to review, but does not remove the potential conflict. The supplied source also ends partway through a discussion of GLM-5.3 FlashX Legal’s other ranking; no complete explanation of that ranking or response from Z.ai is provided here.
Review Sources Before Relying
The publisher says the public answers, gold answers, citation checks and scoring method are available for readers to examine; five task prompts remain hidden, with scores published instead. Readers seeking to verify a specific result can compare the disclosed answer and citation against the relevant official law or court decision. The source does not announce a date for another test or specify when additional results will be released.
For users, the immediate practical step is to verify every legal citation and avoid relying on a chatbot for a filing deadline, contract decision or communication with an authority. The report advises asking a lawyer when money, a deadline or an official matter is at stake. It describes an evaluation of AI systems, not legal advice.
Key Questions
What did the Lexassay test measure?
It assessed 42 AI setups on 15 questions involving German, U.S. and EU law. The publisher says models answered without internet access or tools and were graded against prepared answers, with citations checked against official databases.
Which setup had the highest reported score?
According to ThorstenMeyerAI.com, Claude Opus 5.5 led with 79 out of 100, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The publisher says the leading model fully met every criterion on eight of the ten public tasks.
How did the law-focused GLM model perform?
GLM-5.3 FlashX Legal scored 8 out of 100 and zero on the hidden questions, according to the report. Its base model scored 3. The test publisher says the legal version also returned no answer on two tasks after using its full stated thinking budget.
Does this test show that AI can provide legal advice?
No. The publisher calls Lexassay a spot check, and the test excluded search and tools. Its results do not establish reliability for a particular legal matter. The report advises checking citations and consulting a lawyer about deadlines, contracts or matters involving authorities.
Who ran and graded the test?
Thorsten Meyer AI runs Lexassay, and the publisher says several Claude agents graded answers blind. It disclosed that connection and that a Claude model ranked first, and says the method and public scoring materials are available for review.
Source: ThorstenMeyerAI.com
