🔍 Read the full analysis: We Put AI To Work On A Local Model, And The Legal Score Stayed At Zero on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment edited the output weights of the local model Ornith 1.5 35B-A3B to reduce looping. In a comparison on the same setup, completed answers rose from 4 of 15 to 12 of 15, but the legal score remained zero; the unedited model finished 13 of 15 with its recommended sampling settings.
A local-model experiment reported November 5 found that editing Ornith 1.5 35B-A3B’s weights helped it finish more legal benchmark answers, but its legal score stayed at zero. In a same-setup comparison, the edited model completed 12 of 15 answers, up from 4 of 15, while a recommended-settings run without the edit completed 13 of 15.
The experiment followed a report from four MIT CSAIL researchers on using AI to edit another model’s weights. The home test, conducted from Wednesday evening to Thursday morning, asked whether reducing a local model’s repetitive output would improve its performance on legal questions. The authors say the edit was technically successful at reducing the completion problem, but did not supply the legal knowledge or accuracy the benchmark required.
The team measured an internal direction associated with looping by running Ornith, then removed it at half strength from the output weights of 27 of the model’s 40 layers. The work involved no additional training, legal data or direct reading and editing of weights by an AI. Instead, a Claude Opus 5.5 session wrote code to run the model and apply the change, described in the report as directional ablation.
Among the 12 edited answers, the team counted 62 wrong or invented citations, an average of 5.2 per answer. On 75 checked criteria, the edited model met seven, compared with three before editing. The report says the legal score was zero in both cases. It also reports that the original model completed 13 of 15 answers when run with the sampling settings recommended by its model card and without any weight edit.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completions, Not Better Law
The result illustrates why finishing a response is not the same as answering correctly. A model that stops looping may be easier to use, but in a high-stakes area such as law, more complete output can still contain incorrect citations and weak analysis. A completion count alone would have made the intervention look successful; the benchmark’s quality checks showed a different outcome.
The comparison also puts the intervention’s practical value in doubt for this particular test. The edited model completed 12 answers on the stated same-setup comparison, while the unedited model completed 13 under recommended sampling settings. Those are reported runs, not a broad evaluation, but they show that a simpler configuration change may address the visible symptom without modifying weights. The report does not establish that the edit improves legal performance across models or tasks.
For readers evaluating AI systems, the distinction matters: reliability measures should include whether an answer is accurate and supported, not just whether the system produces a response before a limit. The experiment’s zero score and citation errors are especially relevant because the benchmark checks references against primary legal sources.
Popular right now
Hisense 40-Inch Class QD4SR Series Hi-QLED FHD Smart Roku TV (2026 Model)
Hisense 55-Inch E6SR Series Hi-QLED 4K UHD Smart Roku TV (2026 Model)
Shark AV2501AE AI Robot Vacuum with XL HEPA Self-Empty Base
As an affiliate, we earn on qualifying purchases.
Why Ornith Was Tested
The experiment was inspired by work the source describes as “weight-space programming,” in which an AI directly reasons about and changes another network’s weights. The MIT researchers’ reported Qwen example involved a different method: according to the source, GPT-6 Astra identified a looping direction by reasoning over the weights and changed a small portion of them. The home experiment did not replicate that technique; it used a more established approach based on model activations and a computed direction.
The chosen model was Ornith because the team’s preliminary checks suggested that looping, rather than only missing knowledge, was a visible problem. Its model card describes it as a mixture-of-experts model with about 35 billion total parameters and about 3 billion active per token, built for coding and agent work rather than law. The report says its earlier legal benchmark score was 1 out of 100 and that five of 15 answers did not finish, reaching a 62,000-token output limit.
The Lexassay benchmark described in the report uses 15 questions across German, U.S. and European Union law, without internet access or tools. It checks cited statutes and court decisions against primary sources; ten tasks are published and five are hidden. The article says its new experiment runs are not reflected on the benchmark website, which still displays the original Ornith result.
“The model finishes is not the answer is right.”
— Thorsten Meyer, reporting the experiment
Limits of the Test Results
The findings are preliminary. The report describes one run per setting, a small development set and a loop detector that changed during the experiment. It also says a change in the local setup by itself shifted the original model from 10 completed answers to four, making it difficult to isolate the effect of the weight edit from other conditions.
The report discloses that Claude carried out the edit and Claude agents performed blind grading. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, numbers and claims and found faults, according to the authors. That was an independent review by another AI system, not a second experiment. Three graders from the same model family were also used. The material provided does not detail the faults found, show a larger replication, or establish how the edit performs on other legal tasks or models.
Replication and Benchmark Checks
The next useful test would repeat the comparison under stable, documented settings, with more runs and a larger evaluation set. It would also compare the weight edit directly with the model-card sampling settings, since the unedited recommended-settings run completed more answers in the reported results. Independent grading against primary legal sources would help determine whether any change affects accuracy, not just response length.
The article does not announce a planned follow-up date or a scheduled release of new results. The Lexassay site, according to the report, has not been updated with these experimental runs. Until further evidence is available, the result supports a narrow conclusion: the edit reduced an observed completion problem in this test, but did not improve the reported legal score.
Key Questions
What did the weight edit change?
The team removed half of a measured loop-related direction from the output weights of 27 of Ornith’s 40 layers. It used code to run the model and apply the change; it did not train the model on new legal data.
Did the edit improve the model’s legal score?
No. The report says the legal score remained zero. The edited answers also contained 62 wrong or invented citations across 12 responses.
What happened when the original model used recommended settings?
In the reported comparison, the unedited model completed 13 of 15 answers using its model-card-recommended sampling settings. The report gives its legal score as zero.
Was this the same technique used in the MIT example?
No. The source says the MIT example involved an AI reasoning directly over another model’s weights. This experiment used directional ablation: it ran the model, measured an internal direction and removed that direction from selected weights.
How strong is the evidence?
The result is limited to one experiment with one run per setting, a small development set and reported grading limitations. It has not established that the edit improves legal answers across repeated trials, other models or other benchmarks.
Source: ThorstenMeyerAI.com
