🔍 Read the full analysis: Our At-Home AI Experiment: Weight-Space Programming And A Zero Legal Score on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment used directional ablation to reduce looping in the local AI model Ornith 1.5 35B-A3B. In one comparison, completed answers rose from 4 of 15 to 12 of 15, but the edited model still scored zero on the legal benchmark and produced many incorrect or invented citations.
A home experiment reported November 5 found that a direct edit to a local AI model’s weights helped it finish more legal benchmark answers but did not improve its score: Ornith 1.5 35B-A3B completed 12 of 15 answers after editing, compared with 4 of 15 in the stated same-setup comparison, while scoring zero on both runs. The result suggests the edit reduced repetitive output, not that it gave the model legal knowledge.
The experiment followed a report from MIT CSAIL researchers on “weight-space programming,” in which one AI system edited another model’s weights. The home test used a related but different approach. According to ThorstenMeyerAI.com, a Claude Opus 5.5 session wrote code that ran Ornith, measured an internal direction associated with looping, and removed that direction at half strength from the output weights of 27 of the model’s 40 layers. The test involved no new training or legal data, and no AI inspected the weights as text.
On the reported same-setup comparison, finished responses increased from 4 of 15 to 12 of 15. However, the edited answers met only 7 of 75 checked legal criteria, up from 3, and contained 62 wrong or invented citations—an average of 5.2 per completed answer. The source says the benchmark checks cited statutes and court decisions against primary sources. The legal score remained zero.
The report also gives a result that complicates the apparent gain: with the model card’s recommended sampling settings and no weight edit, the original model finished 13 of 15 answers, scoring zero. The article says the experiment’s local setup changed during testing and that this alone shifted the original model’s completion count from 10 to 4. These figures are not a controlled demonstration that the edit outperforms ordinary settings.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completions, Not Better Legal Answers
The experiment illustrates a distinction that matters when AI is used for research or other high-stakes tasks: producing a complete response is not the same as producing a correct one. An intervention may help a system stop repeating itself while leaving its underlying knowledge and accuracy unchanged. Here, the edited model still scored zero under the report’s legal benchmark, and its completed responses included dozens of citation errors.
That makes the result a caution against treating output volume or fluency as evidence of reliability. For lawyers, researchers, and anyone using local models to find legal authorities, inaccurate citations can make an answer actively misleading. The source explicitly says the experiment is not legal advice; it does not establish that the model is suitable for legal work.
The findings are also limited in scope. They come from one model, one small set of questions, and one run per setting. They show a reported technical effect on looping, not a general method for improving legal reasoning. The experiment’s own comparison with recommended sampling settings leaves open whether the weight edit added value beyond adjusting how the model generates text.
Products Worth Considering
Apple MacBook Air 15-inch with M5 chip, 24GB RAM, 512GB SSD, Midnight
Samsung Galaxy Tab A11+ (6 GB RAM, 128 GB storage, expandable) – Gray
Samsung Galaxy Tab A11 Wi‑Fi (International Version), 8.7, 64GB + 4GB, Gray
As an affiliate, we earn on qualifying purchases.
From MIT’s Edit to a Home Test
The motivating MIT CSAIL work was described by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. In their reported Qwen case, GPT-6 Astra identified a direction associated with looping and altered a small portion of the model’s weights; the source says the model’s mathematics score rose from 7.9% to 48.1%. The researchers’ stated caveat was that an edit could remove an obstacle hiding an ability, but could not add an ability the model lacked.
The home report says it did not replicate that weight-reading method. Instead, it used directional ablation: run the model, collect internal signals, calculate a direction, and project it out of selected weights. The article describes this as a longstanding technique, distinct from the newer method it was inspired by.
Ornith was selected because the report says it was the local model in the benchmark that showed the relevant looping problem and could be tested on the available machine. Its publisher describes it as a model built for coding and agent work, with about 35 billion parameters overall and roughly 3 billion active per token. The legal test, Lexassay, asks 15 questions across German, U.S. and EU law without internet access or tools; the report says five questions are hidden. The new runs described in the article are not reflected on the benchmark website.
“The edit repairs finishing, not legal knowledge.”
— ThorstenMeyerAI.com experiment report
Limits of the One-Run Comparison
The results are preliminary: the report describes one run per setting, a small development set, and a loop detector that changed during the experiment. It also says a local setup change moved the unedited model’s completion count from 10 to 4, making the comparison sensitive to conditions. The source does not provide enough information here to determine how much of the 4-to-12 difference came from the weight edit rather than other settings or run-to-run variation.
The report says Claude carried out the edit and Claude agents graded responses, creating a potential evaluation conflict. OpenAI’s GPT-6 Astra, run in Codex, independently reviewed code, numbers, and claims and found faults, according to the report. That was a review of the experiment, not a second experiment; the article also notes that three graders came from the same model family. The precise corrected results and the effect of each identified fault are not detailed in the supplied material.
It is also unclear whether the edit’s effect would hold across repeated runs, other models, or legal tasks beyond this benchmark. The reported unedited run with recommended sampling finished 13 of 15, so the available figures do not establish that editing weights is a better way to improve completion than using the model’s recommended settings.
Further Testing Needed
The source does not announce a date for follow-up testing. The immediate next step for evaluating the claim would be repeated comparisons under a fixed local setup, with identical prompts, token limits, and sampling settings, across edited and unedited versions. Reporting the results across multiple runs would help separate a consistent effect from variation in a single test.
Further checks would also need independent grading and a clear account of how identified review faults affect the measurements. The report says the new results have not been posted on Lexassay’s public site, which still displays the original Ornith result. Until additional results are available, the finding should be read narrowly: the reported edit was associated with fewer unfinished answers in one comparison, while legal benchmark performance remained at zero.
Key Questions
What did the experiment change?
It removed an internal direction linked to looping from the output weights of 27 of Ornith’s 40 layers, at half strength. The report says this used directional ablation, not the MIT team’s method of reasoning directly over another model’s weights.
Did the edit improve the model’s legal score?
No. In the reported same-setup comparison, the legal score was zero before and after the edit. The edited answers met 7 of 75 checked criteria and included 62 wrong or invented citations.
Why did the model finish more answers?
The report links the increase to reducing a direction associated with repetitive looping. But it describes only one run per setting and notes that the local setup changed, so the available evidence does not isolate the edit’s effect with confidence.
Did an independent reviewer check the work?
According to the report, OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, numbers, and claims and found faults. It was an independent review, not a separate experiment; the report says Claude carried out the edit and Claude agents graded responses.
Can this result be used as legal advice?
No. The source explicitly says the experiment is not legal advice. The model scored zero on the reported legal benchmark, and the experiment was small and preliminary.
Source: ThorstenMeyerAI.com
