🔍 Read the full analysis: What A Second AI Reviewer Saw In Claude’s Model Edit on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A report published Nov. 5 says OpenAI’s GPT-6 Astra reviewed a Claude Opus 5.5-led experiment that edited the weights of the local model Ornith 1.5. The reviewer found reproducible bugs despite passing tests and prompted corrections to the article’s claims; it did not run a competing model-improvement experiment or inspect the hidden benchmark tasks.
A review by OpenAI’s GPT-6 Astra found reproducible faults in the test rig and analysis for an experiment led by Anthropic’s Claude Opus 5.5, which edited the weights of the local model Ornith 1.5. The review matters because some faults could have made old or empty results look like valid new measurements, even while the project’s tests passed.
The account, published by Thorsten Meyer on Nov. 5, describes a division of labor between the AI systems. Claude handled the experiment, including the edit, benchmark runs and blind grading by three Claude agents. GPT-6 Astra, running in Codex, was asked to review and validate the implementation. The report says it did not run a competing model-improvement experiment and did not inspect the hidden benchmark prompts, topics, answers or reference solutions.
In its first review on Nov. 4, GPT-6 Astra reported four problems. Resuming a run could reuse answers from a different model or sampling setup; an unknown task ID could produce a nominally successful run with no evaluations; a comma conflict broke the real command line even though the new parser’s unit tests passed; and converting weights altered them even when the edit strength was set to zero. The author says Claude’s session fixed the issues and added tests that fail against the old code.
The report distinguishes two measurements of weight conversion. In the first review, conversion at zero edit strength changed 105,315 of 1,048,576 reconstructed values, or 0.84% of the matrix’s overall size, according to the project’s measurement. A later review measured a different conversion: storing one real matrix in another number format moved it by 0.51% and changed 915,360 of 1,048,576 values. These are measures of numerical changes during conversion, not legal-performance scores or general error rates.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
When Green Tests Miss Real Faults
The review illustrates a practical risk in AI experiments: a passing test suite does not establish that the actual command, data flow and result handling work as intended. Here, the reported resume behavior could have carried results from one setup into another, while a mistyped task could have appeared to finish without testing anything. Those faults would affect whether the benchmark results were trustworthy, not merely whether the software was convenient to use.
The weight-conversion findings also bear on interpretation. If a model’s values change during conversion, then an observed difference after an edit might not be caused solely by the intended intervention. The report says this possibility was identified and addressed, but the conversion measurements alone do not show whether the edit improved legal reasoning. The source says the model completed 12 of 15 legal answers after the edit, compared with 4 before, while its legal score remained zero. That result is a reported outcome from this experiment, not evidence of a general improvement in legal ability.
A second reviewer from another company offers a form of independent scrutiny, but it is not equivalent to a separate replication. GPT-6 Astra checked code and claims under defined limits; Claude’s session made the fixes, and the benchmark grading remained within the original project. The report’s value is therefore in the specific faults it reproduced and the qualifications it prompted, rather than proof that a second system confirmed the experiment’s overall conclusion.
Popular right now
Philips Café Aromis Automatic Espresso Machine with LatteGo Pro Milk System
De'Longhi Rivelia Fully Automatic Espresso Machine with Milk Frother, White
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Black
As an affiliate, we earn on qualifying purchases.
How the Review Roles Were Split
The experiment began on Nov. 4, when two AI sessions were working on the same project. According to Meyer’s account, the roles were separated once the overlap became clear. Claude retained the main implementation and experiment. GPT-6 Astra was instructed to review it, write separate check programs and report findings; Claude’s session applied the fixes. The reviewer’s preparatory files were set aside, and the report says they did not produce a second edited model.
The model-editing work was described as a directional ablation inspired by a post from MIT’s Computer Science and Artificial Intelligence Laboratory on weight-space programming. Claude’s session identified what the author called a “loop direction” in Ornith 1.5 and weakened it in the weights without training. The published account says the model then completed more legal answers, but the legal score stayed at zero. The article about GPT-6 Astra focuses on review of the implementation and claims, rather than presenting that outcome as a validated legal capability.
The source says the review followed three rules: do not alter the main implementation, test real entry points rather than trusting green unit tests, and avoid the hidden benchmark tasks. Those boundaries were intended to keep the review separate from both the code being checked and the benchmark material used to evaluate the model.
“Review and validate the other session’s implementation.”
— GPT-6 Astra’s review instruction, quoted in Meyer’s report
What the Review Did Not Establish
The report does not establish that the weight edit caused the change in answer completion, or that the approach improves legal accuracy: it says the legal score remained zero. It also does not describe a separate replication by GPT-6 Astra. The reviewer did not inspect hidden tasks, act as a fourth grader or run a competing model-improvement experiment.
The author also reports limits on the review itself: GPT-6 Astra did not rerun every test during its final documentation step, and its weight-conversion measurement was made on a CPU rather than the production GPU path. The account does not provide enough information to determine how those differences would affect the production results, or whether the reported fixes have been independently checked by another reviewer.
The article corrects several statements it says were too strong, including describing weights as “exactly identical at strength zero,” presenting “5.2 wrong citations per answer” without a needed qualifier, calling the review a “parallel Codex experiment,” and making a causal claim from one control that exceeded the evidence. The full definitions and supporting data for each correction are not supplied in the provided account, so their size and broader implications cannot be independently assessed here.
Further Validation of the Experiment
The immediate next step described in the report is the set of fixes made by Claude’s session, including rejecting unknown task IDs, changing the conflicting command syntax and retaining edited tensors in higher precision. The account says tests were added to catch the earlier faults. It does not announce a new benchmark run, a full independent replication or a further review milestone.
Readers assessing the result would need to see the corrected experiment’s methods and outputs, including how runs are tied to their model and sampling settings, how conversion is handled on the production GPU path, and whether the reported answer-completion change holds under suitable controls. Until those details or additional results are available, the review supports the narrower conclusion that it found implementation problems and prompted more cautious wording—not that the model edit has been shown to improve legal performance.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and claims in a Claude Opus 5.5-led experiment that edited the weights of the local model Ornith 1.5. It did not run a competing model-improvement experiment, according to the report.
What faults did the reviewer find?
The first review reported that resuming a run could reuse results from another setup, an unknown task ID could yield a successful empty run, a command-line comma conflict escaped unit tests, and converting weights changed values even at zero edit strength.
Did the model become better at legal questions?
The source reports that it completed 12 of 15 legal answers after the edit, compared with 4 before, but says the legal score stayed at zero. The report does not establish that the edit caused an improvement in legal accuracy.
Did the second AI inspect the hidden benchmark?
No. Meyer’s account says GPT-6 Astra did not inspect the hidden prompts, topics, answers or reference solutions.
Was the review a full independent replication?
No. GPT-6 Astra reviewed code and claims but did not produce a competing edited model. The author also says it did not rerun every test during its final documentation step and measured conversion on a CPU rather than the production GPU path.
Source: ThorstenMeyerAI.com
