🔍 Read the full analysis: When LLMs Judge Research Ideas, How Much Should We Rely On Them? on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Ai2 researchers report that six large language models gave unstable judgments of scientific idea novelty across evaluation setups. In one pairwise test, changing a few prompt words shifted accuracy by more than 50 percentage points; literature retrieval and additional reasoning did not consistently help. The team released a benchmark and code, while warning that some idea labels are imperfect.
Six large language models gave unstable judgments about whether scientific ideas were novel when researchers changed evaluation setups, according to the original analysis from the Allen Institute for AI (Ai2). In one pairwise test, changing a few words in the prompt shifted accuracy by more than 50 percentage points, raising questions about using these systems to screen research proposals before human review.
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Researchers assessed the models using both absolute novelty ratings and pairwise comparisons. They report that adding literature retrieval or allowing more reasoning did not consistently improve judgments. The findings indicate that a score from one prompt or configuration may not reliably represent how a model would assess the same idea under another setup.
For one benchmark, the researchers used explicit originality comments in ICLR 2026 peer reviews to select 154 ideas from accepted papers rated in the top 10% of their research areas and 145 ideas from rejected papers in the bottom 10%. The selected reviews included praise for novelty in the high-novelty group and originality concerns in the lower-novelty group, without opposing comments. An annotator agreed with the extracted novelty-signal direction in 47 of 50 checked cases; the authors say the three disagreements did not change the review classifications.
A second test paired the 154 human-authored ideas with 154 proposals from a basic language-model generator. The researchers treated the generated proposals as less novel, but described those labels as weak and not individually verified. In a separate expert check of 30 pairs, selected heavily from cases the strongest judge had misclassified, an expert preferred the human-authored idea in every pair and judged none of the generated proposals novel. Ai2 released the paper, benchmark data, code and an automated pipeline for building further evaluation sets.
Why Unstable Scores Matter
Research teams may use automated novelty judgments to decide which ideas merit literature searches, experiments or expert attention. If small prompt edits cause large changes in a model’s verdict, its score may reflect evaluation choices as well as the idea itself. That makes it harder to compare proposals across projects or to know whether a screening system is consistently identifying originality.
The finding also bears on efforts to automate parts of scientific discovery. Systems that generate hypotheses or summarize research need credible ways to assess whether a proposal adds something new. The study does not show that all LLM novelty assessments fail, or that these models cannot help researchers. It does show why performance on one chosen evaluation setup is not, by itself, evidence of dependable judgment.
Popular right now
Breville Bambino Espresso Machine, Brushed Stainless Steel
Dyson Ball Animal 2 Plus Corded Vacuum
Sony WH-1000XM6 Wireless Noise-Canceling Over-Ear Headphones, Blue, Bundle
As an affiliate, we earn on qualifying purchases.
How Ai2 Built Its Test Sets
In the study, a novelty judge is a model asked to assess whether an idea is new relative to previous work, either by rating it alone or comparing it with another idea. The authors say there is no standard evaluation recipe: researchers can vary the model, prompt, comparison format, reasoning budget and use of literature retrieval. They also argue that judges are often tested on human-written papers even when their intended purpose is to evaluate AI-generated proposals.
The benchmark therefore includes a set based on human ideas with reviewer evidence about originality and a second setting pairing human ideas with model-generated proposals. The authors caution that acceptance or rejection alone is not a sound novelty label, because papers can succeed or fail for reasons other than originality. Their selection criteria use review comments and rankings as signals, not as a definitive measure of whether an idea is genuinely new.
““Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Novelty Labels
The benchmark’s labels are based partly on reviewer comments and selection criteria, which cannot establish originality conclusively. In the human-and-generated comparison, the generated ideas were treated as less novel without individual verification across the full set. The expert assessment covered 30 pairs, not every benchmark example, and those pairs were selected heavily from cases the strongest judge got wrong.
The supplied study description does not give full results for every model, prompt, reasoning budget or evaluation format. It also does not establish whether the findings apply across other research fields, different idea generators or systems with broader access to scientific literature. The authors say some conclusions about disagreement between evaluation setups do not depend on benchmark labels, but the available description does not detail how broadly that result holds.
Further Testing Across Research Fields
Ai2 has made the benchmark, data, code and pipeline available so other researchers can compare evaluation choices and build additional test sets. Further work could examine whether the reported instability appears in other disciplines, with different sources of ideas and under other judge designs, as well as whether particular methods make assessments more consistent.
The release material does not announce a specific follow-up study or timeline. For now, the benchmark offers a resource for testing novelty judges, while the study leaves open which combination of models, prompts and retrieval methods, if any, can support dependable assessments.
Key Questions
Which models did the study test?
The researchers tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, using absolute ratings and pairwise comparisons.
What does the reported 50-percentage-point shift mean?
In one pairwise experiment, changing a few prompt words shifted the models’ accuracy by more than 50 percentage points. The supplied description does not provide the full underlying results or a broader comparison baseline.
Did literature retrieval or more reasoning improve the results?
The authors report that adding literature retrieval or extra reasoning did not consistently improve judgments. That does not establish that those approaches never help in other settings.
Are the benchmark’s novelty labels definitive?
No. The human-idea labels rely on reviewer comments and selection criteria as evidence of relative novelty. The authors also say the generated ideas were treated as less novel using weak labels that were not individually verified.
What has Ai2 released?
Ai2 released the study, benchmark data, code and an automated pipeline intended to help researchers build additional evaluation sets. The supplied material does not specify a follow-up study or schedule.
Primary source: Hugging Face · via ThorstenMeyerAI.com
