🔍 Read the full analysis: Microsoft’s AI Beats Doctors Claim: What The Paper Actually Tested on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A Science paper published Oct. 22 reports that Microsoft’s MAI-DxO system reached 83.2% accuracy on 304 difficult, historical medical cases, compared with 36.1% for generalist physicians using reference materials. The test simulated diagnostic decisions from text records; it did not assess patient outcomes or compare the system with specialists or physician teams.
Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 difficult medical cases, compared with 36.1% for generalist physicians given textbooks and internet access, according to a peer-reviewed Science paper published Oct. 22. The result measures performance on simulated, historical case records—not diagnosis or treatment of real patients—and the study compared the system with individual generalists rather than specialists or clinical teams.
The paper, Sequential diagnosis with language models, tested whether a system could work through a case by requesting information over multiple rounds. Each case began with a short summary. The AI or physician could ask for further details or tests from a simulated gatekeeper holding the full record; tests carried estimated costs. The benchmark scored both the final diagnosis and the expense of gathering information.
Microsoft reported 83.2% accuracy on 304 complex New England Journal of Medicine cases. Physicians scored 36.1% when allowed textbooks and internet resources, and 22.3% without those resources. On a separate set of 325 emergency department cases, the reported scores were 80.3% for the system and 46.8% for physicians. These are results reported by Microsoft’s research team; the paper’s publication in Science establishes peer review, but does not by itself show that outside researchers have replicated them.
MAI-DxO coordinates a language model through a simulated panel of virtual physicians that suggest diagnoses, request questions or tests, and challenge one another. The paper’s results also point to the underlying model’s contribution: GPT-5 alone scored 78.9% on the complex-case benchmark, according to Microsoft’s chart data. The source material says the orchestration approach added about four percentage points and lowered estimated testing costs, but those modeled costs are not evidence of savings in hospitals.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
Why Clinical Comparisons Matter
The findings matter because diagnostic AI is often presented as a potential aid to clinicians, while a headline comparison can sound like evidence that it outperforms doctors in practice. This study supports a narrower conclusion: in a controlled benchmark built from difficult cases, the tested system achieved higher diagnostic scores than the participating generalists under the study’s rules. It does not establish that MAI-DxO improves care, reduces missed diagnoses, or is safe and effective in routine clinical use.
The comparison also affects how readers should interpret the gap. The AI operated as a coordinated group of agents, while the physician participants worked individually. The case collection focused on challenging published cases, and the source says it contained no healthy patients or harmless explanations. That means the test could not measure how often the system would raise unnecessary alarms in a broader patient population. Results from such a benchmark may be relevant to future research, but they do not settle how the technology would perform across ordinary clinics, with full patient records and clinical accountability.
As an affiliate, we earn on qualifying purchases.
How the Benchmark Was Built
The study’s Sequential Diagnosis Benchmark was designed to resemble one part of diagnostic work: choosing what information to seek after an initial presentation. Unlike a test that gives a model every fact at once, this setup charged a notional price for each additional test or detail. That design lets researchers examine both diagnostic answers and information-gathering choices, but it remains a simulation using text cases, not a live clinical encounter involving examination, imaging, or patient interaction.
Some methodological details in the supplied source come from the earlier, non-peer-reviewed 2025 preprint, and may not match every detail in the final Science paper. In that version, a model judged answers against the published diagnosis, while additional test results absent from a case record could be generated synthetically. Test prices were estimates based on billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit charge. The authors described the estimates as approximations. The preprint also said physicians re-graded a sample of answers and showed good, but imperfect, agreement with the automated judge.
The physician comparison in the preprint involved 21 U.S. and UK generalists: 17 in primary care and four hospital generalists, with a median of 12 years’ experience. They did not use search engines or language models in the closed-book arm. The final paper reports an additional open-book comparison, reflected in the 36.1% figure. The source notes that the AI was evaluated across all 304 complex cases, while physicians in the preprint were assessed on a smaller held-out subset, a difference readers should keep in mind when interpreting the headline comparison.
“The system was described as 1.7 times as accurate as physicians on emergency department cases and more than twice as accurate as physicians using textbooks and the internet on complex cases.”
— Microsoft, in its blog post, “Building Medical Superintelligence”
As an affiliate, we earn on qualifying purchases.
What the Scores Cannot Show
The study does not show whether MAI-DxO can improve outcomes, avoid harmful delays, or work reliably with real patients. It also does not compare the system with specialists, multidisciplinary teams, or doctors using their full clinical workflows. The supplied source says the benchmark cases were unusually difficult and that the NEJM set lacked healthy patients and benign explanations, leaving false alarms unmeasured.
Other limits include the use of a model to grade answers, synthetic results for some tests not present in records, and estimated rather than observed test costs. Details available from the preprint may differ from the final peer-reviewed version, whose full text is behind a paywall according to the source. The paper’s authors are all listed with Microsoft affiliations, and the source reports no independent reproduction of the results. How performance transfers to routine care remains unknown.
As an affiliate, we earn on qualifying purchases.
A Study Alongside Patient Care
The next stated milestone is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source, consenting patients are expected to interact with the system alongside their normal care. That setting could provide evidence about how the tool behaves in real clinical workflows, though the available material does not give a start date, study size, protocol, or planned outcome measures.
Until results from that work or independent evaluations are available, the Science paper should be read as a benchmark report rather than proof that the system is ready to replace or outperform clinicians in hospitals. Key questions include whether the AI can handle ordinary cases as well as rare, difficult ones, how often it produces false alarms, and whether its suggestions help clinicians make safer decisions without adding cost or delay.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Microsoft’s study find?
Microsoft reported that MAI-DxO reached 83.2% accuracy on 304 difficult NEJM cases. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without them, according to the paper’s reported results.
Did the study show that the AI is better than doctors in hospitals?
No. It compared a simulated AI system with individual generalist physicians on historical case records. It did not test patient outcomes or compare the AI with specialist doctors or clinical teams.
Was the paper peer-reviewed?
Yes. The source identifies the paper, Sequential diagnosis with language models, as published in Science on October 22, 2026. Peer review does not mean the results have been independently reproduced.
What is MAI-DxO?
MAI-DxO is Microsoft’s system for coordinating a language model through a panel of virtual physician agents. They propose diagnoses and request information or tests over multiple rounds.
What happens next?
The source says a prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would use the system alongside their usual care. Details such as the timetable and study measures have not been provided in the supplied material.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
