Microsoft says its AI beat doctors at diagnosing hard cases

Microsoft AI said Thursday that its diagnostic system is “more than twice as accurate as physicians using textbooks and the internet” on medicine’s most complex cases, in a post announcing peer-reviewed results published in Science. The paper reports that the system, the Microsoft AI Diagnostic Orchestrator, or MAI-DxO, reached 83.2% accuracy on case records from The New England Journal of Medicine, against 36.1% for generalist physicians.

The study: The Science paper, “Sequential diagnosis with language models,” lists Harsha Nori as first author, with co-authors including Mustafa Suleyman, Dominic King and Eric Horvitz. It introduces the Sequential Diagnosis Benchmark, or SDBench, which turns 304 NEJM clinicopathological conference cases and 325 emergency department–to–admission encounters from MIMIC-IV-ED into interactive, stepwise encounters. Each case begins with a brief vignette, and the diagnostician, either an AI agent or a physician, may ask questions, order tests or commit to a diagnosis, the authors wrote. Performance was scored on accuracy and on the estimated cost of tests ordered.

How it works: Microsoft said MAI-DxO “operates like a panel of virtual physicians,” each proposing diagnoses, selecting questions or tests and challenging the emerging view over several rounds of debate, while weighing each test’s likely value against its cost.

The results: On the emergency cases, the paper reports 80.3% accuracy for the system against 46.8% for generalist physicians. Microsoft said physicians reached 22% on the NEJM cases without reference resources, making MAI-DxO about 3.7 times more accurate in that comparison. The company said one cost-focused configuration reached 81% on NEJM cases “at less than half the cost of comparative systems,” and that “no other human or AI approach was both more accurate and less costly.” Microsoft said the gains “persisted – and improved” when it repeated the evaluation with newer models, including GPT-5.5, without changing the orchestration method.

The limits: The authors wrote that their findings “were obtained in a controlled benchmark setting.” Microsoft said that “benchmarks matter, but the true test is performance in real clinical environments.”

Why it matters: SDBench is built to grade the path to a diagnosis as well as the answer, scoring each step and the cost of the tests ordered, and the system’s reported margins over physicians come on that harder, stepwise format. The widest gap Microsoft cites, 83.2% to 36.1% on the NEJM cases, is on the cases Microsoft calls medicine’s most complex; on the more typical emergency cases, the reported margin is smaller but still large. The numbers are the authors’ own and come from interactive benchmark encounters, which the paper itself describes as a controlled benchmark setting. Microsoft also says the approach held up when it swapped in newer models, without changing the orchestration method. By the company’s own account, the real test is how the system performs in clinical environments, which these results do not yet show.

What’s next: Microsoft said that through a new partnership with Beth Israel Deaconess Medical Center in Boston, it will run a prospective study in which patients interact with the system directly, with performance measured against physicians’ independent assessments. The company said it is open-sourcing SDBench for select NEJM cases, including the benchmark, gatekeeper, judge and cost-estimation code.