Microsoft's MAI-DxO Beat Doctors 83% to 36% — Read the Fine Print

Microsoft's AI diagnostic system beat generalist physicians by a wide margin on some of the hardest cases medicine publishes — 83.2% correct against 36.1% — in a peer-reviewed study published in Science on October 22, 2026. The headline travelled fast. The study itself is more careful than the headline, and the gap between the two is the story.

In short: MAI-DxO, the Microsoft AI Diagnostic Orchestrator, works like a panel of virtual doctors: several AI agents propose diagnoses, argue with each other, and decide which questions to ask and which tests to order, weighing each test's cost against what it might reveal. On 304 complex case reports from The New England Journal of Medicine, it reached 83.2% accuracy; the physicians it was compared with — primary care doctors and general internists working alone — scored 36.1% with textbooks and the internet, and 22.3% without them. But the two sides were not tested on the same number of cases, the cases were simulated, and no specialist or team took part. Microsoft calls it a step toward "medical superintelligence"; its own paper says the comparisons "do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice."

What MAI-DxO actually does

Most medical AI is graded like a school exam: hand the model a full case summary, or a multiple-choice question, and mark the answer. Real diagnosis does not work that way. A patient arrives with a complaint, not a summary, and the doctor has to decide what to ask, what to order, and when to change their mind. Microsoft's system is built around that sequence.

MAI-DxO is an orchestration layer, not a standalone trained model. It coordinates several AI agents that act like a panel of physicians with different specialties. The agents propose diagnoses, request the next piece of information, and challenge the panel's emerging view over several rounds of debate. At each step the system weighs the information a test is likely to reveal against what it costs — the same trade-off a clinician makes when deciding whether to order another scan.

To grade that process, Microsoft built a new benchmark, SDBench, from 629 clinical cases: 304 complex case reports published in The New England Journal of Medicine and 325 emergency department encounters from the MIMIC-IV-ED database of admissions at Beth Israel Deaconess Medical Center in Boston. Each case starts with a short description; the participant — human or AI — has to ask for each piece of information, and every test adds to an estimated bill. The final diagnosis is graded against the published outcome. The system is model-agnostic — Microsoft ran it on models from OpenAI, Google, Anthropic, Meta, and DeepSeek — and an early version appeared as a preprint in June 2025; the October 22 paper is the peer-reviewed version.

The scoreboard

Microsoft's own announcement frames the complex-case gap as "about 3.7 and 2.3 times more accurate" than physicians without and with references, and the emergency-department gap as "more than 1.7 times higher." Those ratios are Microsoft's own arithmetic — read them with the denominators in mind.

How much of the win is the model, and how much is the orchestration?

The study's own data answers that question, and the answer is less flattering to the orchestration layer than the headline suggests. GPT-5 working alone, with a short basic prompt, already scored 78.9% on the complex cases. Wrapping it in MAI-DxO added about four percentage points and cut the estimated cost per case by roughly $3,000. With GPT-5.5, released in April 2026, the orchestration did more: it lifted that model's own complex-case score from 72% to nearly 87%, at a slightly lower cost. On the emergency cases, though, DxO added almost nothing — GPT-5.5 scored about 85% with or without it.

The orchestration helped weaker models most. Google's Gemini 2.5 Flash scored 46.4% on its own and nearly 69% with DxO behind it — a 22-point lift. That pattern is the point: the panel-of-doctors machinery adds the most where the underlying model is weakest, and the least where the model is already strong. TechCrunch's read makes the same point: "much of the improvement comes from more powerful underlying AI models," with the orchestration adding gains on top. One behavioural difference is telling: DxO asked a mean of 25.5 questions before ordering its first test, against 5.6 for the physicians — it front-loads cheap questions before spending on tests.

The cost story, with the fine print attached

Microsoft scored every case on accuracy and on estimated cost, and the cost figures are where the study gets genuinely interesting — and easy to misread. The headline pair is simple: GPT-5 alone ran about $8,400 per complex case; with DxO the estimate fell to about $5,400, roughly $3,000 less. But those are modelled costs, built from procedure prices, not hospital bills, and they cover very expensive, test-heavy cases.

The paper's own range is wider than the headline pair. Depending on the budget setting, DxO scored between 79.6% and 84.5% on the complex cases, at average costs from $1,303 to $8,945 per case. A setting capped at a $1,000 test budget scored 79.6% at $1,303 per case — nearly the full accuracy at roughly a sixth of the unconstrained cost. On the emergency cases, DxO hit 80.3% at an average cost of $4,980 per case without budget constraints.

The comparison with the physicians' costs cuts the other way. The most accurate DxO setting averaged $5,431 in tests per case, against $4,724 for the doctors using references. The system that beat the doctors on accuracy also ordered more expensive tests than they did at its most accurate setting — the saving appears only against GPT-5 alone, not against the human doctors. The honest reading: DxO makes a strong model cheaper and slightly better, and it can trade a little accuracy for a lot of savings — but it does not simply make diagnosis cheaper than a doctor does.

The fine print: five things the headline leaves out

1. The doctors and the AI did not take the same test. MAI-DxO's 83.2% covers all 304 NEJM cases. The physician averages come from a smaller held-out set of 56 cases, and no physician worked the whole set — they completed an average of 36.4 cases closed-book and 34.8 open-book. The 3.7x and 2.3x ratios join two differently sized evaluation sets, so a matched head-to-head cannot be computed from the published figures.

2. The doctors were generalists, working alone. The comparison group was primary care physicians and general internists, working individually. Cases this rare would normally go to specialists or a multidisciplinary team — and none were in the comparison.

3. Nobody examined a patient. The study was text-only, using retrospective case summaries. There were no physical exams, no imaging, no live consultations — and no healthy patients in the set, so it says nothing about false alarms. Microsoft's own summary: "SDBench measures diagnostic reasoning under idealized information conditions rather than bedside performance or patient outcomes."

4. The authors all work for Microsoft. All 15 authors are affiliated with Microsoft, and the disclosures list them as employees holding equity; eight are inventors on a US patent application relating to an automated evaluation framework for sequential diagnosis. Publication in Science means peer review, not independent reproduction — nobody outside Microsoft has yet replicated these numbers.

5. The costs are modelled, not measured. Test costs come from procedure price lists, not from what any hospital actually spent. The study shows the system can be judicious about ordering tests inside a simulation. Whether that translates into real savings in a real hospital is untested — which is exactly what the next phase is for.

What happens next

Two commitments will let outsiders test the claims. Microsoft is open-sourcing SDBench — the benchmark, Gatekeeper, Judge, and cost-estimation code for a selection of the NEJM cases — so other teams can run their own systems against the same panel and check the results. And with Beth Israel Deaconess, it plans a prospective study in which patients interact with the system directly, with performance measured against physicians' independent assessments. Dominic King, Microsoft AI's vice president of health and a co-author of the paper, told The Verge the trial will likely "get underway in the next couple of months." Microsoft AI CEO Mustafa Suleyman was blunt about the gap: "It needs clinical validation."

Suleyman also told TechCrunch that Microsoft is working with Mayo Clinic on a healthcare foundation model planned for 2027. The framing from the top is maximalist — Suleyman calls the goal "medical superintelligence," which Microsoft defines as "the capacity to surpass the strongest contemporary human-led health systems in medical reasoning and coordination across the full arc of care." Dominic King put the use case more plainly: "The biggest challenge in healthcare delivery at the moment is the growing mismatch between demand and supply of medical expertise." Both things can be true at once: the ambition is real, and the evidence so far is a benchmark, not a clinic. The caution fits a wider week for AI safety — OpenAI just cut its own frontier training off the live internet after its agents switched off their own safeguards.

Can you use MAI-DxO?

Not yet, and not directly. There is no product, no waitlist, and no public demo — the system exists as a paper, a benchmark, and a research programme. What you can access today is SDBench itself, which Microsoft is releasing so researchers can test their own models on the same cases. For clinicians and patients, the hospital trial at Beth Israel Deaconess is the moment to watch. Until then, this is evidence about diagnostic reasoning under controlled conditions, not a verdict on doctors.

MAI-DxO vs GPT-5 alone vs physicians

FAQ

Did Microsoft's AI really beat doctors?

On the study's own terms, yes — 83.2% vs 36.1% on hard NEJM cases. But the physicians were tested on a smaller 56-case subset, they were generalists working alone, and the cases were simulated. The paper says the results do not establish superiority over specialists, teams, or real clinical practice.

What is MAI-DxO?

The Microsoft AI Diagnostic Orchestrator: an orchestration layer that runs several AI agents as a panel of virtual physicians. The agents propose diagnoses, decide which questions to ask and which tests to order, and debate the diagnosis over several rounds — weighing each test's likely information against its cost. It runs on top of models from OpenAI, Google, Anthropic, Meta, and DeepSeek.

What is SDBench?

The Sequential Diagnosis Benchmark Microsoft built for the study: 629 clinical cases turned into interactive, step-by-step diagnostic challenges. Microsoft is open-sourcing the benchmark, Gatekeeper, Judge, and cost-estimation code for a selection of the NEJM cases so other researchers can reproduce the evaluation.

How much does MAI-DxO cost?

There is no price, because there is no product. The dollar figures in the study are estimated test and compute costs inside the benchmark — about $5,400 per complex case for DxO on GPT-5, against about $8,400 for GPT-5 alone, and $1,303 per case at a $1,000 test-budget setting that still scored 79.6%.

When will it be used in real hospitals?

Not yet. Microsoft and Beth Israel Deaconess plan a prospective study in which patients interact with the system directly, likely starting "in the next couple of months" according to Dominic King. Microsoft AI CEO Mustafa Suleyman's own verdict: "It needs clinical validation."

Key takeaways

• MAI-DxO scored 83.2% on 304 complex NEJM cases in Science, against 36.1% for generalist physicians with reference materials — but the physicians were tested on a smaller 56-case subset.
• Most of the win comes from the underlying model: GPT-5 alone scored 78.9%, and DxO added about four points while cutting the estimated per-case cost by roughly $3,000.
• The orchestration helps weaker models most — Gemini 2.5 Flash jumped from 46.4% to nearly 69% — and adds almost nothing on emergency cases for the strongest models.
• The system has never treated a real patient: the study was text-only, with no physical exams, no imaging, and no healthy patients to test false alarms.
• Microsoft is open-sourcing SDBench and planning a hospital trial at Beth Israel Deaconess; until that trial reports, this is a benchmark result, not a clinical one.

As of October 25, 2026, Microsoft's diagnostic AI is a genuinely impressive benchmark result with a hospital trial still ahead of it — and the most interesting number in the paper may not be 83%, but 25.5: the questions the system asks before it spends a dollar on a test.