Microsoft’s AI Diagnoses Diseases 4 Times More Accurately Than Human Doctors

Microsoft has taken a significant step toward medical superintelligence, according to Mustafa Suleyman, CEO of the company’s artificial intelligence division. The tech giant claims that its powerful new AI tool, MAI-DxO, can diagnose diseases four times more accurately and at a significantly lower cost than a panel of human physicians.

The experiment tested MAI-DxO's ability to diagnose patients by mimicking the work typically done by a human doctor. The Microsoft team used 304 case studies from the New England Journal of Medicine to create the Sequential Diagnosis Benchmark. A language model broke down each case into a step-by-step process that a doctor would follow to reach a diagnosis.

MAI-DxO, which queries several leading AI models including OpenAI’s GPT, Google’s Gemini, Anthropic’s Claude, Meta’s Llama, and xAI’s Grok, outperformed human doctors with an accuracy of 80 percent compared to the doctors’ 20 percent. It also reduced costs by 20 percent by selecting less expensive tests and procedures.

“This orchestration mechanism—multiple agents working together in a chain-of-debate style—is what's driving us closer to medical superintelligence,” Suleyman explains.

The company poached several Google AI researchers to help with the project, reflecting the growing competition for top AI talent in the tech industry. Suleyman, a former Google executive, emphasizes the importance of this collaboration.

AI is already widely used in parts of the US healthcare industry, such as radiology, where it helps interpret scans. The latest multimodal AI models have the potential to serve as more general diagnostic tools. However, the use of AI in healthcare raises concerns about bias in training data, which can be skewed toward certain demographics.

Microsoft has not yet decided whether to commercialize the technology but could integrate it into Bing for user diagnostics or develop tools to assist medical experts. “What you'll see over the next couple of years is us doing more and more work proving these systems out in the real world,” Suleyman says.

The project is part of a growing body of research showing how AI models can diagnose diseases. Both Microsoft and Google have published papers demonstrating that large language models can accurately diagnose ailments when given access to medical records.

The new Microsoft research stands out because it more accurately replicates the way human physicians diagnose diseases—by analyzing symptoms, ordering tests, and performing further analysis until a diagnosis is reached. Microsoft describes the combination of several frontier AI models as “a path to medical superintelligence.”

The project also suggests that AI could help lower healthcare costs, a critical issue in the US. “Our model performs incredibly well, both in reaching the diagnosis and doing so cost-effectively,” says Dominic King, a vice president at Microsoft involved in the project.

David Sontag, a scientist at MIT and co-founder of Layer Health, a startup that builds medical AI tools, finds the work exciting. Sontag notes that the research is important not only because it closely mirrors how physicians operate but also because it rigorously addresses potential issues with the methodology. “That’s what makes this paper strong,” he says.

However, Sontag cautions that Microsoft’s findings should be treated with some skepticism because the doctors in the study were not allowed to use additional tools to help with their diagnoses, which may not reflect real-world conditions. He adds that it remains to be seen whether the AI system would significantly reduce costs in practice.

Eric Topol, a scientist at the Scripps Research Institute, finds the report impressive because it tackles highly complex cases for diagnosis. He adds that the potential for AI to reduce medical costs is a novel and important aspect of the research.

Both Topol and Sontag agree that the next step in validating the potential of Microsoft’s system would be demonstrating its effectiveness in a clinical trial comparing its results with those of real doctors treating real patients. “Then you can get a very rigorous evaluation of cost,” Sontag says.