Microsoft AI Outperforms Doctors in Complex Health Diagnoses

Microsoft has unveiled an artificial intelligence system that outperforms human doctors in diagnosing complex health conditions, paving the way for what the company calls a “path to medical superintelligence.”

The AI unit, led by British tech pioneer Mustafa Suleyman, has developed a system that mimics a panel of expert physicians tackling “diagnostically complex and intellectually demanding” cases. When paired with OpenAI’s advanced o3 AI model, the approach “solved” more than eight out of 10 case studies specifically chosen for the diagnostic challenge. In contrast, human doctors, who had no access to colleagues, textbooks, or chatbots, achieved an accuracy rate of only two out of 10.

Microsoft emphasized that the AI system is more cost-effective than using human doctors because it is more efficient at ordering tests. However, the company downplayed the job implications, stating that AI will complement, rather than replace, doctors. “Their clinical roles are much broader than simply making a diagnosis. They need to navigate ambiguity and build trust with patients and their families in a way that AI isn’t set up to do,” the company wrote in a blog post.

The term “path to medical superintelligence” raises the prospect of significant changes in the healthcare market. While artificial general intelligence (AGI) refers to systems that match human cognitive abilities, superintelligence denotes a system that exceeds human intellectual performance across the board.

Suleyman, the CEO of Microsoft AI, told the Guardian that the system will be operational within the next decade. “It’s pretty clear that we are on a path to these systems getting almost error-free in the next 5-10 years. It will be a massive weight off the shoulders of all health systems around the world,” he said.

Microsoft raised doubts about AI’s ability to score well in the United States Medical Licensing Examination (USMLE), a key test for obtaining a medical license in the US. The company argued that multiple-choice tests favor memorizing answers over deep understanding, which could overstate the competence of an AI model.

To address this, Microsoft developed a system that, like a real-world clinician, takes step-by-step measures such as asking specific questions and requesting diagnostic tests to arrive at a final diagnosis. For instance, a patient with symptoms of a cough and fever may require blood tests and a chest X-ray before a diagnosis of pneumonia is confirmed.

Suleyman’s team transformed more than 300 case studies from the New England Journal of Medicine (NEJM) into “interactive case challenges” to test the approach. Microsoft’s system used existing AI models, including those produced by OpenAI, Meta, Anthropic, Elon Musk’s Grok, and Google’s Gemini.

The company’s “diagnostic orchestrator” system works with a given model to determine what tests to order and what the diagnosis might be. This orchestrator imitates a panel of physicians, ultimately arriving at the diagnosis. When paired with OpenAI’s advanced o3 model, the system “solved” more than eight of 10 NEJM case studies, compared to a two out of 10 success rate for human doctors.

Microsoft stated that the system can wield a “breadth and depth of expertise” that goes beyond individual physicians because it can span multiple medical disciplines. The company added, “Scaling this level of reasoning – and beyond – has the potential to reshape healthcare. AI could empower patients to self-manage routine aspects of care and equip clinicians with advanced decision support for complex cases.”

However, Microsoft acknowledged that its work is not yet ready for clinical use. Further testing is needed on the “orchestrator” to assess its performance on more common symptoms.