Microsoft researchers have developed an artificial intelligence system that diagnoses patients by working through medical cases step by step, achieving more than twice the accuracy of general physicians in solving complex diagnostic problems, according to a study published on Thursday in the journal Science.
Unlike conventional medical AI systems, which typically analyse a patient’s symptoms, laboratory results and medical imaging together to produce a diagnosis, the new system follows a process closer to that used by doctors. It determines what information to gather, which examinations and tests to request, and whether further investigations are necessary as evidence emerges.
The system uses an “orchestrator” to organise its diagnostic process, drawing on a simulated panel of clinical experts that collaborate to solve medical cases. It begins by identifying relevant questions about the patient’s medical history and determining which findings from a physical examination are significant. It then selects appropriate tests, evaluates their results and decides whether additional investigations are needed, continuously updating its working hypotheses and plans.
According to the research leaders, this sequential approach enables the system to manage the diagnostic process rather than simply generate an answer from a collection of symptoms and test results.
In assessments involving more than 300 highly unusual and confusing medical cases, the AI system was more than twice as accurate at conducting the diagnostic work-up and identifying the condition as general physicians who used textbooks and the internet. The tool also outperformed human physicians when tested against scenarios drawn from a major database of emergency room admissions.
The researchers said conventional assessments of medical AI often rely on multiple-choice questions, which may not adequately reflect the complexity of diagnosing patients in clinical practice. Such tests can reward systems for selecting the correct answer from a limited set of options without requiring them to determine what information is missing or which investigations should be carried out.
“AI has become very good at getting a 100% pass rate on those,” said Dr Dominic King, corporate vice-president of the health unit at Microsoft AI, whose team is developing the technology.
“But of course, that’s not real-world. No patient ever came to see me with a nice summary.”
The researchers also found that diagnostic processes designed by the AI orchestrator cost less than those conducted by human doctors. The findings suggest that the system’s performance involves not only identifying the correct diagnosis but also determining which investigations are appropriate during the process.
“We’re not just measuring the accuracy of the systems. We’re also measuring how judiciously and carefully it goes through a diagnostic work-up,” said Harsha Nori, head of AI at Microsoft AI Health.
The researchers’ approach places emphasis on the decisions made along the way to a diagnosis, including the selection of questions and tests, rather than assessing accuracy solely by whether a final answer is correct. The study’s results cover the medical cases and emergency department scenarios used in the assessments; the source does not establish how the system would perform across routine clinical practice.
Microsoft’s next step will be to assess the technology in collaboration with Beth Israel Deaconess Medical Center in Boston. The researchers plan to test whether the tool can triage patients in emergency department waiting rooms, extending the evaluation beyond diagnostic case assessments to a hospital setting.
