Two studies published in Nature Health on 6 February reveal that cheap, readily available AI chatbots, or large language models (LLMs), can enhance clinical decision-making and even outperform trained doctors in controlled, real-world settings with limited resources. Researchers in Rwanda and Pakistan tested the tools in environments where physician availability is low, patient loads are high, and health systems face significant constraints.
In Rwanda, LLMs were evaluated for their ability to answer clinical questions posed by community health workers, who often serve as the first point of care for patients. The study compared responses from five LLMs to answers provided by local clinicians for roughly 500 commonly asked medical questions. Across 11 metrics, including accuracy, alignment with medical consensus, comprehension of patient queries, and risk of harm, the AI consistently outperformed human respondents. The models were also able to provide reliable answers in Kinyarwanda, the country’s national language, highlighting their adaptability to local contexts. Beyond accuracy, researchers noted that LLM responses cost less than half a cent per query, compared with $3.80 to $5.43 for clinician-generated answers, and were available 24/7, providing constant support to health workers.
In Pakistan, a randomized controlled trial tested whether LLMs could assist physicians in diagnosing patients more accurately. Fifty-eight licensed doctors received 20 hours of training on how to use an AI chatbot safely and critically, including guidance on avoiding errors or hallucinations. Physicians with access to GPT-4o achieved a mean diagnostic reasoning score of 71%, compared with 43% for those relying on conventional resources such as PubMed or Internet searches. A secondary analysis found that the AI alone performed better than physicians using it in some cases, though doctors outperformed the model in 31% of instances, typically where contextual “red flags” were involved.
Experts emphasized that while the studies demonstrate the potential of AI to support health-care delivery in low-resource settings, caution is needed. Concerns remain about bias in AI outputs, patient data privacy, and the risk of clinicians uncritically accepting flawed AI recommendations. Both studies’ authors stressed that LLMs are meant to complement, not replace, human clinicians, and highlighted the importance of training health workers to use AI effectively.
Researchers also warned that improving early diagnosis with AI could strain already limited health-care systems if treatment capacity does not expand in parallel. Nonetheless, the findings underline the potential for AI to strengthen access to medical care, reduce diagnostic errors, and improve outcomes in regions where human resources are scarce, while ensuring that tools are culturally and linguistically appropriate for the populations they serve.
