Artificial intelligence (AI) is transforming the way science is conducted—but not without raising serious concerns. According to a recent report in Nature, the proportion of scientific papers using AI across 20 different fields—ranging from economics to psychology—quadrupled between 2012 and 2022. While this growth fuels hopes that AI could speed up the pace of discovery, some researchers caution that the rush to adopt AI may be outpacing the scientific rigor required to use it responsibly.
Despite increased funding, publications, and personnel, the rate of fundamental scientific breakthroughs appears to be slowing. AI is seen as a potential remedy—capable of sifting through massive data sets, identifying patterns, and even predicting future outcomes. However, Nature highlights growing evidence that the widespread application of AI in science is fraught with technical pitfalls, methodological flaws, and philosophical challenges.
One of the biggest dangers lies in AI’s “black-box” nature—its predictions often lack transparency and are prone to critical errors, especially when off-the-shelf tools are used by researchers without sufficient training. A common issue is data leakage, where models are improperly trained using information from their evaluation datasets, leading to misleadingly high performance. Nature reports that this problem has already affected machine-learning studies in over 30 fields, including psychiatry, molecular biology, and even cybersecurity.
A stark example emerged during the COVID-19 pandemic. Hundreds of AI models claimed to diagnose the virus via X-rays or CT scans, but a systematic review found that the vast majority of these studies were flawed. In several cases, the models were inadvertently trained to differentiate between adults and children, not COVID-positive and COVID-negative patients, calling into question the validity of their findings.
The article goes beyond technical concerns to highlight an even deeper problem: the difference between producing more scientific findings and achieving true scientific understanding. Unlike engineering, where predictive accuracy may suffice, the natural sciences demand explanations that connect observations to underlying theories. Overreliance on predictive models, the authors argue, risks mistaking technological output for genuine knowledge.
“Today, AI excels at producing the equivalent of epicycles,” Nature warns, referencing outdated but precise models used to predict planetary movement before the heliocentric model revolutionized astronomy. In the same way, AI may reinforce flawed scientific paradigms by offering increasingly accurate—but ultimately superficial—predictions.
To address these issues, researchers advocate for multiple remedies, including better training for scientists in AI and machine learning, clearer guidelines, stronger incentives for reproducibility, and funding strategies that prioritize quality over volume. Initiatives such as REFORMS—a consensus-based checklist to guide machine-learning research—are already being developed.
Ultimately, the article urges a reset in expectations. While AI can undoubtedly enhance research workflows and uncover hidden patterns, its use in scientific discovery must be tempered with caution, critical thinking, and a deeper understanding of its limitations. As Nature aptly concludes, “In most scientific fields, AI is unlikely to be the solution to concerns about slowing progress. In fact, if used carelessly, it may become part of the problem.”
