Neurology: human brain beats artificial intelligence

Can artificial intelligence replace doctors in diagnosing neurological conditions? Large Language Models, which are used for AI chatbots such as ChatGPT and Gemini, are being increasingly studied for their potential in medicine. However, their efficacy in a real-life clinical context has not yet been tested adequately.

A team of researchers of the University of Milan and ASST Santi Paolo e Carlo (a hospital network in Milan) conducted an experimental study on the use of ChatGPT and Gemini in first neurological examinations, to assess the quality and reliability of their clinical interpretations. ChatGPT and Gemini are two of the most popular generative AI models, but are not specifically trained in medicine.

The study, which was published on the Journal of Medical Informatics Research, involved 28 anonymous patients under treatment at the Neurology Unit of the San Paolo hospital. Results showed that neurologists achieved a diagnostic accuracy of 75%, whereas ChatGTP reached 55% and Gemini 46%. Moreover, both AI models showed a tendency to over-prescribe diagnostic exams, in 17-25% of cases.

"This study proves that, despite their interesting potential as support tools, AI chatbots like ChatGPT and Gemini are not yet ready to make clinical decisions on their own, especially in complex fields like neurology", commented Natale Maiorana, neuropsychologist and first author of the study.

"It is important to remark that we used a free and non-tailored version of these tools, so they were not specifically trained in the medical field. What emerges is that artificial intelligence can be helpful, but must be integrated responsibly in clinical processes, with a strong human supervision", added Sara Marceglia, professor of Bioengineering at the University of Milan and corresponding author of the study.

The study provides a cautious but optimistic outlook: the authors claim that AI can be a useful support tool for future clinical practice, provided that it is adequately refined, customised and substantiated with rigorous clinical studies.

"Artificial intelligence looks promising, but to this day cannot replace human clinical evaluation. Our study paves the way for a new phase of research, one that will focus on how to integrate AI tools into neurology, and more generally into medicine, in an effective and safe way. Of course, medical students and residents will need to receive specific and certified training on the use of artificial intelligence over the course of their university studies", concluded Alberto Priori, Director of the Neurology Unit of the San Paolo hospital and the Aldo Ravelli Centre of the University of Milan, who is also the person who came up with the idea of this study. Another important aspect outlined by the study is that people with no medical background should be wary of relying on LLMs to self-assess symptoms and interpret diagnostic exams.