A new study published in The BMJ finds that AI chatbots are showing signs of cognitive impairment just like humans and this pattern is mostly seen in older models. The study is a great challenge to different studies and researches saying that AI is going to replace humans in medicine and teaching because now AI is showing signs of dementia and other cognitive problems like the ones seen in older humans. There are many studies that state that artificial intelligence will be able to do accurate medical diagnosis soon but this study says that it doesn't seem possible now that AI is showing cognitive decline.
Many AI models and LLMs like Google Gemini 1.0 and 1.5, OpenAI's ChatGPT-4 and 4o and Anthropic’s Claude 3.5 were assessed for the studies so the researchers could know which ones are showing cognitive decline. It was found that these AI models, especially the older ones, showed signs of cognitive impairment and performed the worst on tests which were done on them. Researchers used Montreal Cognitive Assessment (MoCA) on the models which are used to test early signs of dementia in older people. The maximum score of the test is 30 and includes questions related to language, attention, executive functions, memory and visuospatial skills, and a score above 26 is considered normal.
The LLMs were tested and were asked questions according to the test Gemini 1.0 scored the lowest with 16 out of 30. The highest score was achieved by GPT-4o (26 out of 30), followed by Claude and GPT-4 (25 out of 30). A practicing neurologist did all the tests and evaluated the results. The test showed that all AI models did the worst in visuospatial skills and executive tasks as well as a clock drawing test. Gemini models also didn't do well in delayed recall tasks where a sequence of five-word sentences is memorized and then recalled.
Most of the AI models which were assessed did well in language, naming, abstraction and attention. The researchers say that results of this test shows that AI models cannot perform perfectly in a clinical setting because they are showing some signs of weaknesses. So, this means that AI models aren't going to replace humans anytime soon because they are experiencing cognitive impairment and as long as this issue isn't solved, humans are going to take the lead. Researchers also suggested treating AI models with cognitive impairment the same way we treat human patients with similar issues.
Read next: Survey Predicts Global Warming, Economic Struggles, and AI Job Shifts in 2025
Many AI models and LLMs like Google Gemini 1.0 and 1.5, OpenAI's ChatGPT-4 and 4o and Anthropic’s Claude 3.5 were assessed for the studies so the researchers could know which ones are showing cognitive decline. It was found that these AI models, especially the older ones, showed signs of cognitive impairment and performed the worst on tests which were done on them. Researchers used Montreal Cognitive Assessment (MoCA) on the models which are used to test early signs of dementia in older people. The maximum score of the test is 30 and includes questions related to language, attention, executive functions, memory and visuospatial skills, and a score above 26 is considered normal.
The LLMs were tested and were asked questions according to the test Gemini 1.0 scored the lowest with 16 out of 30. The highest score was achieved by GPT-4o (26 out of 30), followed by Claude and GPT-4 (25 out of 30). A practicing neurologist did all the tests and evaluated the results. The test showed that all AI models did the worst in visuospatial skills and executive tasks as well as a clock drawing test. Gemini models also didn't do well in delayed recall tasks where a sequence of five-word sentences is memorized and then recalled.
Most of the AI models which were assessed did well in language, naming, abstraction and attention. The researchers say that results of this test shows that AI models cannot perform perfectly in a clinical setting because they are showing some signs of weaknesses. So, this means that AI models aren't going to replace humans anytime soon because they are experiencing cognitive impairment and as long as this issue isn't solved, humans are going to take the lead. Researchers also suggested treating AI models with cognitive impairment the same way we treat human patients with similar issues.
Read next: Survey Predicts Global Warming, Economic Struggles, and AI Job Shifts in 2025
