Artificial intelligence helps us in countless things and it also proves itself to be of great help in healthcare systems. But according to a new study by Stanford University and Harvard Medical School, artificial intelligence cannot talk to the patients in a way that can lead to their accurate diagnosis. In the medical field, communicating with patients is an important thing so the doctors and healthcare specialists can understand what patients are feeling but AI is showing limitations in its communication skills when it comes to diagnosing patients.
The researchers used Conversational Reasoning Assessment Framework for Testing in Medicine or CRAFT-MD to test AI how well it can work during communication in healthcare. It is important to note how interaction with patience and AI can help the system understand the diseases or problems patients are going through, especially now that many people are turning to AI like ChatGPT for their health concerns. The result of the test shows that even though many AI chatbots were successful in passing medical exams, they struggle when it comes to basic communication with the patients.
During medical diagnosis, doctors need to ask questions at the right time to the patients, and list down the information the patient gives in a proper way to find out what is actually wrong. As AI cannot communicate well, it raises alarms of how well AI can actually do in medical settings. The researchers tested on four prominent AI models and gave them 2,000 medical cases. It was found that GPT-4 gave correct diagnosis 82% of the times when it was given case summaries, but the correctness of diagnosis dropped to 63% when it had to interact with the patient to gather information. The correctness level dropped more when AI models had to go through open ended questions rather than multiple choice questions.
The areas where AI models struggled the most were in gathering critical information, case history of patients and not asking questions that could help connect all the conversation going on. One of the researchers, Daneshjou, said that he is interested in working with AI in a clinical setting but right now the results AI is showing aren't satisfactory. CRAFT-MD framework is a great way to test AI responses in healthcare fields as it can monitor real time conversations. The researchers suggest that developers should work more on communication skills of AI and make it capable so it can handle unstructured conversation.
Image: DIW-Aigen
Read next: AI Fact-Checking Results in Mixed Outcomes, Sometimes Boosting Misinformation and Distrusting Truthful News
The researchers used Conversational Reasoning Assessment Framework for Testing in Medicine or CRAFT-MD to test AI how well it can work during communication in healthcare. It is important to note how interaction with patience and AI can help the system understand the diseases or problems patients are going through, especially now that many people are turning to AI like ChatGPT for their health concerns. The result of the test shows that even though many AI chatbots were successful in passing medical exams, they struggle when it comes to basic communication with the patients.
During medical diagnosis, doctors need to ask questions at the right time to the patients, and list down the information the patient gives in a proper way to find out what is actually wrong. As AI cannot communicate well, it raises alarms of how well AI can actually do in medical settings. The researchers tested on four prominent AI models and gave them 2,000 medical cases. It was found that GPT-4 gave correct diagnosis 82% of the times when it was given case summaries, but the correctness of diagnosis dropped to 63% when it had to interact with the patient to gather information. The correctness level dropped more when AI models had to go through open ended questions rather than multiple choice questions.
The areas where AI models struggled the most were in gathering critical information, case history of patients and not asking questions that could help connect all the conversation going on. One of the researchers, Daneshjou, said that he is interested in working with AI in a clinical setting but right now the results AI is showing aren't satisfactory. CRAFT-MD framework is a great way to test AI responses in healthcare fields as it can monitor real time conversations. The researchers suggest that developers should work more on communication skills of AI and make it capable so it can handle unstructured conversation.
Image: DIW-Aigen
Read next: AI Fact-Checking Results in Mixed Outcomes, Sometimes Boosting Misinformation and Distrusting Truthful News
