AI chatbots may seem medical–book smart but their grades falter when interacting with real people.
In the lab, AI chatbots could identify medical issues with 95 percent accuracy and correctly recommend actions such as calling a doctor or going to urgent care more than 56 percent of the time. When humans conversationally presented medical scenarios to the AI chatbots, things got messier. Accuracy dropped to less than 35 percent for diagnosing the condition and about 44 percent for identifying the right action, researchers report February 9 in Nature Medicine.