The Lancet The barrage of misinformation in the field of health care is persistent and growing. The advent of artificial intelligence (AI) and large language models (LLMs) in health care has expedited the increase in misinformation, and LLMs are susceptible to false output if they are trained on incorrect health-care information. This risk of misinformation is especially true for LLMs trained on vast datasets of information originating from online sources and can be particularly difficult to navigate when developers do not disclose the databases used to train such tools. Incorrect medical advice generated from LLMs have serious consequences for patients. How can we quantify and ultimately reduce the misinformation caused by LLMs to ensure better patient health outcomes?
This month in The Lancet Digital Health, Mahmud Omar and colleagues present a benchmark study testing the susceptibility of general-purpose LLMs, as well as LLMs specifically trained for medical use, to medical misinformation embedded in prompts which are the inputs users provide to LLMs as instructions. 20 LLMs were evaluated using 3·4 million prompts from a collection of hospital discharge notes, simulated clinical vignettes, and social media posts, all containing fabricated medical information. Performance in two tasks, detecting misinformation in a recommendation and identifying a logical fallacy (a flaw in the LLM’s reasoning process), varied by model. Interestingly, the popular general-purpose GPT-4o model was both the least susceptible and most accurate at fallacy detection; furthermore, the medical fined-tuned tools performed consistently worse than the general tools. This study shows that LLMs are vulnerable to misinformation, particularly when it is conveyed in an authoritative tone. The study also represents the first large-scale, structured benchmarking exercise that assesses how LLMs manage prompts containing medical misinformation. The study’s strengths lie in testing a wide range of models, including general-purpose and medical tools, as well as both open-source and proprietary models. However, it is important to acknowledge limitations such as the text-only format involved, which does not reflect the multimodal real-world and fabricated medical data that could be fed into LLMs. Furthermore, the downstream effects on clinical impact, such as health outcomes or user trust in the tools, were not investigated.