A recent study has brought to light concerns regarding the free version of ChatGPT, suggesting that it may provide incorrect or incomplete responses, or even fail to answer at all when queried about medications. This could pose a potential risk to patients relying on OpenAI’s popular chatbot for crucial health-related information.
Research pharmacists conducted a study where they posed 39 drug-related questions to the free version of ChatGPT. Surprisingly, only 10 of the responses were deemed satisfactory based on the established criteria, indicating a significant room for improvement.
The study found that ChatGPT’s responses to the remaining 29 drug-related questions either did not directly address the queries, were inaccurate, incomplete, or a combination of these issues. Sara Grossman, the study’s author, emphasized the need for caution among patients and healthcare professionals when relying on ChatGPT for drug information, urging verification from trusted sources such as doctors, pharmacists, or government-based medication information websites.
Key Findings from the ChatGPT Study:
- Real questions were posed to Long Island University College of Pharmacy’s drug information service from January 2022 to April 2023.
- Pharmacists answered 45 questions in May, serving as a standard for accuracy against ChatGPT.
- ChatGPT did not directly address 11 questions, provided inaccurate responses to 10 questions, and offered wrong or incomplete answers to another 12 questions.
- For verification purposes, researchers requested references for each response from ChatGPT. Only eight responses included references, and each referred to non-existent sources.
- An example involved ChatGPT stating there was no drug interaction between the antiviral drug Paxlovid and the antihypertensive drug Verapamil. In reality, this combination causes an excessive lowering of blood pressure, potentially leading to unwanted side effects.
Despite the free version’s data limitation up to September 2021, and the fact that the paid version now incorporates real-time internet browsing, the study focused primarily on the free version. This choice aimed to replicate the user experience of the general population, considering ChatGPT’s widespread usage with around 1.7 billion visits worldwide in October 2023.
