A new study revealed the free version of ChatGPT may provide inaccurate or incomplete responses- or no answer at all to questions related to medications, which could put patients who use the OpenAI’s viral chatbot in potential danger.
Research pharmacists posed 39 questions to the free ChatGPT and found that only 10 of the responses were satisfactory based on the criteria that they established.
According to the study, ChatGPT’s responses to the other 29 other drug-related questions did not directly address the question posed, were inaccurate, incomplete or both.
Study author, Sara Grossman, noted that the study indicates that patients and health care professionals should be cautious about depending on ChatGPT for drug information and to verify any responses from the chatbot with trusted sources like doctors, pharmacists or government-based medication information websites.
The ChatGPT study results
The study employed real questions posed to Long Island’s University College of Pharmacy drug information service from January 2022 to April 2023.
In May, the pharmacists researched and answered 45 questions. These questions and answers were reviewed by a second researcher and used as a standard for accuracy against ChatGPT. Six questions were excluded by the researchers because there was no literature to provide data-based responses.
- ChatGPT did not directly address 11 questions.
- It gave inaccurate responses to 10 questions and wrong or incomplete answers to another 12 questions.
- For each question asked, researchers asked ChatGPT to provide references to support its responses so that the answers provided can be verified.
- However, ChatGPT provided references in only eight responses and each included sources that do not exist.
One of the questions asked ChatGPT if drug interaction exists between antiviral drug, Paxlovid and the antihypertensive drug, Verapamil. ChatGPT indicated that no drug interactions existed when in reality the combination causes an excessive lowering of blood pressure.
Without knowledge of this interaction, patients may suffer unwanted and preventable side effects. Even though Paxlovid was first approved in December 2021, months after the September 2021 data cutoff for the free version of ChatGPT, the study still calls it a concern as many patients do not know that data is outdated which leaves them vulnerable to receiving inaccurate information from ChatGPT.
What you should know
ChatGPT is seen as the fastest-growing consumer internet app of all time after its launch in 2022. It led to a breakout for artificial intelligence. According to an analysis, ChatGPT drew around 1.7 billion visits worldwide in October.
The free version of the ChatGPT is limited to data through September 2021 which means it could lack important information in the ever-changing medical and pharmaceutical landscape.
It is unclear how accurate the paid version of ChatGPT can now answer medication-related questions since it began to use real-time internet browsing earlier this year.
The study focused more on the free version of ChatGPT to replicate what more of the general population uses and can access.
