Article of the Week: Sepetmber 8

Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum

Author

Marina Pulini-Franks MD
September 08, 2023

Comparing Physician and Chatbot Responses to Patient Questions

This cross-sectional study evaluates the ability of an artificial intelligence chatbot to provide quality and empathetic responses to patient questions.

jamanetwork.com/journals/jamainternalmedicine/article-abstract/2804309

Main Point

The study evaluates the efficacy of an AI chatbot, ChatGPT, in responding to patient questions compared to physician responses in an online forum, finding that the chatbot's responses were preferred by evaluators due to their higher quality and empathy, suggesting potential benefits for enhancing patient communication and clinical workflows.

5 Salient Points

  • AI Chatbot Performance: The study examines ChatGPT's ability to address patient inquiries, highlighting the increased adoption of virtual healthcare due to the COVID-19 pandemic and the subsequent surge in electronic patient messages, leading to concerns about clinician burnout and quality of responses.

  • Messaging Volume Impact: With rising messaging volumes, the risk of unanswered or unhelpful patient queries increases. While some messages are simple, others require more time and skill to address, indicating the need for effective strategies to manage this burden without limiting access to quality healthcare.

  • AI Assistant Potential: The study introduces ChatGPT as a new generation AI technology, noting its widespread recognition for producing human-like text. Although not designed for healthcare, the study explores the chatbot's ability to respond empathetically and effectively to patients' healthcare queries.

  • Study Design: Using publicly available data from an online forum, the study compares responses generated by ChatGPT with those from physicians. A total of 195 exchanges were evaluated, and the responses were assessed for quality and empathy by licensed healthcare professionals.

  • Key areas of focus for AI implementation in healthcare include digitalization of information, communication systems, patient empowerment, and system automation. These areas have improved patient safety, increased nurse-to-patient ratios, and greater efficiency in hospital processes.

🎯Supportive Figures

Example Questions with Physician and Chatbot Response (continued)

Example Questions with Physician and Chatbot Response

Comparison of Empathy Ratings for Chatbot and Physician Responses