Comparative analysis of large language models in medical counseling: A focus on Helicobacter pylori infection

Abstract

Background: Large language models (LLMs) are promising medical counseling tools, but the reliability of responses remains unclear. We aimed to assess the feasibility of three popular LLMs as counseling tools for Helicobacter pylori infection in different counseling languages.

Materials and methods: This study was conducted between November 20 and December 1, 2023. Three large language models (ChatGPT 4.0 [LLM1], ChatGPT 3.5 [LLM2], and ERNIE Bot 4.0 [LLM3]) were input 15 H. pylori related questions each, once in English and once in Chinese. Each chat was conducted using the "New Chat" function to avoid bias from correlation interference. Responses were recorded and blindly assigned to three reviewers for scoring on three established Likert scales: accuracy (ranged 1-6 point), completeness (ranged 1-3 point), and comprehensibility (ranged 1-3 point). The acceptable thresholds for the scales were set at a minimum of 4, 2, and 2, respectively. Final various source and interlanguage comparisons were made.

Results: The overall mean (SD) accuracy score was 4.80 (1.02), while 1.82 (0.78) for completeness score and 2.90 (0.36) for comprehensibility score. The acceptable proportions for the accuracy, completeness, and comprehensibility of the responses were 90%, 45.6%, and 100%, respectively. The acceptable proportion of overall completeness score for English responses was better than for Chinese responses (p = 0.034). For accuracy, the English responses of LLM3 were better than the Chinese responses (p = 0.0055). As for completeness, the English responses of LLM1 was better than the Chinese responses (p = 0.0257). For comprehensibility, the English responses of LLM1 was better than the Chinese responses (p = 0.0496). No differences were found between the various LLMs.

Conclusions: The LLMs responded satisfactorily to questions related to H. pylori infection. But further improving completeness and reliability, along with considering language nuances, is crucial for optimizing overall performance.

Keywords: Helicobacter pylori; counseling tool; language model; reliability.