Performance of ChatGPT and Google Translate for Pediatric Discharge Instruction Translation

Abstract

Background and objectives: Patients who speak languages other than English face barriers to equitable healthcare delivery. Machine translation systems, including emerging large language models, have the potential to expand access to translation services, but their merits and limitations in clinical practice remain poorly defined. We aimed to assess the performance of Google Translate and ChatGPT for multilingual translation of pediatric discharge instructions.

Methods: Twenty standardized discharge instructions for pediatric conditions were translated into Spanish, Brazilian Portuguese, and Haitian Creole by professional translation services, Google Translate and ChatGPT-4.0, and evaluated for adequacy (preserved information), fluency (grammatical correctness), meaning (preserved connotation), and severity (clinical harm), along with assessment of overall preference. Domain-level ratings and preferred translation source were summarized with descriptive statistics and compared with professional translations.

Results: Google Translate and ChatGPT demonstrated similar domain-level ratings to professional translations for Spanish and Portuguese. For Haitian Creole, compared with both Google Translate and ChatGPT, professional translations demonstrated significantly greater adequacy, fluency meaning, and severity scores. ChatGPT (33.3%, P < .001) and Google Translate (23.3%, P = .024) contained more potentially clinically significant errors (severity score ≤3) for Haitian Creole than professional translations (8.3%). Professional Haitian Creole (48.3%) and Portuguese (43.3%), but not Spanish (15%), translations were most frequently preferred among translation sources.

Conclusions: Machine translation platforms have comparable performance to professional translations for Spanish and Portuguese but shortcomings in quality, accuracy, and preference persist for Haitian Creole. Diverse multilingual training data are needed, along with regulations ensuring safe and equitable applications of machine translation in clinical practice.