Development and Internal-external Validation of a Post-operative Mortality Risk Calculator for Pediatric Surgical Patients in Low- and Middle- Income Countries Using Machine Learning

Lauren Eyler Dang; Greg Klazura; Ava Yap; Doruk Ozgediz; Emma Bryce; Maija Cheung; Maíra Fedatto; Emmanuel A Ameh

doi:10.1016/j.jpedsurg.2024.161883

Development and Internal-external Validation of a Post-operative Mortality Risk Calculator for Pediatric Surgical Patients in Low- and Middle- Income Countries Using Machine Learning

J Pediatr Surg. 2024 Aug 29:161883. doi: 10.1016/j.jpedsurg.2024.161883. Online ahead of print.

Authors

Lauren Eyler Dang¹, Greg Klazura², Ava Yap³, Doruk Ozgediz³, Emma Bryce⁴, Maija Cheung⁵, Maíra Fedatto⁶, Emmanuel A Ameh⁷

Affiliations

¹ NIAID Biostatistics Research Branch, Bethesda, MD, USA.
² University of Illinois at Chicago Department of Surgery, Chicago, IL, USA; Loyola University Medical Center Department of Surgery, Maywood, IL, USA. Electronic address: greg.klazura@gmail.com.
³ UCSF Center for Health Equity in Surgery and Anesthesia, San Francisco, CA, USA.
⁴ Usher Institute of Population Health Sciences and Informatics at the University of Edinburgh, Edinburgh, Scotland, UK; KidsOR Research Team, Edinburgh, Scotland, UK.
⁵ Yale University Medical Center, New Haven, CT, USA.
⁶ KidsOR Research Team, Edinburgh, Scotland, UK.
⁷ National Hospital Abuja, Abuja, Nigeria.

PMID: 39317568
DOI: 10.1016/j.jpedsurg.2024.161883

Abstract

Background: The purpose of this study was to develop and validate a mortality risk algorithm for pediatric surgery patients treated at KidsOR sites in 14 low- and middle-income countries.

Methods: A SuperLearner machine learning algorithm was trained to predict post-operative mortality by hospital discharge using the retrospectively and prospectively collected KidsOR database including patients treated at 20 KidsOR sites from June 2018 to June 2023. Algorithm performance was evaluated by internal-external cross-validated AUC and calibration.

Findings: Of 23,905 eligible patients, 21,703 with discharge status recorded were included in the analysis, representing a post-operative mortality rate of 3.1% (671 mortality events). The candidate algorithm with the best cross-validated performance was an extreme gradient boosting model. The cross-validated AUC was 0.945 (95% CI 0.936 to 0.954) and cross-validated calibration slope and intercept were 1.01 (95% CI 0.96 to 1.06) and 0.05 (95% CI -0.10 to 0.21). For Super Learner models trained on all but one site and evaluated in the holdout site for sites with at least 25 mortality events, overall external validation AUC was 0.864 (95% CI 0.846 to 0.882) with calibration slope and intercept of 1.03 (95% CI 0.97 to 1.09) and 1.18 (95% CI 0.98 to 1.39).

Interpretation: The KidsOR post-operative mortality risk algorithm had outstanding cross-validated discrimination and strong cross-validated calibration. Across all external validation sites, discrimination of Super Learner models trained on the remaining sites was excellent, though re-calibration may be necessary prior to use at new sites. This model has the potential to inform clinical practice and guide resource allocation at KidsOR sites world-wide.

Type of study and level of evidence: Observational Study, Level III.

Keywords: Low resource settings; Mortality prediction algorithm; Pediatric surgery; Public health.