Objective: To develop and validate a predictive algorithm that identifies pediatric patients at risk of asthma-related emergencies, and to test whether algorithm performance can be improved in an external site via local retraining.Methods: In a retrospective cohort at the first site, data from 26 008 patients with asthma aged 2-18 years (2012-2017) were used to develop a lasso-regularized logistic regression model predicting emergency department visits for asthma within one year of a primary care encounter, known as the Asthma Emergency Risk (AER) score. Internal validation was conducted on 8634 patient encounters from 2018. External validation of the AER score was conducted using 1313 pediatric patient encounters from a second site during 2018. The AER score components were then reweighted using logistic regression using data from the second site to improve local model performance. Prediction intervals (PI) were constructed via 10 000 bootstrapped samples.Results: At the first site, the AER score had a cross-validated area under the receiver operating characteristic curve (AUROC) of 0.768 (95% PI: 0.745-0.790) during model training and an AUROC of 0.769 in the 2018 internal validation dataset (p = 0.959). When applied without modification to the second site, the AER score had an AUROC of 0.684 (95% PI: 0.624-0.742). After local refitting, the cross-validated AUROC improved to 0.737 (95% PI: 0.676-0.794; p = 0.037 as compared to initial AUROC).Conclusions: The AER score demonstrated strong internal validity, but external validity was dependent on reweighting model components to reflect local data characteristics at the external site.
Keywords: Machine learning; clinical decision support; pediatric asthma; population health; predictive modeling.