Objective Prediction of Next-Day's Affect Using Multimodal Physiological and Behavioral Data: Algorithm Development and Validation Study

Salar Jafarlou; Jocelyn Lai; Iman Azimi; Zahra Mousavi; Sina Labbaf; Ramesh C Jain; Nikil Dutt; Jessica L Borelli; Amir Rahmani

doi:10.2196/39425

Objective Prediction of Next-Day's Affect Using Multimodal Physiological and Behavioral Data: Algorithm Development and Validation Study

JMIR Form Res. 2023 Mar 15:7:e39425. doi: 10.2196/39425.

Authors

Salar Jafarlou¹, Jocelyn Lai², Iman Azimi^{1

3}, Zahra Mousavi², Sina Labbaf¹, Ramesh C Jain^{1

3}, Nikil Dutt¹, Jessica L Borelli⁴, Amir Rahmani^{1

3

5}

Affiliations

¹ Donald Bren School of Information and Computer Sciences, University of California, Irvine, Irvine, CA, United States.
² Department of Psychological Science, University of California, Irvine, Irvine, CA, United States.
³ Institute for Future Health, University of California, Irvine, Irvine, CA, United States.
⁴ Department of Cognitive Science, University of California, Irvine, Irvine, CA, United States.
⁵ School of Nursing, University of California, Irvine, Irvine, CA, United States.

PMID: 36920456
PMCID: PMC10131982
DOI: 10.2196/39425

Abstract

Background: Affective states are important aspects of healthy functioning; as such, monitoring and understanding affect is necessary for the assessment and treatment of mood-based disorders. Recent advancements in wearable technologies have increased the use of such tools in detecting and accurately estimating mental states (eg, affect, mood, and stress), offering comprehensive and continuous monitoring of individuals over time.

Objective: Previous attempts to model an individual's mental state relied on subjective measurements or the inclusion of only a few objective monitoring modalities (eg, smartphones). This study aims to investigate the capacity of monitoring affect using fully objective measurements. We conducted a comparatively long-term (12-month) study with a holistic sampling of participants' moods, including 20 affective states.

Methods: Longitudinal physiological data (eg, sleep and heart rate), as well as daily assessments of affect, were collected using 3 modalities (ie, smartphone, watch, and ring) from 20 college students over a year. We examined the difference between the distributions of data collected from each modality along with the differences between their rates of missingness. Out of the 20 participants, 7 provided us with 200 or more days' worth of data, and we used this for our predictive modeling setup. Distributions of positive affect (PA) and negative affect (NA) among the 7 selected participants were observed. For predictive modeling, we assessed the performance of different machine learning models, including random forests (RFs), support vector machines (SVMs), multilayer perceptron (MLP), and K-nearest neighbor (KNN). We also investigated the capability of each modality in predicting mood and the most important features of PA and NA RF models.

Results: RF was the best-performing model in our analysis and performed mood and stress (nervousness) prediction with ~81% and ~72% accuracy, respectively. PA models resulted in better performance compared to NA. The order of the most important modalities in predicting PA and NA was the smart ring, phone, and watch, respectively. SHAP (Shapley Additive Explanations) analysis showed that sleep and activity-related features were the most impactful in predicting PA and NA.

Conclusions: Generic machine learning-based affect prediction models, trained with population data, outperform existing methods, which use the individual's historical information. Our findings indicated that our mood prediction method outperformed the existing methods. Additionally, we found that sleep and activity level were the most important features for predicting next-day PA and NA, respectively.

Keywords: affective computing; mental health; wearable devices.

©Salar Jafarlou, Jocelyn Lai, Iman Azimi, Zahra Mousavi, Sina Labbaf, Ramesh C Jain, Nikil Dutt, Jessica L Borelli, Amir Rahmani. Originally published in JMIR Formative Research (https://formative.jmir.org), 15.03.2023.