BROWSE
Detail
Journal Article
Autonomous vehicle passenger emotion prediction: Physics-informed graph neural networks with behavioral and environmental signals
Cited 0 time in 
Share 
Share
- Authors
- Mi Chang, Jiwoo Han, Daesub Yoon
- Issue Date
- 2026-06
- Citation
- Expert Systems with Applications, v.331, pp.1-17
- ISSN
- 0957-4174
- Publisher
- Elsevier
- Language
- English
- Type
- Journal Article
- Abstract
- Predicting passenger emotional states in autonomous vehicles is essential for developing intelligent in-vehicle systems that enhance user acceptance and safety. Existing approaches predominantly rely on single-modality sensing or treat internal physiological responses and external environmental dynamics separately, limiting their ability to capture the complex human-environment interactions underlying passenger anxiety. This study proposes a physics-informed graph neural network (PIGNN) framework that constructs spatiotemporal graphs integrating passengers’ internal responses with external environmental dynamics from virtual reality-based multimodal data and trains the model using physics-informed constraints. We conducted virtual reality-based autonomous driving experiments using CARLA and collected multimodal data from 100 participants, including gaze tracking, head rotation, electrodermal activity, vehicle dynamics, and surrounding object movements. Statistical analysis revealed strong coupling between internal behavioral indicators and external environmental dynamics during tense states. Gaze velocity and head rotation exhibited high correlations with lateral deviation (r = 0.899 and r = 0.858, respectively) and distance variation (r = 0.715 and r = 0.676, respectively), while electrodermal responses showed significant correlations with proximity entropy (r = 0.331) and distance variation (r = 0.301). Based on these empirically validated internal-external relationships, we formulated nine physics-informed constraints grounded in cognitive science and traffic psychology, guiding the model to learn behaviorally meaningful patterns. The resulting PIGNN achieved R2 = 0.974 and MAE = 0.070 under leave-one-subject-out cross-validation across all 100 participants. The proposed framework provides a foundation for intelligent in-vehicle systems capable of passenger state estimation and adaptive response generation.
- Keyword
- Intelligent in-vehicle system, Passenger emotion prediction, Physics-informed graph neural network, Spatiotemporal graph
- KSP Keywords
- Adaptive response, Autonomous vehicle, Cognitive science, Cross validation(CV), Electrodermal responses, Emotion prediction, Existing Approaches, Human-environment interactions, Neural network, Physiological responses, Statistical Analysis