驾驶员的负面情绪可能导致一些危险的驾驶行为,这反过来又导致严重的交通事故。然而,当前大多数关于驾驶员情绪的研究都使用单一模态,比如脑电图,眼睛跟踪器,和驱动数据。在复杂的情况下,单一模式可能无法充分考虑驾驶员的完整情绪特征,并提供较差的鲁棒性。近年来,一些研究使用多模态思维来监测单一情绪,如驾驶员疲劳和愤怒,但在实际驾驶环境中,负面情绪,如悲伤,愤怒,恐惧,和疲劳都对驾驶安全有重大影响。然而,很少有研究案例使用多模态数据来准确预测驾驶员的综合情绪。因此,基于多模态思想,本文旨在提高驾驶员的综合情绪识别。通过结合驾驶员声音的三种模态,面部图像,和视频序列,司机情绪的六个分类任务如下:悲伤,愤怒,恐惧,疲劳,幸福,和情感中立。为了准确识别驾驶员的负面情绪,提高行车安全,本文提出了一种基于CNN+Bi-LSTM+HAM的多模态融合框架来识别驾驶员情绪。该框架融合了驱动程序音频的特征向量,面部表情,和视频序列,用于全面的驾驶员情绪识别。实验证明了本文提出的多模态数据用于驾驶员情感识别的有效性,识别准确率达到85.52%。同时,通过对比实验和准确率、F1评分等评价指标,验证了该方法的有效性。
Negative emotions of drivers may lead to some dangerous driving behaviors, which in turn lead to serious traffic accidents. However, most of the current studies on driver emotions use a single modality, such as EEG, eye trackers, and driving data. In complex situations, a single modality may not be able to fully consider a driver\'s complete emotional characteristics and provides poor robustness. In recent years, some studies have used multimodal thinking to monitor single emotions such as driver fatigue and anger, but in actual driving environments, negative emotions such as sadness, anger, fear, and fatigue all have a significant impact on driving safety. However, there are very few research cases using multimodal data to accurately predict drivers\' comprehensive emotions. Therefore, based on the multi-modal idea, this paper aims to improve drivers\' comprehensive emotion recognition. By combining the three modalities of a driver\'s voice, facial image, and video sequence, the six classification tasks of drivers\' emotions are performed as follows: sadness, anger, fear, fatigue, happiness, and emotional neutrality. In order to accurately identify drivers\' negative emotions to improve driving safety, this paper proposes a multi-modal fusion framework based on the CNN + Bi-LSTM +
HAM to identify driver emotions. The framework fuses feature vectors of driver audio, facial expressions, and video sequences for comprehensive driver emotion recognition. Experiments have proved the effectiveness of the multi-modal data proposed in this paper for driver emotion recognition, and its recognition accuracy has reached 85.52%. At the same time, the validity of this method is verified by comparing experiments and evaluation indicators such as accuracy and F1 score.