多模态可穿戴技术为人类活动识别带来了广泛的可能性,更具体地说是个性化的饮食习惯监测。现在出现的挑战是从多个来源收集的高维数据中选择最具鉴别力的信息。可用的具有复杂结构的融合算法很少用于计算受限的环境,该环境需要直接在源处集成信息。因此,需要更简单的低级融合方法。
在没有数据合并过程的情况下,直接将高维原始数据应用于深度分类器的成本在响应时间方面在计算上是昂贵的,能源消耗,和内存要求。考虑到这一点,我们的目标是以一种计算有效的方式开发一种数据融合技术,以在较低的维度上更全面地了解人类活动动态。主要目标是考虑多感官数据的统计依赖性,并探索不同活动的模态间相关模式。
在这种技术中,时间上的信息(无论源的数量)被转换为2D空间,以促进对其他饮食事件的分类。这是基于以下假设:由各种传感器捕获的数据在统计上彼此相关联,并且所有这些信号的协方差矩阵具有与每个活动相关的唯一分布,其可以被编码在轮廓表示上。然后将这些表示用作深度模型的输入,以学习与特定活动相关联的特定模式。
为了显示所提出的融合算法的泛化性,考虑了2种不同的情况。这些场景在时间片段大小方面是不同的,活动类型,可穿戴设备,主题,和深度学习架构。第一种情况使用了一个数据集,其中单个参与者在佩戴EmpaticaE4腕带时执行了有限数量的活动。在第二种情况下,使用了与日常生活活动相关的数据集,其中10名不同的参与者在执行一组更复杂的活动时佩戴惯性测量单元.从第二个场景的离开一个主题交叉验证获得的精度度量达到0.803。还评估了缺失数据对性能下降的影响。
最后,所提出的融合技术提供了在单个2D表示中嵌入不同模态上的联合变异性信息的可能性,这导致获得手头日常人类活动不同方面的更全局视图。并在活动识别中保持所需的性能水平。
Multimodal wearable technologies have brought forward wide possibilities in human activity recognition, and more specifically personalized monitoring of eating habits. The emerging challenge now is the selection of most discriminative information from high-dimensional data collected from multiple sources. The available fusion algorithms with their complex structure are poorly adopted to the computationally constrained environment which requires integrating information directly at the source. As a result, more simple low-level fusion methods are needed.
In the absence of a data combining process, the cost of directly applying high-dimensional raw data to a deep classifier would be computationally expensive with regard to the response time, energy consumption, and memory requirement. Taking this into account, we aimed to develop a data fusion technique in a computationally efficient way to achieve a more comprehensive insight of human activity dynamics in a lower dimension. The major objective was considering statistical dependency of multisensory data and exploring intermodality correlation patterns for different activities.
In this technique, the information in time (regardless of the number of sources) is transformed into a 2D space that facilitates classification of eating episodes from others. This is based on a hypothesis that data captured by various sensors are statistically associated with each other and the covariance matrix of all these signals has a unique distribution correlated with each activity which can be encoded on a contour representation. These representations are then used as input of a deep model to learn specific patterns associated with specific activity.
In order to show the generalizability of the proposed fusion algorithm, 2 different scenarios were taken into account. These scenarios were different in terms of temporal segment size, type of activity, wearable device, subjects, and deep learning architecture. The first scenario used a data set in which a single participant performed a limited number of activities while wearing the Empatica E4 wristband. In the second scenario, a data set related to the activities of daily living was used where 10 different participants wore inertial measurement units while performing a more complex set of activities. The precision metric obtained from leave-one-subject-out cross-validation for the second scenario reached 0.803. The impact of missing data on performance degradation was also evaluated.
To conclude, the proposed fusion technique provides the possibility of embedding joint variability information over different modalities in just a single 2D representation which results in obtaining a more global view of different aspects of daily human activities at hand, and yet preserving the desired performance level in activity recognition.