PDF(Sci-hub)
Abstract：
The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.
摘要：
在过去的十年中，多模态学习的研究进展在多个领域迅速发展，尤其是在计算机视觉中。多模式数据流和深度学习算法的不断增长的潜力促进了深度多模式学习的日益普及。这涉及能够统一处理和分析多模态信息的模型的开发。非结构化的现实世界数据可以固有地采取多种形式，也被称为模态，通常包括视觉和文本内容。从这类数据中提取相关模式仍然是深度学习研究人员的激励目标。在本文中,我们寻求通过探索如何生成考虑跨感官模式的异构视觉线索的集成和组合的深度模型来提高对计算机视觉社区深度多模式学习的关键概念和算法的理解。特别是,我们从当前关于深度多模态学习的文献中总结了六个观点，即：多模态数据表示，多模态融合(即，传统和基于深度学习的方案)，多任务学习，多模态对齐，多模态迁移学习，和零射学习。我们还调查了当前的多模式应用程序，并提供了一组基准数据集，用于解决各个视觉领域的问题。最后,我们强调了深度多模态学习的局限性和挑战，并为未来的研究提供了见解和方向。
