PDF(Pubmed)
Abstract：
Cancer research encompasses data across various scales, modalities, and resolutions, from screening and diagnostic imaging to digitized histopathology slides to various types of molecular data and clinical records. The integration of these diverse data types for personalized cancer care and predictive modeling holds the promise of enhancing the accuracy and reliability of cancer screening, diagnosis, and treatment. Traditional analytical methods, which often focus on isolated or unimodal information, fall short of capturing the complex and heterogeneous nature of cancer data. The advent of deep neural networks has spurred the development of sophisticated multimodal data fusion techniques capable of extracting and synthesizing information from disparate sources. Among these, Graph Neural Networks (GNNs) and Transformers have emerged as powerful tools for multimodal learning, demonstrating significant success. This review presents the foundational principles of multimodal learning including oncology data modalities, taxonomy of multimodal learning, and fusion strategies. We delve into the recent advancements in GNNs and Transformers for the fusion of multimodal data in oncology, spotlighting key studies and their pivotal findings. We discuss the unique challenges of multimodal learning, such as data heterogeneity and integration complexities, alongside the opportunities it presents for a more nuanced and comprehensive understanding of cancer. Finally, we present some of the latest comprehensive multimodal pan-cancer data sources. By surveying the landscape of multimodal data integration in oncology, our goal is to underline the transformative potential of multimodal GNNs and Transformers. Through technological advancements and the methodological innovations presented in this review, we aim to chart a course for future research in this promising field. This review may be the first that highlights the current state of multimodal modeling applications in cancer using GNNs and transformers, presents comprehensive multimodal oncology data sources, and sets the stage for multimodal evolution, encouraging further exploration and development in personalized cancer care.
摘要：
癌症研究涵盖了各种规模的数据，模态，和决议，从筛查和诊断成像到数字化组织病理学幻灯片，再到各种类型的分子数据和临床记录。将这些不同的数据类型集成到个性化癌症护理和预测建模中，有望提高癌症筛查的准确性和可靠性。诊断,和治疗。传统的分析方法,通常专注于孤立或单峰信息，未能捕捉到癌症数据的复杂性和异质性。深度神经网络的出现刺激了能够从不同来源提取和合成信息的复杂多模态数据融合技术的发展。其中,图神经网络(GNN)和变形金刚已经成为多模态学习的强大工具，展示显著的成功。这篇综述介绍了多模式学习的基本原理，包括肿瘤学数据模式，多模态学习的分类法，和融合策略。我们深入研究了GNN和Transformers在肿瘤学中多模态数据融合方面的最新进展，聚焦关键研究及其关键发现。我们讨论了多模态学习的独特挑战，例如数据异质性和集成复杂性，除了它提供的机会，对癌症有更细致和全面的了解。最后,我们提供了一些最新的综合多模式泛癌症数据来源。通过调查肿瘤学中多模态数据集成的情况，我们的目标是强调多模态GNN和变形金刚的变革潜力。通过本综述中提出的技术进步和方法创新，我们的目标是为这个有前途的领域的未来研究绘制一条路线。这篇综述可能是第一个突出使用GNN和变压器在癌症中的多模态建模应用现状的综述，提供全面的多模式肿瘤学数据源，并为多模态进化奠定了基础，鼓励在个性化癌症护理方面进一步探索和发展。
