Abstract：
The fusion of multi-modal data, e.g., medical images and genomic profiles, can provide complementary information and further benefit disease diagnosis. However, multi-modal disease diagnosis confronts two challenges: (1) how to produce discriminative multi-modal representations by exploiting complementary information while avoiding noisy features from different modalities. (2) how to obtain an accurate diagnosis when only a single modality is available in real clinical scenarios. To tackle these two issues, we present a two-stage disease diagnostic framework. In the first multi-modal learning stage, we propose a novel Momentum-enriched Multi-Modal Low-Rank (M3LR) constraint to explore the high-order correlations and complementary information among different modalities, thus yielding more accurate multi-modal diagnosis. In the second stage, the privileged knowledge of the multi-modal teacher is transferred to the unimodal student via our proposed Discrepancy Supervised Contrastive Distillation (DSCD) and Gradient-guided Knowledge Modulation (GKM) modules, which benefit the unimodal-based diagnosis. We have validated our approach on two tasks: (i) glioma grading based on pathology slides and genomic data, and (ii) skin lesion classification based on dermoscopy and clinical images. Experimental results on both tasks demonstrate that our proposed method consistently outperforms existing approaches in both multi-modal and unimodal diagnoses.
摘要：
多模态数据的融合,例如，医学图像和基因组概况，可以提供补充信息，进一步有利于疾病诊断。然而,多模态疾病诊断面临两个挑战：（1）如何通过利用互补信息同时避免来自不同模态的噪声特征来产生有区别的多模态表示。（2）在真实临床场景中，当只有单一模态可用时，如何获得准确的诊断。为了解决这两个问题，我们提出了一个两阶段的疾病诊断框架.在第一个多模态学习阶段，我们提出了一种新的富含动量的多模态低秩（M3LR）约束来探索不同模态之间的高阶相关性和互补信息，从而产生更准确的多模态诊断。在第二阶段,多模式教师的特权知识通过我们提出的差异监督对比蒸馏（DSCD）和梯度指导知识调制（GKM）模块转移给单峰学生，这有利于基于单峰的诊断。我们在两个任务上验证了我们的方法：（i）基于病理幻灯片和基因组数据的神经胶质瘤分级，和(ii)基于皮肤镜和临床图像的皮肤病变分类。两项任务的实验结果表明，我们提出的方法在多模态和单峰诊断中始终优于现有方法。
