PDF(Pubmed)
Abstract：
The broad adoption of electronic health record (EHR) systems brings us a tremendous amount of clinical data and thus provides opportunities to conduct data-based healthcare research to solve various clinical problems in the medical domain. Machine learning and deep learning methods are widely used in the medical informatics and healthcare domain due to their power to mine insights from raw data. When adapting deep learning models for EHR data, it is essential to consider its heterogeneous nature: EHR contains patient records from various sources including medical tests (e.g. blood test, microbiology test), medical imaging, diagnosis, medications, procedures, clinical notes, etc. Those modalities together provide a holistic view of patient health status and complement each other. Therefore, combining data from multiple modalities that are intrinsically different is challenging but intuitively promising in deep learning for EHR. To assess the expectations of multimodal data, we introduce a comprehensive fusion framework designed to integrate temporal variables, medical images, and clinical notes in EHR for enhanced performance in clinical risk prediction. Early, joint, and late fusion strategies are employed to combine data from various modalities effectively. We test the model with three predictive tasks: in-hospital mortality, long length of stay, and 30-day readmission. Experimental results show that multimodal models outperform uni-modal models in the tasks involved. Additionally, by training models with different input modality combinations, we calculate the Shapley value for each modality to quantify their contribution to multimodal performance. It is shown that temporal variables tend to be more helpful than CXR images and clinical notes in the three explored predictive tasks.
摘要：
电子健康记录（EHR）系统的广泛采用为我们带来了大量的临床数据，从而为进行基于数据的医疗保健研究提供了机会，以解决医疗领域的各种临床问题。机器学习和深度学习方法由于其从原始数据中挖掘见解的能力而被广泛用于医学信息学和医疗保健领域。在为EHR数据调整深度学习模型时，必须考虑其异质性：EHR包含来自各种来源的患者记录，包括医学检查（例如血液检查，微生物学测试)，医学成像,诊断,药物,程序,临床笔记，等。这些模式共同提供了患者健康状况的整体视图，并相互补充。因此,将来自本质上不同的多种模式的数据组合在一起具有挑战性，但在EHR的深度学习中具有直观的前景。为了评估多峰数据的预期，我们引入了一个旨在整合时间变量的综合融合框架，医学图像，和EHR中的临床注释，以增强临床风险预测的性能。早期，接头,并采用后期融合策略有效地组合来自各种模态的数据。我们用三个预测任务来测试模型：住院死亡率，长时间的逗留，30天的重新接纳。实验结果表明，在所涉及的任务中，多模态模型优于单模态模型。此外,通过训练具有不同输入模态组合的模型，我们计算每种模态的Shapley值，以量化它们对多模态性能的贡献。结果表明，在三个探索的预测任务中，时间变量往往比CXR图像和临床注释更有帮助。
