PDF(Pubmed)
Abstract：
Pulmonary disease is a commonly occurring abnormality throughout this world. The pulmonary diseases include Tuberculosis, Pneumothorax, Cardiomegaly, Pulmonary atelectasis, Pneumonia, etc. A timely prognosis of pulmonary disease is essential. Increasing progress in Deep Learning (DL) techniques has significantly impacted and contributed to the medical domain, specifically in leveraging medical imaging for analysis, prognosis, and therapeutic decisions for clinicians. Many contemporary DL strategies for radiology focus on a single modality of data utilizing imaging features without considering the clinical context that provides more valuable complementary information for clinically consistent prognostic decisions. Also, the selection of the best data fusion strategy is crucial when performing Machine Learning (ML) or DL operation on multimodal heterogeneous data. We investigated multimodal medical fusion strategies leveraging DL techniques to predict pulmonary abnormality from the heterogeneous radiology Chest X-Rays (CXRs) and clinical text reports. In this research, we have proposed two effective unimodal and multimodal subnetworks to predict pulmonary abnormality from the CXR and clinical reports. We have conducted a comprehensive analysis and compared the performance of unimodal and multimodal models. The proposed models were applied to standard augmented data and the synthetic data generated to check the model\'s ability to predict from the new and unseen data. The proposed models were thoroughly assessed and examined against the publicly available Indiana university dataset and the data collected from the private medical hospital. The proposed multimodal models have given superior results compared to the unimodal models.
摘要：
肺部疾病是世界上常见的异常。肺部疾病包括肺结核，气胸,心脏肥大,肺不张,肺炎,等。肺部疾病的及时预后至关重要。深度学习（DL）技术的进步对医学领域产生了重大影响并做出了贡献，特别是在利用医学成像进行分析方面，预后,和临床医生的治疗决策。用于放射学的许多当代DL策略关注于利用成像特征的数据的单一模态,而不考虑为临床一致的预后决策提供更有价值的补充信息的临床背景。此外，在多模态异构数据上执行机器学习(ML)或DL操作时，选择最佳数据融合策略至关重要。我们研究了利用DL技术从异质放射学胸部X射线（CXR）和临床文本报告中预测肺部异常的多模式医学融合策略。在这项研究中,我们从CXR和临床报告提出了两种有效的单峰和多模态子网络来预测肺异常.我们进行了全面的分析，并比较了单峰和多峰模型的性能。将所提出的模型应用于标准增强数据和生成的合成数据，以检查模型从新的和看不见的数据进行预测的能力。根据公开的印第安纳大学数据集和从私人医院收集的数据，对提出的模型进行了全面评估和检查。与单峰模型相比，所提出的多峰模型具有出色的结果。
