■该研究旨在提出一种多模态模型,该模型结合了宏观和微观图像,并分析了其对具有不同经验水平的临床医生决策的影响。
■首先,我们构建了5种皮肤病的多模式数据集.接下来,我们在三种不同类型的图像上训练了单峰模型,并选择了表现最好的模型作为基础学习者。然后,我们使用软投票策略来创建多模态模型。最后,12名临床医生分为三组,每组包括一名皮肤科主任,一位负责皮肤科的医生,一位常驻皮肤科医生,和一名全科医生。他们被要求在四种情况下诊断皮肤疾病(仅宏观图像,只有皮肤病理学图像,宏观和皮肤病理学图像,所有图像和元数据),和三种辅助情况(带有模型1辅助的宏观图像,使用模型2和3辅助的皮肤病理学图像,所有具有多模态模型4辅助的图像)。记录临床医生诊断的准确性和每次诊断的时间。
■在训练的模型中,视觉变压器(ViT)实现了最佳性能,训练集上的准确度为0.8636、0.9545、0.9673,AUC为0.9823、0.9952、0.9989,分别。然而,在外部验证集上,它们的精度分别仅为0.70、0.90和0.94。与单峰模型相比,多峰模型表现良好,在外部验证集上实现0.98的精度。logit回归分析的结果表明,所有模型都有助于临床医生做出诊断决策[赔率比(OR)>1],而元数据不为临床医生提供帮助(OR<1)。线性分析结果表明,元数据显著增加了临床医生的诊断时间(P<0.05),而模范援助没有(P>0.05)。
■这项研究的结果表明,多模态模型有效地提高了临床医生\'的诊断性能,而不会显着增加诊断时间。然而,进一步的大规模前瞻性研究是必要的。
UNASSIGNED: The study aimed to propose a multimodal model that incorporates both macroscopic and microscopic images and analyze its influence on clinicians\' decision-making with different levels of experience.
UNASSIGNED: First, we constructed a multimodal dataset for five skin disorders. Next, we trained unimodal models on three different types of images and selected the best-performing models as the base learners. Then, we used a soft voting strategy to create the multimodal model. Finally, 12 clinicians were divided into three groups, with each group including one director dermatologist, one dermatologist-in-charge, one resident dermatologist, and one general practitioner. They were asked to diagnose the skin disorders in four unaided situations (macroscopic images only, dermatopathological images only, macroscopic and dermatopathological images, all images and metadata), and three aided situations (macroscopic images with model 1 aid, dermatopathological images with model 2&3 aid, all images with multimodal model 4 aid). The clinicians\' diagnosis accuracy and time for each diagnosis were recorded.
UNASSIGNED: Among the trained models, the vision transformer (ViT) achieved the best performance, with accuracies of 0.8636, 0.9545, 0.9673, and AUCs of 0.9823, 0.9952, 0.9989 on the training set, respectively. However, on the external validation set, they only achieved accuracies of 0.70, 0.90, and 0.94, respectively. The multimodal model performed well compared to the unimodal models, achieving an accuracy of 0.98 on the external validation set. The results of logit regression analysis indicate that all models are helpful to clinicians in making diagnostic decisions [Odds Ratios (OR) > 1], while metadata does not provide assistance to clinicians (OR < 1). Linear analysis results indicate that metadata significantly increases clinicians\' diagnosis time (P < 0.05), while model assistance does not (P > 0.05).
UNASSIGNED: The results of this study suggest that the multimodal model effectively improves clinicians\' diagnostic performance without significantly increasing the diagnostic time. However, further large-scale prospective studies are necessary.