目的:人工智能(AI)辅助方法在正常组织的自动勾画方面取得了重大进展。然而,这些方法与放疗目标体积的自动轮廓有关。我们的目标是将目标体积的描绘建模为临床决策问题,通过利用大型语言模型辅助的多模态学习方法来解决。
方法:视觉语言模型,被称为Medformer,已经开发出来了,采用分层视觉转换器作为其骨干,并结合大型语言模型来提取文本丰富的特征。上下文嵌入的语言特征通过视觉语言注意模块被无缝地集成到用于语言感知的视觉编码的视觉特征中。Metrics,包括骰子相似系数(DSC),联合交叉点(IOU),和95百分位数Hausdorff距离(HD95),用于定量评估我们模型的性能。评估是在内部前列腺癌数据集和公共口咽癌(OPC)数据集上进行的,共668个科目。
结果:我们的Medformer的DSC为0.81±0.10对0.72±0.10,IOU为0.73±0.12对0.65±0.09,HD95为9.86±9.77mm对19.13±12.96mm。同样,在OPC数据集上,它实现了0.77±0.11对0.72±0.09的DSC,0.70±0.09对0.65±0.07的IOU,以及7.52±4.8mm对13.63±7.13mm的HD95,代表显著改善(p<0.05)。为了描绘临床目标体积(CTV),Medformer的DSC为0.91±0.04,IOU为0.85±0.05,HD95为2.98±1.60mm,与其他最先进的算法相当。
结论:基于多模态学习的治疗目标自动描绘优于单纯依赖视觉特征的传统方法。我们的方法可以用于常规实践中,以快速绘制CTV/GTV的轮廓。
OBJECTIVE: Artificial intelligence (AI)-aided methods have made significant progress in the auto-delineation of normal tissues. However, these approaches struggle with the auto-contouring of radiotherapy target volume. Our goal is to model the delineation of target volume as a clinical decision-making problem, resolved by leveraging large language model-aided multimodal learning approaches.
METHODS: A vision-language model, termed Medformer, has been developed, employing the hierarchical vision transformer as its backbone, and incorporating large language models to extract text-rich features. The contextually embedded linguistic features are seamlessly integrated into visual features for language-aware visual encoding through the visual language attention module. Metrics, including Dice similarity coefficient (DSC), intersection over union (IOU), and 95th percentile Hausdorff distance (HD95), were used to quantitatively evaluate the performance of our model. The evaluation was conducted on an in-house prostate cancer dataset and a public oropharyngeal carcinoma (OPC) dataset, totaling 668 subjects.
RESULTS: Our Medformer achieved a DSC of 0.81 ± 0.10 versus 0.72 ± 0.10, IOU of 0.73 ± 0.12 versus 0.65 ± 0.09, and HD95 of 9.86 ± 9.77 mm versus 19.13 ± 12.96 mm for delineation of gross tumor volume (GTV) on the prostate cancer dataset. Similarly, on the OPC dataset, it achieved a DSC of 0.77 ± 0.11 versus 0.72 ± 0.09, IOU of 0.70 ± 0.09 versus 0.65 ± 0.07, and HD95 of 7.52 ± 4.8 mm versus 13.63 ± 7.13 mm, representing significant improvements (p < 0.05). For delineating the clinical target volume (CTV), Medformer achieved a DSC of 0.91 ± 0.04, IOU of 0.85 ± 0.05, and HD95 of 2.98 ± 1.60 mm, comparable to other state-of-the-art algorithms.
CONCLUSIONS: Auto-delineation of the treatment target based on multimodal learning outperforms conventional approaches that rely purely on visual features. Our method could be adopted into routine practice to rapidly contour CTV/GTV.