分段任意模型(SAM),一般图像分割的基础模型,在许多自然图像分割任务中表现出令人印象深刻的零镜头性能。然而,SAM的性能在应用于医学图像时显著下降,主要是由于自然和医学图像领域之间的巨大差异。为了使SAM有效地适应医学图像,重要的是要纳入关键的三维信息,即,体积或时间知识,在微调期间。同时,我们的目标是充分利用SAM在其原始2D骨干中的预训练权重。在本文中,我们引入了一个与模态无关的SAM适应框架,命名为MA-SAM,适用于各种体积和视频医疗数据。我们的方法源于参数有效的微调策略,即仅更新一小部分权重增量,同时保留SAM的大部分预训练权重。通过将一系列3D适配器注入到图像编码器的变压器块中,我们的方法使预先训练的2D骨干能够从输入数据中提取三维信息。我们在五个医学图像分割任务上综合评估了我们的方法,通过跨CT使用11个公共数据集,MRI,和手术视频数据。值得注意的是,不使用任何提示,我们的方法始终优于各种最先进的3D方法,超过NNU-Net0.9%,2.6%,和9.9%的骰子用于CT多器官分割,MRI前列腺分割,和手术场景分割。我们的模型也展示了很强的泛化能力,并且在使用提示时擅长具有挑战性的肿瘤分割。我们的代码可在以下网址获得:https://github.com/cchen-cc/MA-SAM。
The Segment Anything Model (SAM), a foundation model for general image segmentation, has demonstrated impressive zero-shot performance across numerous natural image segmentation tasks. However, SAM\'s performance significantly declines when applied to medical images, primarily due to the substantial disparity between natural and medical image domains. To effectively adapt SAM to medical images, it is important to incorporate critical third-dimensional information, i.e., volumetric or temporal knowledge, during fine-tuning. Simultaneously, we aim to harness SAM\'s pre-trained weights within its original 2D backbone to the fullest extent. In this paper, we introduce a modality-agnostic SAM adaptation framework, named as MA-SAM, that is applicable to various volumetric and video medical data. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments while preserving the majority of SAM\'s pre-trained weights. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data. We comprehensively evaluate our method on five medical image segmentation tasks, by using 11 public datasets across CT, MRI, and surgical video data. Remarkably, without using any prompt, our method consistently outperforms various state-of-the-art 3D approaches, surpassing nnU-Net by 0.9%, 2.6%, and 9.9% in Dice for CT multi-organ segmentation, MRI prostate segmentation, and surgical scene segmentation respectively. Our model also demonstrates strong generalization, and excels in challenging tumor segmentation when prompts are used. Our code is available at: https://github.com/cchen-cc/MA-SAM.