背景:基于卷积神经网络(CNN)的方法在超分辨结构化照明显微镜(SR-SIM)数据的去噪和重建方面显示出出色的性能。因此,基于CNN的架构一直是现有研究的焦点。然而,双变压器,另一种最近提出的基于深度学习的图像复原架构,尚未对SR-SIM图像进行充分的去噪研究。此外,对于这些不同类型的基于深度学习的方法,迁移学习策略对于具有不同噪声特性和记录细胞结构的SR-SIM图像的去噪效果如何尚未得到充分探索。目前,公开可用的SR-SIM数据集的稀缺性限制了对深度学习方法的性能和泛化能力的探索。
结果:在这项工作中,我们介绍SwinT-fairSIM,一种基于SwinTransformer的新方法,用于恢复低信噪比的SR-SIM图像。实验结果表明,SwinT-fairsim的去噪效果优于以往基于CNN的去噪方法。此外,作为第二个贡献,两种类型的迁移学习-即,直接传输和微调-结合SwinT-fairsim和基于CNN的SR-SIM数据去噪方法进行基准测试。直接转移并不是一个可行的策略,但是微调产生的结果与从零开始的常规训练相当,同时节省了计算时间,并可能减少所需的训练数据量。作为第三个贡献,我们发布了四个原始SIM图像和已经重建的SR-SIM图像的数据集。这些数据集涵盖两种不同类型的细胞结构,微管蛋白丝和囊泡结构。微管蛋白丝可获得不同的噪声水平。
结论:SwinT-fairSIM方法非常适合对SR-SIM图像进行去噪。通过微调,已经训练的模型可以很容易地适应不同的噪声特性和细胞结构。此外,提供的数据集的结构使得研究界可以很容易地使用它们进行去噪研究,超分辨率,和迁移学习策略。
Convolutional neural network (CNN)-based methods have shown excellent performance in denoising and reconstruction of super-resolved structured illumination microscopy (SR-SIM) data. Therefore, CNN-based architectures have been the focus of existing studies. However, Swin Transformer, an alternative and recently proposed deep learning-based image restoration architecture, has not been fully investigated for denoising SR-SIM images. Furthermore, it has not been fully explored how well transfer learning strategies work for denoising SR-SIM images with different noise characteristics and recorded cell structures for these different types of deep learning-based methods. Currently, the scarcity of publicly available SR-SIM datasets limits the exploration of the performance and generalization capabilities of deep learning methods.
In this work, we present SwinT-fairSIM, a novel method based on the Swin Transformer for restoring SR-SIM images with a low signal-to-noise ratio. The experimental results show that SwinT-fairSIM outperforms previous CNN-based denoising methods. Furthermore, as a second contribution, two types of transfer learning-namely, direct transfer and fine-tuning-were benchmarked in combination with SwinT-fairSIM and CNN-based methods for denoising SR-SIM data. Direct transfer did not prove to be a viable strategy, but fine-tuning produced results comparable to conventional training from scratch while saving computational time and potentially reducing the amount of training data required. As a third contribution, we publish four datasets of raw SIM images and already reconstructed SR-SIM images. These datasets cover two different types of cell structures, tubulin filaments and vesicle structures. Different noise levels are available for the tubulin filaments.
The SwinT-fairSIM method is well suited for denoising SR-SIM images. By fine-tuning, already trained models can be easily adapted to different noise characteristics and cell structures. Furthermore, the provided datasets are structured in a way that the research community can readily use them for research on denoising, super-resolution, and transfer learning strategies.