油茶是一种具有很高经济价值的作物,然而,它特别容易受到各种疾病和害虫的影响,这大大降低了它的产量和质量。因此,病害山茶叶的精确分割和分类对于有效管理病虫害至关重要。深度学习在植物病虫害分割方面表现出显著优势,特别是在复杂的图像处理和自动特征提取中。然而,在采用单模态模型分割油茶病害时,出现了三个关键挑战:(A)病变可能与复杂背景的颜色非常相似;(B)病叶的小部分重叠;(C)单叶上存在多种疾病。这些因素极大地阻碍了分割准确性。一种新颖的多模态模型,CNN-变压器双U形网络(CTDUNet),基于CNN-Transformer架构,已经被提出来集成图像和文本信息。该模型首先利用文本数据来解决单模态图像特征的缺点,增强其区分病变与环境特征的能力,即使是在彼此非常相似的条件下。此外,我们引入坐标空间注意力(CSA),它专注于目标之间的位置关系,从而改善了重叠叶边的分割。此外,交叉注意力(CA)用于有效地对齐图像和文本特征,保存本地信息,增强对各种疾病的感知和区分。CTDUNet模型在自制的多模态数据集上进行了评估,并与几个模型进行了比较,包括DeeplabV3+,UNet,PSPNet,Segformer,HrNet,和语言满足视觉转换(LVIT)。实验结果表明,CTDUNet实现了86.14%的平均交集(mIoU),分别超过多模态模型和最佳单模态模型3.91%和5.84%,分别。此外,CTDUNet在油茶病虫害的多分类中表现出高度平衡。这些结果表明融合图像和文本多模态信息在山茶病分割中的成功应用,实现了出色的性能。
Camellia oleifera is a crop of high economic value, yet it is particularly susceptible to various diseases and pests that significantly reduce its yield and quality. Consequently, the precise segmentation and classification of diseased Camellia leaves are vital for managing pests and diseases effectively. Deep learning exhibits significant advantages in the segmentation of plant diseases and pests, particularly in complex image processing and automated feature extraction. However, when employing single-modal models to segment Camellia oleifera diseases, three critical challenges arise: (A) lesions may closely resemble the colors of the complex background; (B) small sections of diseased leaves overlap; (C) the presence of multiple diseases on a single leaf. These factors considerably hinder segmentation accuracy. A novel
multimodal model, CNN-Transformer Dual U-shaped Network (CTDUNet), based on a CNN-Transformer architecture, has been proposed to integrate image and text information. This model first utilizes text data to address the shortcomings of single-modal image features, enhancing its ability to distinguish lesions from environmental characteristics, even under conditions where they closely resemble one another. Additionally, we introduce Coordinate Space Attention (CSA), which focuses on the positional relationships between targets, thereby improving the segmentation of overlapping leaf edges. Furthermore, cross-attention (CA) is employed to align image and text features effectively, preserving local information and enhancing the perception and differentiation of various diseases. The CTDUNet model was evaluated on a self-made
multimodal dataset compared against several models, including DeeplabV3+, UNet, PSPNet, Segformer, HrNet, and Language meets Vision Transformer (LViT). The experimental results demonstrate that CTDUNet achieved an mean Intersection over Union (mIoU) of 86.14%, surpassing both
multimodal models and the best single-modal model by 3.91% and 5.84%, respectively. Additionally, CTDUNet exhibits high balance in the multi-class segmentation of Camellia oleifera diseases and pests. These results indicate the successful application of fused image and text
multimodal information in the segmentation of Camellia disease, achieving outstanding performance.