物种识别是了解各种生物的丰度和分布的关键部分,对于生物多样性的保护和管理至关重要。传统的基于视觉的深度学习驱动的物种识别需要大量的标记良好,高质量的图像数据,对稀有和濒危物种的收集具有挑战性。此外,基于特定物种设计的识别方法泛化能力差,难以适应新的物种识别场景。为了解决这些问题,基于对比语言-图像预训练(CLIP)的零镜头物种识别已成为研究热点。然而,以前的研究主要利用物种的视觉描述信息和分类学信息来提高零射识别性能,利用物种的地理分布特征来提高零射识别性能尚未得到探索。为了填补这个空白,我们提出了一种CLIP驱动的零射物种识别方法,该方法结合了物种地理分布的知识。首先,我们根据物种地理分布统计数据设计了三个提示。然后,附加到物种数据集中的每个图像的纬度和经度坐标信息被转换为地址,它们被整合在一起,形成每个物种的地理分布知识。最后,通过训练后的CLIP图像编码器和文本编码器获取特征后,通过计算相似度得出物种识别结果。我们对iNaturalist2021数据集中的多个物种数据集进行了广泛的实验,哺乳动物的零射识别精度,软体动物,爬行动物,两栖动物,鸟,昆虫占44.96%,15.27%,17.51%,9.47%,28.35%,和7.03%,改善了2.07%,0.48%,0.35%,1.12%,1.64%,和0.61%,分别,与默认提示的CLIP相比。实验结果表明,融合地理分布统计数据可以有效提高零拍物种识别的性能,为物种领域知识的利用提供了新的途径。
Species recognition is a crucial part of understanding the abundance and distribution of various organisms and is important for biodiversity conservation and management. Traditional vision-based deep learning-driven species recognition requires large amounts of well-labeled, high-quality image data, the collection of which is challenging for rare and endangered species. In addition, recognition methods designed based on specific species have poor generalization ability and are difficult to adapt to new species recognition scenarios. To address these issues, zero-shot species recognition based on Contrastive Language-Image Pre-training (
CLIP) has become a research hotspot. However, previous studies have primarily utilized visual descriptive information and taxonomic information of species to improve zero-shot recognition performance, and the use of geographic distribution characteristics of species to improve zero-shot recognition performance has not been explored. To fill this gap, we proposed a
CLIP-driven zero-shot species recognition method that incorporates knowledge of the geographic distribution of species. First, we designed three prompts based on the species geographic distribution statistical data. Then, the latitude and longitude coordinate information attached to each image in the species dataset was converted into addresses, and they were integrated together to form the geographical distribution knowledge of each species. Finally, species recognition results were derived by calculating the similarity after acquiring features by the trained
CLIP image encoder and text encoder. We conducted extensive experiments on multiple species datasets from the iNaturalist 2021 dataset, where the zero-shot recognition accuracies of mammals, mollusks, reptiles, amphibians, birds, and insects were 44.96%, 15.27%, 17.51%, 9.47%, 28.35%, and 7.03%, an improvement of 2.07%, 0.48%, 0.35%, 1.12%, 1.64%, and 0.61%, respectively, as compared to
CLIP with default prompt. The experimental results show that the fusion of geographic distribution statistical data can effectively improve the performance of zero-shot species recognition, which provides a new way to utilize species domain knowledge.