Abstract：
Convolutional Neural Networks (CNNs) have demonstrated outstanding performance in various domains, such as face recognition, object detection, and image segmentation. However, the lack of transparency and limited interpretability inherent in CNNs pose challenges in fields such as medical diagnosis, autonomous driving, finance, and military applications. Several studies have explored the interpretability of CNNs and proposed various post-hoc interpretable methods. The majority of these methods are feature-based, focusing on the influence of input variables on outputs. Few methods undertake the analysis of parameters in CNNs and their overall structure. To explore the structure of CNNs and intuitively comprehend the role of their internal parameters, we propose an Attribution Graph-based Interpretable method for CNNs (AGIC) which models the overall structure of CNNs as graphs and provides interpretability from global and local perspectives. The runtime parameters of CNNs and feature maps of each image sample are applied to construct attribution graphs (At-GCs), where the convolutional kernels are represented as nodes and the SHAP values between kernel outputs are assigned as edges. These At-GCs are then employed to pretrain a newly designed heterogeneous graph encoder based on Deep Graph Infomax (DGI). To comprehensively delve into the overall structure of CNNs, the pretrained encoder is used for two types of interpretable tasks: (1) a classifier is attached to the pretrained encoder for the classification of At-GCs, revealing the dependency of At-GC\'s topological characteristics on the image sample categories, and (2) a scoring aggregation (SA) network is constructed to assess the importance of each node in At-GCs, thus reflecting the relative importance of kernels in CNNs. The experimental results indicate that the topological characteristics of At-GC exhibit a dependency on the sample category used in its construction, which reveals that kernels in CNNs show distinct combined activation patterns for processing different image categories, meanwhile, the kernels that receive high scores from SA network are crucial for feature extraction, whereas low-scoring kernels can be pruned without affecting model performance, thereby enhancing the interpretability of CNNs.
摘要：
卷积神经网络(CNN)在各个领域都表现出了出色的性能，比如人脸识别,物体检测,和图像分割。然而,CNN固有的缺乏透明度和有限的可解释性给医疗诊断等领域带来了挑战，自主驾驶,金融,和军事应用。一些研究探索了CNN的可解释性，并提出了各种事后可解释的方法。这些方法中的大多数是基于特征的，关注输入变量对输出的影响。很少有方法对CNN中的参数及其整体结构进行分析。为了探索CNN的结构并直观地理解其内部参数的作用，我们提出了一种基于归因图的CNN可解释方法（AGIC），该方法将CNN的整体结构建模为图形，并从全局和局部角度提供可解释性。CNN的运行时参数和每个图像样本的特征图被应用于构造归因图(At-GC)，其中卷积内核表示为节点，内核输出之间的SHAP值被分配为边。然后使用这些At-GC来基于DeepGraphInfomax（DGI）预训练新设计的异构图编码器。为了全面研究CNN的整体结构，预训练的编码器用于两种类型的可解释任务：（1）一个分类器被附加到预训练的编码器，用于对At-GC进行分类，揭示了At-GC\的拓扑特征对图像样本类别的依赖性，(2)构建了一个评分聚合(SA)网络来评估At-GC中每个节点的重要性，从而反映出内核在CNN中的相对重要性。实验结果表明，At-GC的拓扑特征对其构造中使用的样本类别具有依赖性，这表明CNN中的内核显示出不同的组合激活模式，用于处理不同的图像类别，同时,从SA网络获得高分的内核对于特征提取至关重要，而低得分的内核可以在不影响模型性能的情况下被修剪，从而增强CNN的可解释性。
