Probing Filters to Interpret CNN Semantic Configurations by Occlusion

Deep neural networks has beenwidely used inmany fields,but there are growing concerns about its black-box nature.Previous interpretability studies provide four types of explanations including logical rules,revealing hidden semantics,sensitivity analysis,and providing examples as prototypes.In this paper,an interpretability method is proposed for revealing semantic representations at hidden layers of CNNs through lightweight annotation by occluding.First,visual semantic configurations are defined for a certain class.Then candidate filters whose activations are related to these specified visual semantics are probed by occluding.Finally,lightweight occlusion annotation and a scoring mechanism is used to screen out the filters that recognize these semantics.The method is applied to the datasets of mechanical equipment,animals and clothing images.The proposed method performs well in the experiments assessing interpretability qualitatively and quantitatively.