A feature-wise attention module based on the difference with surrounding features for convolutional neural networks

Attention mechanism has become a widely researched method to improve the performance of convolutional neural networks(CNNs).Most of the researches focus on designing channel-wise and spatial-wise attention modules but neglect the importance of unique information on each feature,which is critical for deciding both“what”and“where”to focus.In this paper,a feature-wise attention module is proposed,which can give each feature of the input feature map an attention weight.Specifically,the module is based on the well-known surround suppression in the discipline of neuroscience,and it consists of two sub-modules,Minus-Square-Add(MSA)operation and a group of learnable non-linear mapping functions.The MSA imitates the surround suppression and defines an energy function which can be applied to each feature to measure its importance.The group of non-linear functions refines the energy calculated by the MSA to more reasonable values.By these two sub-modules,feature-wise attention can be well captured.Meanwhile,due to the simple structure and few parameters of the two sub-modules,the proposed module can easily be almost integrated into any CNN.To verify the performance and effectiveness of the proposed module,several experiments were conducted on the Cifar10,Cifar100,Cinic10,and Tiny-ImageNet datasets,respectively.The experimental results demonstrate that the proposed module is flexible and effective for CNNs to improve their performance.