A Lightweight Convolutional Neural Network with Hierarchical Multi-Scale Feature Fusion for Image Classification

Convolutional neural networks (CNNs) are widely used in image classification tasks, but their increasing model size and computation make them challenging to implement on embedded systems with constrained hardware resources. To address this issue, the MobileNetV1 network was developed, which employs depthwise convolution to reduce network complexity. MobileNetV1 employs a stride of 2 in several convolutional layers to decrease the spatial resolution of feature maps, thereby lowering computational costs. However, this stride setting can lead to a loss of spatial information, particularly affecting the detection and representation of smaller objects or finer details in images. To maintain the trade-off between complexity and model performance, a lightweight convolutional neural network with hierarchical multi-scale feature fusion based on the MobileNetV1 network is proposed. The network consists of two main subnetworks. The first subnetwork uses a depthwise dilated separable convolution (DDSC) layer to learn imaging features with fewer parameters, which results in a lightweight and computationally inexpensive network. Furthermore, depthwise dilated convolution in DDSC layer effectively expands the field of view of filters, allowing them to incorporate a larger context. The second subnetwork is a hierarchical multi-scale feature fusion (HMFF) module that uses parallel multi-resolution branches architecture to process the input feature map in order to extract the multi-scale feature information of the input image. Experimental results on the CIFAR-10, Malaria, and KvasirV1 datasets demonstrate that the proposed method is efficient, reducing the network parameters and computational cost by 65.02% and 39.78%, respectively, while maintaining the network performance compared to the MobileNetV1 baseline.

Attention Guided Multi Scale Feature Fusion Network for Automatic Prostate Segmentation

The precise and automatic segmentation of prostate magnetic resonance imaging(MRI)images is vital for assisting doctors in diagnosing prostate diseases.In recent years,many advanced methods have been applied to prostate segmentation,but due to the variability caused by prostate diseases,automatic segmentation of the prostate presents significant challenges.In this paper,we propose an attention-guided multi-scale feature fusion network(AGMSF-Net)to segment prostate MRI images.We propose an attention mechanism for extracting multi-scale features,and introduce a 3D transformer module to enhance global feature representation by adding it during the transition phase from encoder to decoder.In the decoder stage,a feature fusion module is proposed to obtain global context information.We evaluate our model on MRI images of the prostate acquired from a local hospital.The relative volume difference(RVD)and dice similarity coefficient(DSC)between the results of automatic prostate segmentation and ground truth were 1.21%and 93.68%,respectively.To quantitatively evaluate prostate volume on MRI,which is of significant clinical significance,we propose a unique AGMSF-Net.The essential performance evaluation and validation experiments have demonstrated the effectiveness of our method in automatic prostate segmentation.

Neighborhood fusion-based hierarchical parallel feature pyramid network for object detection

In order to improve the detection accuracy of small objects,a neighborhood fusion-based hierarchical parallel feature pyramid network(NFPN)is proposed.Unlike the layer-by-layer structure adopted in the feature pyramid network(FPN)and deconvolutional single shot detector(DSSD),where the bottom layer of the feature pyramid network relies on the top layer,NFPN builds the feature pyramid network with no connections between the upper and lower layers.That is,it only fuses shallow features on similar scales.NFPN is highly portable and can be embedded in many models to further boost performance.Extensive experiments on PASCAL VOC 2007,2012,and COCO datasets demonstrate that the NFPN-based SSD without intricate tricks can exceed the DSSD model in terms of detection accuracy and inference speed,especially for small objects,e.g.,4%to 5%higher mAP(mean average precision)than SSD,and 2%to 3%higher mAP than DSSD.On VOC 2007 test set,the NFPN-based SSD with 300×300 input reaches 79.4%mAP at 34.6 frame/s,and the mAP can raise to 82.9%after using the multi-scale testing strategy.

Ship recognition based on HRRP via multi-scale sparse preserving method

In order to extract the richer feature information of ship targets from sea clutter, and address the high dimensional data problem, a method termed as multi-scale fusion kernel sparse preserving projection(MSFKSPP) based on the maximum margin criterion(MMC) is proposed for recognizing the class of ship targets utilizing the high-resolution range profile(HRRP). Multi-scale fusion is introduced to capture the local and detailed information in small-scale features, and the global and contour information in large-scale features, offering help to extract the edge information from sea clutter and further improving the target recognition accuracy. The proposed method can maximally preserve the multi-scale fusion sparse of data and maximize the class separability in the reduced dimensionality by reproducing kernel Hilbert space. Experimental results on the measured radar data show that the proposed method can effectively extract the features of ship target from sea clutter, further reduce the feature dimensionality, and improve target recognition performance.

Feature Fusion-Based Deep Learning Network to Recognize Table Tennis Actions

A system for classifying four basic table tennis strokes using wearable devices and deep learning networks is proposed in this study.The wearable device consisted of a six-axis sensor,Raspberry Pi 3,and a power bank.Multiple kernel sizes were used in convolutional neural network(CNN)to evaluate their performance for extracting features.Moreover,a multiscale CNN with two kernel sizes was used to perform feature fusion at different scales in a concatenated manner.The CNN achieved recognition of the four table tennis strokes.Experimental data were obtained from20 research participants who wore sensors on the back of their hands while performing the four table tennis strokes in a laboratory environment.The data were collected to verify the performance of the proposed models for wearable devices.Finally,the sensor and multi-scale CNN designed in this study achieved accuracy and F1 scores of 99.58%and 99.16%,respectively,for the four strokes.The accuracy for five-fold cross validation was 99.87%.This result also shows that the multi-scale convolutional neural network has better robustness after fivefold cross validation.

Grasp Detection with Hierarchical Multi-Scale Feature Fusion and Inverted Shuffle Residual

Grasp detection plays a critical role for robot manipulation.Mainstream pixel-wise grasp detection networks with encoder-decoder structure receive much attention due to good accuracy and efficiency.However,they usually transmit the high-level feature in the encoder to the decoder,and low-level features are neglected.It is noted that low-level features contain abundant detail information,and how to fully exploit low-level features remains unsolved.Meanwhile,the channel information in high-level feature is also not well mined.Inevitably,the performance of grasp detection is degraded.To solve these problems,we propose a grasp detection network with hierarchical multi-scale feature fusion and inverted shuffle residual.Both low-level and high-level features in the encoder are firstly fused by the designed skip connections with attention module,and the fused information is then propagated to corresponding layers of the decoder for in-depth feature fusion.Such a hierarchical fusion guarantees the quality of grasp prediction.Furthermore,an inverted shuffle residual module is created,where the high-level feature from encoder is split in channel and the resultant split features are processed in their respective branches.By such differentiation processing,more high-dimensional channel information is kept,which enhances the representation ability of the network.Besides,an information enhancement module is added before the encoder to reinforce input information.The proposed method attains 98.9%and 97.8%in image-wise and object-wise accuracy on the Cornell grasping dataset,respectively,and the experimental results verify the effectiveness of the method.

Multi-Scale Feature Fusion Model for Bridge Appearance Defect Detection

Although the Faster Region-based Convolutional Neural Network(Faster R-CNN)model has obvious advantages in defect recognition,it still cannot overcome challenging problems,such as time-consuming,small targets,irregular shapes,and strong noise interference in bridge defect detection.To deal with these issues,this paper proposes a novel Multi-scale Feature Fusion(MFF)model for bridge appearance disease detection.First,the Faster R-CNN model adopts Region Of Interest(ROl)pooling,which omits the edge information of the target area,resulting in some missed detections and inaccuracies in both detecting and localizing bridge defects.Therefore,this paper proposes an MFF based on regional feature Aggregation(MFF-A),which reduces the missed detection rate of bridge defect detection and improves the positioning accuracy of the target area.Second,the Faster R-CNN model is insensitive to small targets,irregular shapes,and strong noises in bridge defect detection,which results in a long training time and low recognition accuracy.Accordingly,a novel Lightweight MFF(namely MFF-L)model for bridge appearance defect detection using a lightweight network EfficientNetV2 and a feature pyramid network is proposed,which fuses multi-scale features to shorten the training speed and improve recognition accuracy.Finally,the effectiveness of the proposed method is evaluated on the bridge disease dataset and public computational fluid dynamic dataset.

Bidirectional parallel multi-branch convolution feature pyramid network for target detection in aerial images of swarm UAVs

In this paper,based on a bidirectional parallel multi-branch feature pyramid network(BPMFPN),a novel one-stage object detector called BPMFPN Det is proposed for real-time detection of ground multi-scale targets by swarm unmanned aerial vehicles(UAVs).First,the bidirectional parallel multi-branch convolution modules are used to construct the feature pyramid to enhance the feature expression abilities of different scale feature layers.Next,the feature pyramid is integrated into the single-stage object detection framework to ensure real-time performance.In order to validate the effectiveness of the proposed algorithm,experiments are conducted on four datasets.For the PASCAL VOC dataset,the proposed algorithm achieves the mean average precision(mAP)of 85.4 on the VOC 2007 test set.With regard to the detection in optical remote sensing(DIOR)dataset,the proposed algorithm achieves 73.9 mAP.For vehicle detection in aerial imagery(VEDAI)dataset,the detection accuracy of small land vehicle(slv)targets reaches 97.4 mAP.For unmanned aerial vehicle detection and tracking(UAVDT)dataset,the proposed BPMFPN Det achieves the mAP of 48.75.Compared with the previous state-of-the-art methods,the results obtained by the proposed algorithm are more competitive.The experimental results demonstrate that the proposed algorithm can effectively solve the problem of real-time detection of ground multi-scale targets in aerial images of swarm UAVs.

Vehicle color recognition based on smooth modulation neural network with multi-scale feature fusion

Vehicle Color Recognition(VCR)plays a vital role in intelligent traffic management and criminal investigation assistance.However,the existing vehicle color datasets only cover 13 classes,which can not meet the current actual demand.Besides,although lots of efforts are devoted to VCR,they suffer from the problem of class imbalance in datasets.To address these challenges,in this paper,we propose a novel VCR method based on Smooth Modulation Neural Network with Multi-Scale Feature Fusion(SMNN-MSFF).Specifically,to construct the benchmark of model training and evaluation,we first present a new VCR dataset with 24 vehicle classes,Vehicle Color-24,consisting of 10091 vehicle images from a 100-hour urban road surveillance video.Then,to tackle the problem of long-tail distribution and improve the recognition performance,we propose the SMNN-MSFF model with multiscale feature fusion and smooth modulation.The former aims to extract feature information from local to global,and the latter could increase the loss of the images of tail class instances for training with class-imbalance.Finally,comprehensive experimental evaluation on Vehicle Color-24 and previously three representative datasets demonstrate that our proposed SMNN-MSFF outperformed state-of-the-art VCR methods.And extensive ablation studies also demonstrate that each module of our method is effective,especially,the smooth modulation efficiently help feature learning of the minority or tail classes.Vehicle Color-24 and the code of SMNN-MSFF are publicly available and can contact the author to obtain.

Face anti-spoofing based on multi-modal and multi-scale features fusion

Face anti-spoofing is used to assist face recognition system to judge whether the detected face is real face or fake face. In the traditional face anti-spoofing methods, features extracted by hand are used to describe the difference between living face and fraudulent face. But these handmade features do not apply to different variations in an unconstrained environment. The convolutional neural network(CNN) for face deceptions achieves considerable results. However, most existing neural network-based methods simply use neural networks to extract single-scale features from single-modal data, while ignoring multi-scale and multi-modal information. To address this problem, a novel face anti-spoofing method based on multi-modal and multi-scale features fusion(MMFF) is proposed. Specifically, first residual network(Resnet)-34 is adopted to extract features of different scales from each modality, then these features of different scales are fused by feature pyramid network(FPN), finally squeeze-and-excitation fusion(SEF) module and self-attention network(SAN) are combined to fuse features from different modalities for classification. Experiments on the CASIA-SURF dataset show that the new method based on MMFF achieves better performance compared with most existing methods.

融合项目特征级信息的稀疏兴趣网络序列推荐

在以往提取多兴趣嵌入的序列推荐模型中仅能通过聚类的方法发现少量兴趣概念,忽视项目交互序列中特征级信息对最终推荐结果的影响。针对此问题,对传统的多兴趣序列推荐模型进行改进,提出一种融合项目特征级信息的稀疏兴趣网络序列推荐模型。实验结果表明,相比其它模型,该模型可以更好捕捉用户的多样化偏好并缓解冷启动问题。在给定数据集上,该模型比传统的序列推荐模型在命中率上平均提高了6.4%,归一化折损累计增益平均提高了8.7%。

seqAFF-ResNet:面向新冠肺炎的诊断模型

新冠肺炎的计算机辅助诊断是一种实现智能化影像诊断、临床诊断及临床分型的方法,在新冠肺炎的辅助诊断过程中,图像的病灶区域与组织边界对比不明显,导致模型不能较好地关注病灶区域,对有效特征的提取不够充分。针对上述问题,提出一个新冠肺炎辅助诊断模型seqAFF-ResNet(sequential attentional feature fusion-residual neural network)。设计串行注意力特征融合(sequential attentional feature fusion,seqAFF)模块,该模块串联条带注意力特征融合(strip attentional feature fusion,SAFF)模块和全局局部注意力特征融合(global local attentional feature fusion,GLAFF)模块,获取图像的纹理信息以及全局和局部信息,弥补卷积神经网络对于细节特征提取能力的不足,使得模型可以更好地关注于病灶区域;构造深浅层特征融合(deep and shallow feature fusion,DSFF)模块,使用深层特征的语义信息来影响浅层信息,同时将浅层的空间信息传入深层特征中,使深浅层特征进行有效融合,捕获丰富的上下文信息,实现跨层注意力特征增强,使网络能够更好地定位病变区域。与残差神经网络(residual neural network,ResNet)相比,seqAFF-ResNet准确率提升了3.42%,精确率提升了3.53%,F1分数提升了2.77%,AUC值提升了0.9%,实验结果表明,所提模型可以提高新冠肺炎的识别准确率,且与同类模型相比具有更好的性能。所提方法为新冠肺炎的辅助诊断提供了有效的识别方法,对新冠肺炎的计算机辅助诊断具有重要意义。

Multi-source Remote Sensing Image Registration Based on Contourlet Transform and Multiple Feature Fusion

Image registration is an indispensable component in multi-source remote sensing image processing. In this paper, we put forward a remote sensing image registration method by including an improved multi-scale and multi-direction Harris algorithm and a novel compound feature. Multi-scale circle Gaussian combined invariant moments and multi-direction gray level co-occurrence matrix are extracted as features for image matching. The proposed algorithm is evaluated on numerous multi-source remote sensor images with noise and illumination changes. Extensive experimental studies prove that our proposed method is capable of receiving stable and even distribution of key points as well as obtaining robust and accurate correspondence matches. It is a promising scheme in multi-source remote sensing image registration.

一种基于DSmT和HMM的序列飞机目标识别算法

针对姿态多变化的飞机自动目标识别中的低识别率问题,提出了一种基于DSm T(Dezert-Smarandache theory)与隐马尔可夫模型(Hidden Markov model,HMM)的飞机多特征序列信息融合识别算法(Multiple features and sequential information fusion,MFSIF).其创新性在于将单幅图像的多特征信息融合识别和序列图像信息融合识别进行有机结合.首先,对图像进行二值化预处理,并提取目标的Hu矩和轮廓局部奇异值特征;然后,利用概率神经网络(Probabilistic neural networks,PNN)构造基本信度赋值(Basic belief assignment,BBA);接着,利用DSm T对该图像的不同特征进行融合,从而获得HMM的观察值序列;再接着,利用隐马尔可夫模型对飞机序列信息融合,计算观察值序列与各隐马尔可夫模型之间的相似度,从而实现姿态多变化的飞机目标自动识别;最后,通过仿真实验,验证了该算法在飞机姿态发生较大变化时,依然可以获得较高的正确识别率,同时在实时性方面也可以满足飞机目标识别的要求.另外,在飞机序列发生连续遮挡帧数τ≤6的情况下,也具有较高的飞机目标正确识别率.

复杂背景下序贯显著性特征海面目标检测算法

根据人眼视觉显著性注意机制,提出基于序贯显著性特征的目标识别算法,把复杂背景下的目标识别问题转化为多特征的融合识别,将目标识别过程分成感兴趣区域检测和感兴趣目标检测两个子过程,并建立序贯显著性特征目标识别算法模型,序贯提取复杂图像区域的显著性特征,逐步缩小感兴趣区域范围,然后以基于D-S证据理论的多特征数据融合方法,通过多特征综合置信度的估计与分析,完成可靠的目标识别.试验研究表明,针对复杂背景下海面舰船目标,在相同的虚警概率下,该算法的检测概率比单一的不变矩算法和基于纹理特征算法平均要高20%.

Industrial Fusion Cascade Detection of Solder Joint

With the remarkable advancements in machine vision research and its ever-expanding applications,scholars have increasingly focused on harnessing various vision methodologies within the industrial realm.Specifically,detecting vehicle floor welding points poses unique challenges,including high operational costs and limited portability in practical settings.To address these challenges,this paper innovatively integrates template matching and the Faster RCNN algorithm,presenting an industrial fusion cascaded solder joint detection algorithm that seamlessly blends template matching with deep learning techniques.This algorithm meticulously weights and fuses the optimized features of both methodologies,enhancing the overall detection capabilities.Furthermore,it introduces an optimized multi-scale and multi-template matching approach,leveraging a diverse array of templates and image pyramid algorithms to bolster the accuracy and resilience of object detection.By integrating deep learning algorithms with this multi-scale and multi-template matching strategy,the cascaded target matching algorithm effectively accurately identifies solder joint types and positions.A comprehensive welding point dataset,labeled by experts specifically for vehicle detection,was constructed based on images from authentic industrial environments to validate the algorithm’s performance.Experiments demonstrate the algorithm’s compelling performance in industrial scenarios,outperforming the single-template matching algorithm by 21.3%,the multi-scale and multitemplate matching algorithm by 3.4%,the Faster RCNN algorithm by 19.7%,and the YOLOv9 algorithm by 17.3%in terms of solder joint detection accuracy.This optimized algorithm exhibits remarkable robustness and portability,ideally suited for detecting solder joints across diverse vehicle workpieces.Notably,this study’s dataset and feature fusion approach can be a valuable resource for other algorithms seeking to enhance their solder joint detection capabilities.This work thus not only presents a novel and effective solution for industrial solder joint detection but lays the groundwork for future advancements in this critical area.

基于Transformer的时序多模态融合特征的异常网络流量检测方法

针对当前卷积和循环神经网络相结合的模型无法有效表征网络流量的问题,提出一种基于Tansformer的时序多模态融合特征的异常网络流量检测方法。首先,以会话为单位切分网络流量,将会话中流量记录的统计特征分组。其次,利用多模态注意力编码器计算特征分组的融合特征,进而利用注意力机制对一个会话的所有流量记录并行建模,挖掘流量记录的时序多模态融合特征。最后,利用全连接层和softmax层对该特征进行线性变换和概率计算。在CIC-ToN-IoT数据集上的实验结果表明该方法切实可行,相较于对比方法,在取得较高的准确率和精度的同时,保持了最低的误报率。

基于序列多尺度特征融合表示的层级舞蹈动作姿态估计方法

人体姿态估计是计算机视觉研究领域的热点研究问题之一,但其在传统民间舞蹈动作姿态估计方面的应用研究尚处于起步阶段.由于舞蹈图像中人体动作复杂多变、舞蹈动作连贯性强、舞蹈者存在严重遮挡不易检测等特点,传统人体姿态估计方法难以准确估计舞蹈者的动作变化,导致舞蹈动作姿态估计准确率较低.针对此问题,本文提出一种基于序列多尺度特征融合表示的层级舞蹈动作姿态估计方法,该方法针对舞蹈动作骨骼关节点尺度变化剧烈的问题,构建基于序列多尺度特征融合表示的关节点估计模型.并且,针对舞蹈姿态形变较大,遮挡严重的问题,设计基于关节点几何关系的层级姿态估计模型,提高舞蹈动作姿态估计的效果.实验结果表明,本文方法在标准人体姿态估计数据集及自建舞蹈数据集上取得较好的姿态估计结果.

基于多任务学习的时序多模态情感分析模型

针对时序多模态情感分析中存在的单模态特征表示和跨模态特征融合问题,结合多头注意力机制,提出一种基于多任务学习的情感分析模型。首先,使用卷积神经网络(CNN)、双向门控循环神经网络(Bi GRU)和多头自注意力(MHSA)实现了对时序单模态的特征表示;然后,利用多头注意力实现跨模态的双向信息融合;最后,基于多任务学习思想,添加额外的情感极性分类和情感强度回归任务作为辅助,从而提升情感评分回归主任务的综合性能。实验结果表明,相较于多模态分解模型,所提模型的二分类准确度指标在CMU-MOSEI和CMU-MOSI多模态数据集上分别提高了7.8个百分点和3.1个百分点。该模型适用于多模态场景下的情感分析问题,能够为商品推荐、股市预测、舆情监控等应用提供决策支持。

融合时序监督和注意力机制的脉络膜新生血管分割

脉络膜新生血管(Choroidal Neovascularization,CNV)一般出现在老年性黄斑变性(Age-related macular degeneration,AMD)晚期,在光学相干断层成像(SD-OCT)中对CNV进行准确分割对AMD的诊疗具有重要意义。文中提出了一种融合时序模型与注意力机制的CNV分割网络。该方法将连续的SD-OCT图像输入分割网络,在编码器部分提取图片多尺度信息,为了更好地提取图片局部特征,又在跳跃连接部分加入注意力门;同时,为了解决分割不连续的问题,在分割网络池化后加入了时序约束网络以构建相邻帧连续性约束,并在损失函数中加入梯度约束以更好地保留病变边界;采用空间金字塔将两部分网络特征图融合以产生分割损失,提高了最终的分割精度。基于患者独立性对12名患者的200组眼睛数据进行实验,该方法的Dice系数为76.3%,overlap达到60.7%,能够在SD-OCT图像中对CNV进行可靠的分割。