A Lightweight Convolutional Neural Network with Hierarchical Multi-Scale Feature Fusion for Image Classification

Convolutional neural networks (CNNs) are widely used in image classification tasks, but their increasing model size and computation make them challenging to implement on embedded systems with constrained hardware resources. To address this issue, the MobileNetV1 network was developed, which employs depthwise convolution to reduce network complexity. MobileNetV1 employs a stride of 2 in several convolutional layers to decrease the spatial resolution of feature maps, thereby lowering computational costs. However, this stride setting can lead to a loss of spatial information, particularly affecting the detection and representation of smaller objects or finer details in images. To maintain the trade-off between complexity and model performance, a lightweight convolutional neural network with hierarchical multi-scale feature fusion based on the MobileNetV1 network is proposed. The network consists of two main subnetworks. The first subnetwork uses a depthwise dilated separable convolution (DDSC) layer to learn imaging features with fewer parameters, which results in a lightweight and computationally inexpensive network. Furthermore, depthwise dilated convolution in DDSC layer effectively expands the field of view of filters, allowing them to incorporate a larger context. The second subnetwork is a hierarchical multi-scale feature fusion (HMFF) module that uses parallel multi-resolution branches architecture to process the input feature map in order to extract the multi-scale feature information of the input image. Experimental results on the CIFAR-10, Malaria, and KvasirV1 datasets demonstrate that the proposed method is efficient, reducing the network parameters and computational cost by 65.02% and 39.78%, respectively, while maintaining the network performance compared to the MobileNetV1 baseline.

Attention Guided Multi Scale Feature Fusion Network for Automatic Prostate Segmentation

The precise and automatic segmentation of prostate magnetic resonance imaging(MRI)images is vital for assisting doctors in diagnosing prostate diseases.In recent years,many advanced methods have been applied to prostate segmentation,but due to the variability caused by prostate diseases,automatic segmentation of the prostate presents significant challenges.In this paper,we propose an attention-guided multi-scale feature fusion network(AGMSF-Net)to segment prostate MRI images.We propose an attention mechanism for extracting multi-scale features,and introduce a 3D transformer module to enhance global feature representation by adding it during the transition phase from encoder to decoder.In the decoder stage,a feature fusion module is proposed to obtain global context information.We evaluate our model on MRI images of the prostate acquired from a local hospital.The relative volume difference(RVD)and dice similarity coefficient(DSC)between the results of automatic prostate segmentation and ground truth were 1.21%and 93.68%,respectively.To quantitatively evaluate prostate volume on MRI,which is of significant clinical significance,we propose a unique AGMSF-Net.The essential performance evaluation and validation experiments have demonstrated the effectiveness of our method in automatic prostate segmentation.

Feature Fusion-Based Deep Learning Network to Recognize Table Tennis Actions

A system for classifying four basic table tennis strokes using wearable devices and deep learning networks is proposed in this study.The wearable device consisted of a six-axis sensor,Raspberry Pi 3,and a power bank.Multiple kernel sizes were used in convolutional neural network(CNN)to evaluate their performance for extracting features.Moreover,a multiscale CNN with two kernel sizes was used to perform feature fusion at different scales in a concatenated manner.The CNN achieved recognition of the four table tennis strokes.Experimental data were obtained from20 research participants who wore sensors on the back of their hands while performing the four table tennis strokes in a laboratory environment.The data were collected to verify the performance of the proposed models for wearable devices.Finally,the sensor and multi-scale CNN designed in this study achieved accuracy and F1 scores of 99.58%and 99.16%,respectively,for the four strokes.The accuracy for five-fold cross validation was 99.87%.This result also shows that the multi-scale convolutional neural network has better robustness after fivefold cross validation.

Grasp Detection with Hierarchical Multi-Scale Feature Fusion and Inverted Shuffle Residual

Grasp detection plays a critical role for robot manipulation.Mainstream pixel-wise grasp detection networks with encoder-decoder structure receive much attention due to good accuracy and efficiency.However,they usually transmit the high-level feature in the encoder to the decoder,and low-level features are neglected.It is noted that low-level features contain abundant detail information,and how to fully exploit low-level features remains unsolved.Meanwhile,the channel information in high-level feature is also not well mined.Inevitably,the performance of grasp detection is degraded.To solve these problems,we propose a grasp detection network with hierarchical multi-scale feature fusion and inverted shuffle residual.Both low-level and high-level features in the encoder are firstly fused by the designed skip connections with attention module,and the fused information is then propagated to corresponding layers of the decoder for in-depth feature fusion.Such a hierarchical fusion guarantees the quality of grasp prediction.Furthermore,an inverted shuffle residual module is created,where the high-level feature from encoder is split in channel and the resultant split features are processed in their respective branches.By such differentiation processing,more high-dimensional channel information is kept,which enhances the representation ability of the network.Besides,an information enhancement module is added before the encoder to reinforce input information.The proposed method attains 98.9%and 97.8%in image-wise and object-wise accuracy on the Cornell grasping dataset,respectively,and the experimental results verify the effectiveness of the method.

Multi-Scale Feature Fusion Model for Bridge Appearance Defect Detection

Although the Faster Region-based Convolutional Neural Network(Faster R-CNN)model has obvious advantages in defect recognition,it still cannot overcome challenging problems,such as time-consuming,small targets,irregular shapes,and strong noise interference in bridge defect detection.To deal with these issues,this paper proposes a novel Multi-scale Feature Fusion(MFF)model for bridge appearance disease detection.First,the Faster R-CNN model adopts Region Of Interest(ROl)pooling,which omits the edge information of the target area,resulting in some missed detections and inaccuracies in both detecting and localizing bridge defects.Therefore,this paper proposes an MFF based on regional feature Aggregation(MFF-A),which reduces the missed detection rate of bridge defect detection and improves the positioning accuracy of the target area.Second,the Faster R-CNN model is insensitive to small targets,irregular shapes,and strong noises in bridge defect detection,which results in a long training time and low recognition accuracy.Accordingly,a novel Lightweight MFF(namely MFF-L)model for bridge appearance defect detection using a lightweight network EfficientNetV2 and a feature pyramid network is proposed,which fuses multi-scale features to shorten the training speed and improve recognition accuracy.Finally,the effectiveness of the proposed method is evaluated on the bridge disease dataset and public computational fluid dynamic dataset.

Novel Face Recognition Method by Combining Spatial Domain and Selected Complex Wavelet Features

A novel face recognition method based on fusion of spatial and frequency features was presented to improve recognition accuracy. Dual-Tree Complex Wavelet Transform derives desirable facial features to cope with the variation due to the illumination and facial expression changes. By adopting spectral regression and complex fusion technologies respectively, two improved neighborhood preserving discriminant analysis feature extraction methods were proposed to capture the face manifold structures and locality discriminatory information. Extensive experiments have been made to compare the recognition performance of the proposed method with some popular dimensionality reduction methods on ORL and Yale face databases. The results verify the effectiveness of the proposed method.

Feature Layer Fusion of Linear Features and Empirical Mode Decomposition of Human EMG Signal

To explore the influence of the fusion of different features on recognition,this paper took the electromyography(EMG)signals of rectus femoris under different motions(walk,step,ramp,squat,and sitting)as samples,linear features(time-domain features(variance(VAR)and root mean square(RMS)),frequency-domain features(mean frequency(MF)and mean power frequency(MPF)),and nonlinear features(empirical mode decomposition(EMD))of the samples were extracted.Two feature fusion algorithms,the series splicing method and complex vector method,were designed,which were verified by a double hidden layer(BP)error back propagation neural network.Results show that with the increase of the types and complexity of feature fusions,the recognition rate of the EMG signal to actions is gradually improved.When the EMG signal is used in the series splicing method,the recognition rate of time-domain+frequency-domain+empirical mode decomposition(TD+FD+EMD)splicing is the highest,and the average recognition rate is 92.32%.And this rate is raised to 96.1%by using the complex vector method,and the variance of the BP system is also reduced.

Vehicle color recognition based on smooth modulation neural network with multi-scale feature fusion

Vehicle Color Recognition(VCR)plays a vital role in intelligent traffic management and criminal investigation assistance.However,the existing vehicle color datasets only cover 13 classes,which can not meet the current actual demand.Besides,although lots of efforts are devoted to VCR,they suffer from the problem of class imbalance in datasets.To address these challenges,in this paper,we propose a novel VCR method based on Smooth Modulation Neural Network with Multi-Scale Feature Fusion(SMNN-MSFF).Specifically,to construct the benchmark of model training and evaluation,we first present a new VCR dataset with 24 vehicle classes,Vehicle Color-24,consisting of 10091 vehicle images from a 100-hour urban road surveillance video.Then,to tackle the problem of long-tail distribution and improve the recognition performance,we propose the SMNN-MSFF model with multiscale feature fusion and smooth modulation.The former aims to extract feature information from local to global,and the latter could increase the loss of the images of tail class instances for training with class-imbalance.Finally,comprehensive experimental evaluation on Vehicle Color-24 and previously three representative datasets demonstrate that our proposed SMNN-MSFF outperformed state-of-the-art VCR methods.And extensive ablation studies also demonstrate that each module of our method is effective,especially,the smooth modulation efficiently help feature learning of the minority or tail classes.Vehicle Color-24 and the code of SMNN-MSFF are publicly available and can contact the author to obtain.

Bidirectional parallel multi-branch convolution feature pyramid network for target detection in aerial images of swarm UAVs

In this paper,based on a bidirectional parallel multi-branch feature pyramid network(BPMFPN),a novel one-stage object detector called BPMFPN Det is proposed for real-time detection of ground multi-scale targets by swarm unmanned aerial vehicles(UAVs).First,the bidirectional parallel multi-branch convolution modules are used to construct the feature pyramid to enhance the feature expression abilities of different scale feature layers.Next,the feature pyramid is integrated into the single-stage object detection framework to ensure real-time performance.In order to validate the effectiveness of the proposed algorithm,experiments are conducted on four datasets.For the PASCAL VOC dataset,the proposed algorithm achieves the mean average precision(mAP)of 85.4 on the VOC 2007 test set.With regard to the detection in optical remote sensing(DIOR)dataset,the proposed algorithm achieves 73.9 mAP.For vehicle detection in aerial imagery(VEDAI)dataset,the detection accuracy of small land vehicle(slv)targets reaches 97.4 mAP.For unmanned aerial vehicle detection and tracking(UAVDT)dataset,the proposed BPMFPN Det achieves the mAP of 48.75.Compared with the previous state-of-the-art methods,the results obtained by the proposed algorithm are more competitive.The experimental results demonstrate that the proposed algorithm can effectively solve the problem of real-time detection of ground multi-scale targets in aerial images of swarm UAVs.

Face anti-spoofing based on multi-modal and multi-scale features fusion

Face anti-spoofing is used to assist face recognition system to judge whether the detected face is real face or fake face. In the traditional face anti-spoofing methods, features extracted by hand are used to describe the difference between living face and fraudulent face. But these handmade features do not apply to different variations in an unconstrained environment. The convolutional neural network(CNN) for face deceptions achieves considerable results. However, most existing neural network-based methods simply use neural networks to extract single-scale features from single-modal data, while ignoring multi-scale and multi-modal information. To address this problem, a novel face anti-spoofing method based on multi-modal and multi-scale features fusion(MMFF) is proposed. Specifically, first residual network(Resnet)-34 is adopted to extract features of different scales from each modality, then these features of different scales are fused by feature pyramid network(FPN), finally squeeze-and-excitation fusion(SEF) module and self-attention network(SAN) are combined to fuse features from different modalities for classification. Experiments on the CASIA-SURF dataset show that the new method based on MMFF achieves better performance compared with most existing methods.

Ship recognition based on HRRP via multi-scale sparse preserving method

In order to extract the richer feature information of ship targets from sea clutter, and address the high dimensional data problem, a method termed as multi-scale fusion kernel sparse preserving projection(MSFKSPP) based on the maximum margin criterion(MMC) is proposed for recognizing the class of ship targets utilizing the high-resolution range profile(HRRP). Multi-scale fusion is introduced to capture the local and detailed information in small-scale features, and the global and contour information in large-scale features, offering help to extract the edge information from sea clutter and further improving the target recognition accuracy. The proposed method can maximally preserve the multi-scale fusion sparse of data and maximize the class separability in the reduced dimensionality by reproducing kernel Hilbert space. Experimental results on the measured radar data show that the proposed method can effectively extract the features of ship target from sea clutter, further reduce the feature dimensionality, and improve target recognition performance.

Smart Devices Based Multisensory Approach for Complex Human Activity Recognition

Sensors based Human Activity Recognition(HAR)have numerous applications in eHeath,sports,fitness assessments,ambient assisted living(AAL),human-computer interaction and many more.The human physical activity can be monitored by using wearable sensors or external devices.The usage of external devices has disadvantages in terms of cost,hardware installation,storage,computational time and lighting conditions dependencies.Therefore,most of the researchers used smart devices like smart phones,smart bands and watches which contain various sensors like accelerometer,gyroscope,GPS etc.,and adequate processing capabilities.For the task of recognition,human activities can be broadly categorized as basic and complex human activities.Recognition of complex activities have received very less attention of researchers due to difficulty of problem by using either smart phones or smart watches.Other reasons include lack of sensor-based labeled dataset having several complex human daily life activities.Some of the researchers have worked on the smart phone’s inertial sensors to perform human activity recognition,whereas a few of them used both pocket and wrist positions.In this research,we have proposed a novel framework which is capable to recognize both basic and complex human activities using builtin-sensors of smart phone and smart watch.We have considered 25 physical activities,including 20 complex ones,using smart device’s built-in sensors.To the best of our knowledge,the existing literature consider only up to 15 activities of daily life.

自适应特征融合的复杂道路场景目标检测算法

针对复杂道路场景下密集遮挡目标、小尺度目标检测精度低,容易出现漏检和误检的问题,以YOLOv5算法为网络基础框架,提出了一种自适应特征融合的复杂道路场景目标检测算法。引入特征融合因子,改进相邻尺度特征融合方式,增加各层网络有效样本从而提升中小尺度目标检测能力;增加浅层特征检测层,提升模型小尺度目标的学习能力;改进感受野模块,允许模型自适应选择有效感受野提取目标特征信息;引入Quality Focal Loss改善密集遮挡目标,小尺度目标的定位精度,并在特征融合网络加入注意力机制,提高算法对特征信息的有效利用。实验结果表明,相比原始算法,改进算法在公开数据集BDD100K(10类)、Udacity及自制数据集CQTransport的检测精度分别提高了6.7、4.9、7.9个百分点;在基本不降低检测速度的前提下,能较好提升复杂道路场景下的检测性能,并在一定程度上解决了检测过程中密集遮挡目标、小尺度目标出现的漏检和误检问题。

基于特征融合的无标复句关系识别

无标复句因缺少关联词的辅助,其关系识别为自然语言处理中的一项较为困难的任务。将词性特征融入到词向量中,训练得到含有外部特征的词向量表示,通过组合BERT模型与BiLSTM模型,将字向量、词向量、词性向量结合进行训练,并在特征融合层添加BiLSTM模型捕获的极性特征信息以及CNN模型捕获的依存句法特征信息。实验结果表明,该方法在汉语复句分类上取得了较好的效果,与基准模型相比在宏F1值与微F1值上均有提升,在顶层分类上取得了83.67%的微F1值,在第二层分类上取得了68.28%的微F1值。

一种联合空间约束与差异特征聚合的变化检测网络

变化检测旨在观测地物在不同时序中的表达差异。深度学习已成为实现这一任务的主流手段,现有基于深度学习的遥感变化检测方法中,普遍更专注于对图像中的深度特征进行学习,而忽略了不同层级特征之间语义优势及差距,从而导致检测性能不足。为此,本文提出了一种联合空间约束与差异特征聚合的变化检测网络,通过控制特征信息在网络中的流动,消除检测对象底层特征和高层语义信息之间差异性,提高预测结果的质量。首先,利用孪生网络并结合特征金字塔结构生成多尺度差异特征;然后,使用所提出的坐标自注意力机制(CSAM)对低层特征进行空间约束,强化对变化区域边缘结构及精确位置的学习,并结合经典的卷积注意力模块充分捕捉上下文变化信息;最后,使用门控融合机制提取通道关系,控制多尺度特征的融合,以生成边界清晰、内部完整的变化图像。在变化检测数据集CDD和LEVIR-CD上对本文方法进行了试验,与已有变化检测网络模型进行比较,本文方法在不同场景下均表现出最佳的检测效果。

改进YOLOv4的实验室设备检测算法

针对实验室设备的检测识别问题,提出一种改进YOLOv4算法。针对K-means聚类算法在尺度分布不均匀场景下的局限性,提出一种将数据集标注框按大小划分区间,分别聚类的IK-means++算法;在主干网络中引入通道注意力模块,提出一种阶梯状特征融合网格加强特征融合能力;以计算机实验室为例构建数据集进行训练。实验结果表明,IK-means++算法聚类效果得到有效提升;改进后的YOLOv4算法检测精度更高,模型复杂度更低,速度更快。

复杂交通环境下多层交叉融合多目标检测

针对目前复杂交通环境下还存在多目标检测精度和速度不高等问题,以特征金字塔网络(Feature Pyramid Network,FPN)为基础,提出了一种多层融合多目标检测与识别算法,以提高目标检测精度和网络泛化能力。首先,采用ResNet101的五层架构将空间分辨率上采样2倍构建自上而下的特征图,按照元素相加的方式将上采样图和自下而上的特征图合并,并构建一个融合高层语义信息与低层几何信息的特征层;然后,根据BBox回归存在训练样本不平衡问题,选择Efficient IOU Loss损失函数并结合Focal Loss提出一种改进Focal EIOU Loss;最后,充分考虑复杂交通环境下的实际情况,进行人工标注混合数据集进行训练。该模型在KITTI测试集上的平均检测精度和速度比FPN分别提升了2.4%和5 frame/s,在Cityscale测试集上平均检测精度和速度比FPN提升了1.9%和4 frame/s。

基于密集连接的双能DR图像融合网络

针对单能X射线对复杂工件不同厚度的区域难以同时有效曝光的问题,提出了一种基于密集连接的双能数字射线成像技术图像融合网络(DR-Net)。具体的,DR-Net由过曝光和欠曝光两个子网络组成,分别以过曝光图像和欠曝光图像作为输入,每个子网络包括1个初始特征提取模型(FEB),1个细节特征增强模块(DFE)和3个级联组成的密集连接融合模块(DCF)。FEB对图像进行初步的特征提取,同时DFE提取出图像的更多细节特征,以增强最终融合图像的细节内容。DCF采用带有密集连接的上采样和下采样操作,利用两个子网络模块中学习到的特征,逐步细化和融合图像特征。最后将每个子网络生成的特征重建,并通过平均融合策略得到最终的融合图像。实验结果显示,相比于单一能量下的DR图像,DR-Net融合图像对比度高,能够更加清晰地再现复杂工件的内部结构。从定量分析结果看,DR-Net在NRSS和AG都是最优值,平均数值与对照实验中最优结果相比分别提升了7.06%和21.1%。

Multi-source Remote Sensing Image Registration Based on Contourlet Transform and Multiple Feature Fusion

Image registration is an indispensable component in multi-source remote sensing image processing. In this paper, we put forward a remote sensing image registration method by including an improved multi-scale and multi-direction Harris algorithm and a novel compound feature. Multi-scale circle Gaussian combined invariant moments and multi-direction gray level co-occurrence matrix are extracted as features for image matching. The proposed algorithm is evaluated on numerous multi-source remote sensor images with noise and illumination changes. Extensive experimental studies prove that our proposed method is capable of receiving stable and even distribution of key points as well as obtaining robust and accurate correspondence matches. It is a promising scheme in multi-source remote sensing image registration.

基于改进YOLOv5的人脸口罩佩戴检测

为对商场、车站等复杂环境中的人脸口罩佩戴情况进行检测,综合考虑目标密集、遮挡和小尺度目标等因素,提出一种复杂环境下基于改进YOLOv5的人脸口罩检测方法。引入改进DenseNet(密集连接卷积网络),提高网络特征利用率以及网络抗干扰能力;增加检测头部参数,对不同尺度特征跨级连接,增强多尺度信息交流,提高网络对小尺度目标的检测性能;将原有损失函数GIoU替换为CIoU,解决模型收敛速度慢的问题。实验结果表明,在人脸口罩佩戴检测任务中,改进YOLOv5算法mAP(平均精度均值)为97.8%,较YOLOv5算法与其它主流算法具有更高的检测精度,对实际场景中的人脸口罩检测任务具有现实意义。