A Lightweight Convolutional Neural Network with Hierarchical Multi-Scale Feature Fusion for Image Classification

Convolutional neural networks (CNNs) are widely used in image classification tasks, but their increasing model size and computation make them challenging to implement on embedded systems with constrained hardware resources. To address this issue, the MobileNetV1 network was developed, which employs depthwise convolution to reduce network complexity. MobileNetV1 employs a stride of 2 in several convolutional layers to decrease the spatial resolution of feature maps, thereby lowering computational costs. However, this stride setting can lead to a loss of spatial information, particularly affecting the detection and representation of smaller objects or finer details in images. To maintain the trade-off between complexity and model performance, a lightweight convolutional neural network with hierarchical multi-scale feature fusion based on the MobileNetV1 network is proposed. The network consists of two main subnetworks. The first subnetwork uses a depthwise dilated separable convolution (DDSC) layer to learn imaging features with fewer parameters, which results in a lightweight and computationally inexpensive network. Furthermore, depthwise dilated convolution in DDSC layer effectively expands the field of view of filters, allowing them to incorporate a larger context. The second subnetwork is a hierarchical multi-scale feature fusion (HMFF) module that uses parallel multi-resolution branches architecture to process the input feature map in order to extract the multi-scale feature information of the input image. Experimental results on the CIFAR-10, Malaria, and KvasirV1 datasets demonstrate that the proposed method is efficient, reducing the network parameters and computational cost by 65.02% and 39.78%, respectively, while maintaining the network performance compared to the MobileNetV1 baseline.

Deepfake Video Detection Based on Improved CapsNet and Temporal–Spatial Features

Rapid development of deepfake technology led to the spread of forged audios and videos across network platforms,presenting risks for numerous countries,societies,and individuals,and posing a serious threat to cyberspace security.To address the problem of insufficient extraction of spatial features and the fact that temporal features are not considered in the deepfake video detection,we propose a detection method based on improved CapsNet and temporal–spatial features(iCapsNet–TSF).First,the dynamic routing algorithm of CapsNet is improved using weight initialization and updating.Then,the optical flow algorithm is used to extract interframe temporal features of the videos to form a dataset of temporal–spatial features.Finally,the iCapsNet model is employed to fully learn the temporal–spatial features of facial videos,and the results are fused.Experimental results show that the detection accuracy of iCapsNet–TSF reaches 94.07%,98.83%,and 98.50%on the Celeb-DF,FaceSwap,and Deepfakes datasets,respectively,displaying a better performance than most existing mainstream algorithms.The iCapsNet–TSF method combines the capsule network and the optical flow algorithm,providing a novel strategy for the deepfake detection,which is of great significance to the prevention of deepfake attacks and the preservation of cyberspace security.

Multi-Scale Feature Extraction for Joint Classification of Hyperspectral and LiDAR Data

With the development of sensors,the application of multi-source remote sensing data has been widely concerned.Since hyperspectral image(HSI)contains rich spectral information while light detection and ranging(LiDAR)data contains elevation information,joint use of them for ground object classification can yield positive results,especially by building deep networks.Fortu-nately,multi-scale deep networks allow to expand the receptive fields of convolution without causing the computational and training problems associated with simply adding more network layers.In this work,a multi-scale feature fusion network is proposed for the joint classification of HSI and LiDAR data.First,we design a multi-scale spatial feature extraction module with cross-channel connections,by which spatial information of HSI data and elevation information of LiDAR data are extracted and fused.In addition,a multi-scale spectral feature extraction module is employed to extract the multi-scale spectral features of HSI data.Finally,joint multi-scale features are obtained by weighting and concatenation operations and then fed into the classifier.To verify the effective-ness of the proposed network,experiments are carried out on the MUUFL Gulfport and Trento datasets.The experimental results demonstrate that the classification performance of the proposed method is superior to that of other state-of-the-art methods.

Attention Guided Multi Scale Feature Fusion Network for Automatic Prostate Segmentation

The precise and automatic segmentation of prostate magnetic resonance imaging(MRI)images is vital for assisting doctors in diagnosing prostate diseases.In recent years,many advanced methods have been applied to prostate segmentation,but due to the variability caused by prostate diseases,automatic segmentation of the prostate presents significant challenges.In this paper,we propose an attention-guided multi-scale feature fusion network(AGMSF-Net)to segment prostate MRI images.We propose an attention mechanism for extracting multi-scale features,and introduce a 3D transformer module to enhance global feature representation by adding it during the transition phase from encoder to decoder.In the decoder stage,a feature fusion module is proposed to obtain global context information.We evaluate our model on MRI images of the prostate acquired from a local hospital.The relative volume difference(RVD)and dice similarity coefficient(DSC)between the results of automatic prostate segmentation and ground truth were 1.21%and 93.68%,respectively.To quantitatively evaluate prostate volume on MRI,which is of significant clinical significance,we propose a unique AGMSF-Net.The essential performance evaluation and validation experiments have demonstrated the effectiveness of our method in automatic prostate segmentation.

Multi-Scale Mixed Attention Tea Shoot Instance Segmentation Model

Tea leaf picking is a crucial stage in tea production that directly influences the quality and value of the tea.Traditional tea-picking machines may compromise the quality of the tea leaves.High-quality teas are often handpicked and need more delicate operations in intelligent picking machines.Compared with traditional image processing techniques,deep learning models have stronger feature extraction capabilities,and better generalization and are more suitable for practical tea shoot harvesting.However,current research mostly focuses on shoot detection and cannot directly accomplish end-to-end shoot segmentation tasks.We propose a tea shoot instance segmentation model based on multi-scale mixed attention(Mask2FusionNet)using a dataset from the tea garden in Hangzhou.We further analyzed the characteristics of the tea shoot dataset,where the proportion of small to medium-sized targets is 89.9%.Our algorithm is compared with several mainstream object segmentation algorithms,and the results demonstrate that our model achieves an accuracy of 82%in recognizing the tea shoots,showing a better performance compared to other models.Through ablation experiments,we found that ResNet50,PointRend strategy,and the Feature Pyramid Network(FPN)architecture can improve performance by 1.6%,1.4%,and 2.4%,respectively.These experiments demonstrated that our proposed multi-scale and point selection strategy optimizes the feature extraction capability for overlapping small targets.The results indicate that the proposed Mask2FusionNet model can perform the shoot segmentation in unstructured environments,realizing the individual distinction of tea shoots,and complete extraction of the shoot edge contours with a segmentation accuracy of 82.0%.The research results can provide algorithmic support for the segmentation and intelligent harvesting of premium tea shoots at different scales.

How do temporal and spectral features .matter in crop classification in Heilongjiang Province, China？

How to fully use spectral and temporal information for efficient identification of crops becomes a crucial issue since each crop has its specific seasonal dynamics. A thorough understanding on the relative usefulness of spectral and temporal features is thus essential for better organization of crop classification information. This study, taking Heilongjiang Province as the study area, aims to use time-series moderate resolution imaging spectroradiometer （MODIS） surface reflectance product （MOD09A1） data to evaluate the importance of spectral and temporal features for crop classification. In doing so, a feature selection strategy based on separability index （SI） was first used to rank the most important spectro-temporal features for crop classification. Ten feature scenarios with different spectral and temporal variable combinations were then devised, which were used for crop classification using the support vector machine and their accuracies were finally assessed with the same crop samples. The results show that the normalized difference tillage index （NDTI）, land surface water index （LSWl） and enhanced vegetation index （EVI） are the most informative spectral features and late August to early September is the most informative temporal window for identifying crops in Heilongjiang for the observed year 2011. Spectral diversity and time variety are both vital for crop classification, and their combined use can improve the accuracy by about 30% in comparison with single image. The feature selection technique based on SI analysis is superior for achieving high crop classification accuracy （producers＇ accuracy of 94.03% and users＇ accuracy of 93.77%） with a small number of features. Increasing temporal resolution is not necessarily important for improving the classification accuracies for crops, and a relatively high classification accuracy can be achieved as long as the images associated with key phenological phrases are retained.

Assessing Landsat-8 and Sentinel-2 spectral-temporal features for mapping tree species of northern plantation forests in Heilongjiang Province,China

Background:Accurate mapping of tree species is highly desired in the management and research of plantation forests,whose ecosystem services are currently under threats.Time-series multispectral satellite images,e.g.,from Landsat-8(L8)and Sentinel-2(S2),have been proven useful in mapping general forest types,yet we do not know quantitatively how their spectral features(e.g.,red-edge)and temporal frequency of data acquisitions(e.g.,16-day vs.5-day)contribute to plantation forest mapping to the species level.Moreover,it is unclear to what extent the fusion of L8 and S2 will result in improvements in tree species mapping of northern plantation forests in China.Methods:We designed three sets of classification experiments(i.e.,single-date,multi-date,and spectral-temporal)to evaluate the performances of L8 and S2 data for mapping keystone timber tree species in northern China.We first used seven pairs of L8 and S2 images to evaluate the performances of L8 and S2 key spectral features for separating these tree species across key growing stages.Then we extracted the spectral-temporal features from all available images of different temporal frequency of data acquisition(i.e.,L8 time series,S2 time series,and fusion of L8 and S2)to assess the contribution of image temporal frequency on the accuracy of tree species mapping in the study area.Results:1)S2 outperformed L8 images in all classification experiments,with or without the red edge bands(0.4%–3.4%and 0.2%–4.4%higher for overall accuracy and macro-F1,respectively);2)NDTI(the ratio of SWIR1 minus SWIR2 to SWIR1 plus SWIR2)and Tasseled Cap coefficients were most important features in all the classifications,and for time-series experiments,the spectral-temporal features of red band-related vegetation indices were most useful;3)increasing the temporal frequency of data acquisition can improve overall accuracy of tree species mapping for up to 3.2%(from 90.1%using single-date imagery to 93.3%using S2 time-series),yet similar overall accuracies were achieved using S2 time-series(93.3%)and the fusion of S2 and L8(93.2%).Conclusions:This study quantifies the contributions of L8 and S2 spectral and temporal features in mapping keystone tree species of northern plantation forests in China and suggests that for mapping tree species in China's northern plantation forests,the effects of increasing the temporal frequency of data acquisition could saturate quickly after using only two images from key phenological stages.

Feature Extraction by Multi-Scale Principal Component Analysis and Classification in Spectral Domain

Feature extraction of signals plays an important role in classification problems because of data dimension reduction property and potential improvement of a classification accuracy rate. Principal component analysis (PCA), wavelets transform or Fourier transform methods are often used for feature extraction. In this paper, we propose a multi-scale PCA, which combines discrete wavelet transform, and PCA for feature extraction of signals in both the spatial and temporal domains. Our study shows that the multi-scale PCA combined with the proposed new classification methods leads to high classification accuracy for the considered signals.

Analysis of Temporal and Spatial Distribution and Large-Scale Circulation Features of Extreme Weather Events in Shanxi Province, China in Recent 30 Years

Extreme weather events such as persistent high temperatures, heavy rains or sudden cold waves in Shanxi Province in China have brought great losses and disasters to people’s production and life. It is of great practical significance to study the temporal and spatial distribution characteristics of extreme weather events and the circulation background field. We selected daily high temperature data (≥35°C), daily minimum temperature data and daily precipitation data (≥50 mm) from 109 meteorological stations in Shanxi Province, China from 1981 to 2010, then set the period in which the temperature is ≥35°C for more than 3 days as a high temperature extreme weather event, define the station in which 24 hour cumulative precipitation is ≥50 mm precipitation on a certain day (20 - 20 hours, Beijing time) as a rainstorm weather, and determine the cold air activity with daily minimum temperature dropped by more than 8°C for 24 hours, or decreased by 10°C for 48 h, and a daily minimum temperature of ≤4°C as a cold weather process. We statistically analyze the temporal and spatial characteristics and trends of high temperature, heavy rain and cold weather and the circulation background field. We count the number of extreme weather events such as persistent high temperatures, heavy rains and cold weather frosts in Shanxi, and analyze the temporal and spatial distribution characteristics, trends and general circulation background of extreme weather events. We analyze and find out the common features of the large-scale circulation background field in various extreme weather events. Through the study of the temporal and spatial distribution characteristics of extreme weather events in Shanxi, including persistent high temperature, heavy rain or sudden cold wave frost weather, we summarize the large-scale circulation characteristics of such extreme weather events. It will provide some reference for future related weather forecasting.

Statistical study on the spatial - temporal distribution features of the arctic sea ice extent

On the basis of the arctic monthly mean sea ice extent data set during 1953-1984, the arctic region is divided into eight subregions,and the analyses of empirical orthogonal functions, power spectrum and maximum entropy spectrum are made to indentify the major spatial and temporal features of the sea ice fluctuations within 32-year period. And then, a brief appropriate physical explanation is tentatively suggested. The results show that both seasonal and non-seasonal variations of the sea ice extent are remarkable, and iis mean annual peripheral positions as well as their interannu-al shifting amplitudes are quite different among all subregions. These features are primarily affected by solar radiation, o-cean circulation, sea surface temperature and maritime-continental contrast, while the non-seasonal variations are most possibly affected by the cosmic-geophysical factors such as earth pole shife, earth rotation oscillation and solar activity.

Fast Face Detection with Multi-Scale Window Search Free from Image Resizing Using SGI Features

Face detection is applied to many tasks such as auto focus control, surveillance, user interface, and face recognition. Processing speed and detection accuracy of the face detection have been improved continuously. This paper describes a novel method of fast face detection with multi-scale window search free from image resizing. We adopt statistics of gradient images (SGI) as image features and append an overlapping cell array to improve detection accuracy. The SGI feature is scale invariant and insensitive to small difference of pixel value. These characteristics enable the multi-scale window search without image resizing. Experimental results show that processing speed of our method is 3.66 times faster than a conventional method, adopting HOG features combined to an SVM classifier, without accuracy degradation.

STGSA:A Novel Spatial-Temporal Graph Synchronous Aggregation Model for Traffic Prediction

The success of intelligent transportation systems relies heavily on accurate traffic prediction,in which how to model the underlying spatial-temporal information from traffic data has come under the spotlight.Most existing frameworks typically utilize separate modules for spatial and temporal correlations modeling.However,this stepwise pattern may limit the effectiveness and efficiency in spatial-temporal feature extraction and cause the overlook of important information in some steps.Furthermore,it is lacking sufficient guidance from prior information while modeling based on a given spatial adjacency graph(e.g.,deriving from the geodesic distance or approximate connectivity),and may not reflect the actual interaction between nodes.To overcome those limitations,our paper proposes a spatial-temporal graph synchronous aggregation(STGSA)model to extract the localized and long-term spatial-temporal dependencies simultaneously.Specifically,a tailored graph aggregation method in the vertex domain is designed to extract spatial and temporal features in one graph convolution process.In each STGSA block,we devise a directed temporal correlation graph to represent the localized and long-term dependencies between nodes,and the potential temporal dependence is further fine-tuned by an adaptive weighting operation.Meanwhile,we construct an elaborated spatial adjacency matrix to represent the road sensor graph by considering both physical distance and node similarity in a datadriven manner.Then,inspired by the multi-head attention mechanism which can jointly emphasize information from different r epresentation subspaces,we construct a multi-stream module based on the STGSA blocks to capture global information.It projects the embedding input repeatedly with multiple different channels.Finally,the predicted values are generated by stacking several multi-stream modules.Extensive experiments are constructed on six real-world datasets,and numerical results show that the proposed STGSA model significantly outperforms the benchmarks.

Incorporation ofκ-carrageenan improves the practical features of agar/konjac glucomannan/κ-carrageenan ternary system

Three materials(agar,konjac glucomannan(KGM)andκ-carrageenan)were used to prepare ternary systems,i.e.,sol-gels and their dried composites conditioned at varied relative humidity(RH)(33%,54%and 75%).Combined methods,e.g.,scanning electron microscopy,small-angle X-ray scattering,infrared spectroscopy(IR)and X-ray diffraction(XRD),were used to disclose howκ-carrageenan addition tailors the features of agar/KGM/κ-carrageenan ternary system.As affirmed by IR and XRD,the ternary systems withκ-carrageenan below 25%(agar/KGM/carrageenan,50:25:25,m/m)displayed proper component interactions,which increased the sol-gel transition temperature and the hardness of obtained gels.For instance,the ternary composites could show hardness about 3 to 4 times higher than that for binary counterpart.These gels were dehydrated to acquire ternary composites.Compared to agar/KGM composite,the ternary composites showed fewer crystallites and nanoscale orders,and newly-formed nanoscale structures from chain assembly.Such multi-scale structures,for composites withκ-carrageenan below 25%,showed weaker changes with RH,as revealed by especially morphologic and crystalline features.Consequently,the ternary composites with lessκ-carrageenan(below 25%)exhibited stabilized elongation at break and hydrophilicity at different RHs.This hints to us that agar/KGM/κ-carrageenan composite systems can display series applications with improved features,e.g.,increased sol-gel transition point.

Feature Fusion-Based Deep Learning Network to Recognize Table Tennis Actions

A system for classifying four basic table tennis strokes using wearable devices and deep learning networks is proposed in this study.The wearable device consisted of a six-axis sensor,Raspberry Pi 3,and a power bank.Multiple kernel sizes were used in convolutional neural network(CNN)to evaluate their performance for extracting features.Moreover,a multiscale CNN with two kernel sizes was used to perform feature fusion at different scales in a concatenated manner.The CNN achieved recognition of the four table tennis strokes.Experimental data were obtained from20 research participants who wore sensors on the back of their hands while performing the four table tennis strokes in a laboratory environment.The data were collected to verify the performance of the proposed models for wearable devices.Finally,the sensor and multi-scale CNN designed in this study achieved accuracy and F1 scores of 99.58%and 99.16%,respectively,for the four strokes.The accuracy for five-fold cross validation was 99.87%.This result also shows that the multi-scale convolutional neural network has better robustness after fivefold cross validation.

RealFuVSR:Feature enhanced real-world video super-resolution

Background Recurrent recovery is a common method for video super-resolution(VSR)that models the correlation between frames via hidden states.However,the application of this structure in real-world scenarios can lead to unsatisfactory artifacts.We found that in real-world VSR training,the use of unknown and complex degradation can better simulate the degradation process in the real world.Methods Based on this,we propose the RealFuVSR model,which simulates real-world degradation and mitigates artifacts caused by the VSR.Specifically,we propose a multiscale feature extraction module(MSF)module that extracts and fuses features from multiple scales,thereby facilitating the elimination of hidden state artifacts.To improve the accuracy of the hidden state alignment information,RealFuVSR uses an advanced optical flow-guided deformable convolution.Moreover,a cascaded residual upsampling module was used to eliminate noise caused by the upsampling process.Results The experiment demonstrates that RealFuVSR model can not only recover high-quality videos but also outperforms the state-of-the-art RealBasicVSR and RealESRGAN models.

Temporally consistent video colorization with deep feature propagation and self-regularization learning

Video colorization is a challenging and highly ill-posed problem.Although recent years have witnessed remarkable progress in single image colorization,there is relatively less research effort on video colorization,and existing methods always suffer from severe flickering artifacts(temporal inconsistency)or unsatisfactory colorization.We address this problem from a new perspective,by jointly considering colorization and temporal consistency in a unified framework.Specifically,we propose a novel temporally consistent video colorization(TCVC)framework.TCVC effectively propagates frame-level deep features in a bidirectional way to enhance the temporal consistency of colorization.Furthermore,TCVC introduces a self-regularization learning(SRL)scheme to minimize the differences in predictions obtained using different time steps.SRL does not require any ground-truth color videos for training and can further improve temporal consistency.Experiments demonstrate that our method can not only provide visually pleasing colorized video,but also with clearly better temporal consistency than state-of-the-art methods.A video demo is provided at https://www.youtube.com/watch?v=c7dczMs-olE,while code is available at https://github.com/lyh-18/TCVC-Tem porally-Consistent-Video-Colorization.

A Soft Sensor with Light and Efficient Multi-scale Feature Method for Multiple Sampling Rates in Industrial Processing

In industrial process control systems,there is overwhelming evidence corroborating the notion that economic or technical limitations result in some key variables that are very difficult to measure online.The data-driven soft sensor is an effective solution because it provides a reliable and stable online estimation of such variables.This paper employs a deep neural network with multiscale feature extraction layers to build soft sensors,which are applied to the benchmarked Tennessee-Eastman process(TEP)and a real wind farm case.The comparison of modelling results demonstrates that the multiscale feature extraction layers have the following advantages over other methods.First,the multiscale feature extraction layers significantly reduce the number of parameters compared to the other deep neural networks.Second,the multiscale feature extraction layers can powerfully extract dataset characteristics.Finally,the multiscale feature extraction layers with fully considered historical measurements can contain richer useful information and improved representation compared to traditional data-driven models.

Grasp Detection with Hierarchical Multi-Scale Feature Fusion and Inverted Shuffle Residual

Grasp detection plays a critical role for robot manipulation.Mainstream pixel-wise grasp detection networks with encoder-decoder structure receive much attention due to good accuracy and efficiency.However,they usually transmit the high-level feature in the encoder to the decoder,and low-level features are neglected.It is noted that low-level features contain abundant detail information,and how to fully exploit low-level features remains unsolved.Meanwhile,the channel information in high-level feature is also not well mined.Inevitably,the performance of grasp detection is degraded.To solve these problems,we propose a grasp detection network with hierarchical multi-scale feature fusion and inverted shuffle residual.Both low-level and high-level features in the encoder are firstly fused by the designed skip connections with attention module,and the fused information is then propagated to corresponding layers of the decoder for in-depth feature fusion.Such a hierarchical fusion guarantees the quality of grasp prediction.Furthermore,an inverted shuffle residual module is created,where the high-level feature from encoder is split in channel and the resultant split features are processed in their respective branches.By such differentiation processing,more high-dimensional channel information is kept,which enhances the representation ability of the network.Besides,an information enhancement module is added before the encoder to reinforce input information.The proposed method attains 98.9%and 97.8%in image-wise and object-wise accuracy on the Cornell grasping dataset,respectively,and the experimental results verify the effectiveness of the method.

Multi-Scale Feature Fusion Model for Bridge Appearance Defect Detection

Although the Faster Region-based Convolutional Neural Network(Faster R-CNN)model has obvious advantages in defect recognition,it still cannot overcome challenging problems,such as time-consuming,small targets,irregular shapes,and strong noise interference in bridge defect detection.To deal with these issues,this paper proposes a novel Multi-scale Feature Fusion(MFF)model for bridge appearance disease detection.First,the Faster R-CNN model adopts Region Of Interest(ROl)pooling,which omits the edge information of the target area,resulting in some missed detections and inaccuracies in both detecting and localizing bridge defects.Therefore,this paper proposes an MFF based on regional feature Aggregation(MFF-A),which reduces the missed detection rate of bridge defect detection and improves the positioning accuracy of the target area.Second,the Faster R-CNN model is insensitive to small targets,irregular shapes,and strong noises in bridge defect detection,which results in a long training time and low recognition accuracy.Accordingly,a novel Lightweight MFF(namely MFF-L)model for bridge appearance defect detection using a lightweight network EfficientNetV2 and a feature pyramid network is proposed,which fuses multi-scale features to shorten the training speed and improve recognition accuracy.Finally,the effectiveness of the proposed method is evaluated on the bridge disease dataset and public computational fluid dynamic dataset.

Bidirectional parallel multi-branch convolution feature pyramid network for target detection in aerial images of swarm UAVs

In this paper,based on a bidirectional parallel multi-branch feature pyramid network(BPMFPN),a novel one-stage object detector called BPMFPN Det is proposed for real-time detection of ground multi-scale targets by swarm unmanned aerial vehicles(UAVs).First,the bidirectional parallel multi-branch convolution modules are used to construct the feature pyramid to enhance the feature expression abilities of different scale feature layers.Next,the feature pyramid is integrated into the single-stage object detection framework to ensure real-time performance.In order to validate the effectiveness of the proposed algorithm,experiments are conducted on four datasets.For the PASCAL VOC dataset,the proposed algorithm achieves the mean average precision(mAP)of 85.4 on the VOC 2007 test set.With regard to the detection in optical remote sensing(DIOR)dataset,the proposed algorithm achieves 73.9 mAP.For vehicle detection in aerial imagery(VEDAI)dataset,the detection accuracy of small land vehicle(slv)targets reaches 97.4 mAP.For unmanned aerial vehicle detection and tracking(UAVDT)dataset,the proposed BPMFPN Det achieves the mAP of 48.75.Compared with the previous state-of-the-art methods,the results obtained by the proposed algorithm are more competitive.The experimental results demonstrate that the proposed algorithm can effectively solve the problem of real-time detection of ground multi-scale targets in aerial images of swarm UAVs.