Audiovisual speech recognition based on a deep convolutional neural network

Audiovisual speech recognition is an emerging research topic.Lipreading is the recognition of what someone is saying using visual information,primarily lip movements.In this study,we created a custom dataset for Indian English linguistics and categorized it into three main categories:(1)audio recognition,(2)visual feature extraction,and(3)combined audio and visual recognition.Audio features were extracted using the mel-frequency cepstral coefficient,and classification was performed using a one-dimension convolutional neural network.Visual feature extraction uses Dlib and then classifies visual speech using a long short-term memory type of recurrent neural networks.Finally,integration was performed using a deep convolutional network.The audio speech of Indian English was successfully recognized with accuracies of 93.67%and 91.53%,respectively,using testing data from 200 epochs.The training accuracy for visual speech recognition using the Indian English dataset was 77.48%and the test accuracy was 76.19%using 60 epochs.After integration,the accuracies of audiovisual speech recognition using the Indian English dataset for training and testing were 94.67%and 91.75%,respectively.

Remaining Useful Life Prediction of Aeroengine Based on Principal Component Analysis and One-Dimensional Convolutional Neural Network

In order to directly construct the mapping between multiple state parameters and remaining useful life(RUL),and reduce the interference of random error on prediction accuracy,a RUL prediction model of aeroengine based on principal component analysis(PCA)and one-dimensional convolution neural network(1D-CNN)is proposed in this paper.Firstly,multiple state parameters corresponding to massive cycles of aeroengine are collected and brought into PCA for dimensionality reduction,and principal components are extracted for further time series prediction.Secondly,the 1D-CNN model is constructed to directly study the mapping between principal components and RUL.Multiple convolution and pooling operations are applied for deep feature extraction,and the end-to-end RUL prediction of aeroengine can be realized.Experimental results show that the most effective principal component from the multiple state parameters can be obtained by PCA,and the long time series of multiple state parameters can be directly mapped to RUL by 1D-CNN,so as to improve the efficiency and accuracy of RUL prediction.Compared with other traditional models,the proposed method also has lower prediction error and better robustness.

Bearings Intelligent Fault Diagnosis by 1-D Adder Neural Networks

Integrated with sensors,processors,and radio frequency(RF)communication modules,intelligent bearing could achieve the autonomous perception and autonomous decision-making,guarantying the safety and reliability during their use.However,because of the resource limitations of the end device,processors in the intelligent bearing are unable to carry the computational load of deep learning models like convolutional neural network(CNN),which involves a great amount of multiplicative operations.To minimize the computation cost of the conventional CNN,based on the idea of AdderNet,a 1-D adder neural network with a wide first-layer kernel(WAddNN)suitable for bearing fault diagnosis is proposed in this paper.The proposed method uses the l1-norm distance between filters and input features as the output response,thus making the whole network almost free of multiplicative operations.The whole model takes the original signal as the input,uses a wide kernel in the first adder layer to extract features and suppress the high frequency noise,and then uses two layers of small kernels for nonlinear mapping.Through experimental comparison with CNN models of the same structure,WAddNN is able to achieve a similar accuracy as CNN models with significantly reduced computational cost.The proposed model provides a new fault diagnosis method for intelligent bearings with limited resources.

基于1DCNN-GRU的启闭机液压系统故障诊断

由于启闭机液压系统内部结构复杂,故障信号不易采集,使用AMESim软件搭建启闭机液压系统仿真模型,构建6种典型故障数据集。基于这些数据集,提出一维卷积神经网络(1 Dimensional Convolutional Neural Network,1DCNN)与门控循环单元(Gated Recurrent Unit,GRU)相结合的故障诊断方法,利用1DCNN提取信号数据的空间特征和GRU提取信号数据的时间特征,实现对信号数据空间及时间特征的融合,并对融合特征进行分类识别。

Risk Assessment and Prediction of Construction Project Based on 1D-CNN-Attention-BP

In order to solve the problem of low accuracy of construction project duration prediction, this paper proposes a CNN attention BP combination model project risk prediction model based on attention mechanism, one-dimensional convolutional neural network (1d-cnn) and BP neural network. Firstly, the literature analysis method is used to select the risk evaluation index value of construction project, and the attention mechanism is used to determine the weight of risk factors on construction period prediction;then, BP neural network is used to predict the project duration, and accuracy, cross entropy loss function and F1 score are selected to comprehensively evaluate the performance of 1d-cnn-attention-bp combined model. The experimental results show that the duration risk prediction accuracy of the risk prediction model proposed in this paper is more than 90%, which can meet the risk prediction of construction projects with high accuracy.

基于LSTM与1DCNN的导弹轨迹预测方法

针对弹道导弹等超远程攻击目标的轨迹难以预测的问题,提出一种基于长短期记忆(long short-term memory,LSTM)网络与一维卷积神经网络(1-dimensional convolutional neural network,1DCNN)的目标轨迹预测方法。首先,建立三自由度导弹运动模型,依据再入类型设计3种目标轨迹数据,构建机动数据库,解决轨迹数据的来源问题。其次,采用重复分割与滑动窗口的方法对轨迹数据进行预处理。然后,基于LSTM与1DCNN设计了一种目标类型分类网络,对目标进行初步分类。最后,基于1DCNN设计轨迹预测网络,对目标轨迹进行预测。仿真结果表明,提出的轨迹预测网络能够完成轨迹预测任务,预测误差在合理范围内。

IDSSCNN-XgBoost:Improved Dual-Stream Shallow Convolutional Neural Network Based on Extreme Gradient Boosting Algorithm for Micro Expression Recognition

Micro-expressions(ME)recognition is a complex task that requires advanced techniques to extract informative features fromfacial expressions.Numerous deep neural networks(DNNs)with convolutional structures have been proposed.However,unlike DNNs,shallow convolutional neural networks often outperform deeper models in mitigating overfitting,particularly with small datasets.Still,many of these methods rely on a single feature for recognition,resulting in an insufficient ability to extract highly effective features.To address this limitation,in this paper,an Improved Dual-stream Shallow Convolutional Neural Network based on an Extreme Gradient Boosting Algorithm(IDSSCNN-XgBoost)is introduced for ME Recognition.The proposed method utilizes a dual-stream architecture where motion vectors(temporal features)are extracted using Optical Flow TV-L1 and amplify subtle changes(spatial features)via EulerianVideoMagnification(EVM).These features are processed by IDSSCNN,with an attention mechanism applied to refine the extracted effective features.The outputs are then fused,concatenated,and classified using the XgBoost algorithm.This comprehensive approach significantly improves recognition accuracy by leveraging the strengths of both temporal and spatial information,supported by the robust classification power of XgBoost.The proposed method is evaluated on three publicly available ME databases named Chinese Academy of Sciences Micro-expression Database(CASMEII),Spontaneous Micro-Expression Database(SMICHS),and Spontaneous Actions and Micro-Movements(SAMM).Experimental results indicate that the proposed model can achieve outstanding results compared to recent models.The accuracy results are 79.01%,69.22%,and 68.99%on CASMEII,SMIC-HS,and SAMM,and the F1-score are 75.47%,68.91%,and 63.84%,respectively.The proposed method has the advantage of operational efficiency and less computational time.

轻量级(2+1)D卷积结构的动态手势识别研究

目前,基于卷积神经网络的动态手势识别方法取得了巨大的进展,但神经网络模型具有很大的参数量,计算成本和内存占用较大,很难应用在设备资源有限的场合.以减少计算量和参数量为出发点,提出了一种轻量级(2+1)D卷积结构.该结构在(2+1)D卷积结构的基础上,将其中的3D卷积替换为3D深度可分离卷积,在输出向量维度不变的前提下,进一步减少了(2+1)D卷积结构的计算量和参数量.为了弥补时空特征在表征动态手势上的不足,融合注意力机制模块,专注于对运动特征的提取,结合轻量级(2+1)D卷积结构提取的时空特征,可以更好地表征手势动作.实验结果表明,注意力机制模块的插入,在不增加太多额外计算和空间成本的前提下,进一步提高了模型的识别精度.基于以上结构构建的模型,在20BN-jester、EgoGesture和IsoGD数据集上分别取得了96.62%、91.83%和60.1%的识别精度,模型参数量和浮点计算量分别为5.05M和12.81GFLOPs,相比于其他手势识别模型,计算成本和内存占用大大减少,实时手势识别速度达到每秒70帧.

Enhancing Human Action Recognition with Adaptive Hybrid Deep Attentive Networks and Archerfish Optimization

In recent years,wearable devices-based Human Activity Recognition(HAR)models have received significant attention.Previously developed HAR models use hand-crafted features to recognize human activities,leading to the extraction of basic features.The images captured by wearable sensors contain advanced features,allowing them to be analyzed by deep learning algorithms to enhance the detection and recognition of human actions.Poor lighting and limited sensor capabilities can impact data quality,making the recognition of human actions a challenging task.The unimodal-based HAR approaches are not suitable in a real-time environment.Therefore,an updated HAR model is developed using multiple types of data and an advanced deep-learning approach.Firstly,the required signals and sensor data are accumulated from the standard databases.From these signals,the wave features are retrieved.Then the extracted wave features and sensor data are given as the input to recognize the human activity.An Adaptive Hybrid Deep Attentive Network(AHDAN)is developed by incorporating a“1D Convolutional Neural Network(1DCNN)”with a“Gated Recurrent Unit(GRU)”for the human activity recognition process.Additionally,the Enhanced Archerfish Hunting Optimizer(EAHO)is suggested to fine-tune the network parameters for enhancing the recognition process.An experimental evaluation is performed on various deep learning networks and heuristic algorithms to confirm the effectiveness of the proposed HAR model.The EAHO-based HAR model outperforms traditional deep learning networks with an accuracy of 95.36,95.25 for recall,95.48 for specificity,and 95.47 for precision,respectively.The result proved that the developed model is effective in recognizing human action by taking less time.Additionally,it reduces the computation complexity and overfitting issue through using an optimization approach.

基于1-D CNN的二阶段OFDM系统定时同步方法

针对存在多径干扰的正交频分复用系统的定时同步准确性低的问题,提出基于一维卷积神经网络(1-D CNN)的二阶段OFDM系统定时同步方法。在第一阶段,利用经典互相关方法实现路径特征初始抽取,捕获可分辨路径上的定时辅助同步点;基于定时辅助同步点构建1-D CNN网络学习第二阶段中的定时偏移;最后,结合两阶段处理,获得系统最终的定时同步偏移估计。相比于基于压缩感知的定时同步方法和基于极限学习机的定时同步方法,所研究的二阶段OFDM系统定时同步方法提高了定时同步准确性,并有效地降低计算复杂度与处理延迟。

基于1d-MSCNN+GRU的工业入侵检测方法研究

针对传统机器学习方法对特征依赖大,以及传统卷积神经网络只通过提取重要的局部特征来完成识别分类,收敛速度慢的问题,提出了一维多尺度卷积神经网络和门控循环单元相结合的入侵检测方法。该方法使用一维多尺度卷积神经网络加强对特征的捕捉能力,加快收敛速度,采用门控循环单元把握空间特征,减少通道数量扩张,降低数据维度。使用KDD CUP 99数据集和密西西比州大学的天然气管道的数据集进行仿真实验,结果表明与经典的机器学习分类器相比,该方法具有较高的入侵检测性能和较好的泛化能力。

Lightweight and highly robust memristor-based hybrid neural networks for electroencephalogram signal processing

Memristor-based neuromorphic computing shows great potential for high-speed and high-throughput signal processing applications,such as electroencephalogram(EEG)signal processing.Nonetheless,the size of one-transistor one-resistor(1T1R)memristor arrays is limited by the non-ideality of the devices,which prevents the hardware implementation of large and complex networks.In this work,we propose the depthwise separable convolution and bidirectional gate recurrent unit(DSC-BiGRU)network,a lightweight and highly robust hybrid neural network based on 1T1R arrays that enables efficient processing of EEG signals in the temporal,frequency and spatial domains by hybridizing DSC and BiGRU blocks.The network size is reduced and the network robustness is improved while ensuring the network classification accuracy.In the simulation,the measured non-idealities of the 1T1R array are brought into the network through statistical analysis.Compared with traditional convolutional networks,the network parameters are reduced by 95%and the network classification accuracy is improved by 21%at a 95%array yield rate and 5%tolerable error.This work demonstrates that lightweight and highly robust networks based on memristor arrays hold great promise for applications that rely on low consumption and high efficiency.

Marine aquaculture mapping using GF-1 WFV satellite images and full resolution cascade convolutional neural network

Growing demand for seafood and reduced fishery harvests have raised intensive farming of marine aquaculture in coastal regions,which may cause severe coastal water problems without adequate environmental management.Effective mapping of mariculture areas is essential for the protection of coastal environments.However,due to the limited spatial coverage and complex structures,it is still challenging for traditional methods to accurately extract mariculture areas from medium spatial resolution(MSR)images.To solve this problem,we propose to use the full resolution cascade convolutional neural network(FRCNet),which maintains effective features over the whole training process,to identify mariculture areas from MSR images.Specifically,the FRCNet uses a sequential full resolution neural network as the first-level subnetwork,and gradually aggregates higher-level subnetworks in a cascade way.Meanwhile,we perform a repeated fusion strategy so that features can receive information from different subnetworks simultaneously,leading to rich and representative features.As a result,FRCNet can effectively recognize different kinds of mariculture areas from MSR images.Results show that FRCNet obtained better performance than other classical and recently proposed methods.Our developed methods can provide valuable datasets for large-scale and intelligent modeling of the marine aquaculture management and coastal zone planning.

基于Sentinel-1A影像和一维CNN的中国南方生长季早期作物种类识别

作物的早期识别对粮食安全至关重要。在以往的研究中,中国南方作物早期识别面临的主要挑战包括:1)云层覆盖时间长、地块尺寸小且作物类型丰富;2)缺少高时空分辨率合成孔径雷达(synthetic aperture radar,SAR)数据。欧洲航天局Sentinel-1A(S1A)卫星提供的SAR图像具有12 d的重访周期,空间分辨率达10 m,为中国南方作物早期识别提供了新的机遇。为在作物早期识别中充分利用S1A影像的时间特征,本研究提出一维卷积神经网络(one-dimensional convolutional neural network,1D CNN)的增量训练方法:首先利用生长季内全时间序列数据来训练1D CNN的超参数,称为分类器;然后从生长季内第一次S1A影像获取开始,在每个数据获取时间点输入该点之前(包括该点)生长季内所有数据训练分类器在该点的其他参数。以中国湛江地区2017年生长季为研究实例,分别基于VV、VH和VH+VV,评估不同极化数据在该地区的作物分类效果。为验证该方法的有效性,本研究同时应用经典的随机森林(random forest,RF)模型对研究区进行试验。结果表明:1)基于VH+VV、VH和VV极化数据的分类精度依次降低,其中,基于VH+VV后向散射系数时间序列1D CNN和RF测试结果的Kappa系数最大值分别为0.924和0.916,说明S1A时间序列数据在该地区作物分类任务中有效;2)在研究区域内2017年生长季早期,基于1D CNN和RF的5种作物的F-measure均达到0.85及以上,说明本文所构建的1D CNN在该地区主要作物早期分类任务中有效。研究结果证明,针对中国南方作物早期分类,本研究提出的1D CNN训练方案可行。研究结果可为深度学习在作物早期分类任务中的应用提供参考。

AR-MED共振特征增强的风电齿轮箱故障诊断

针对风电齿轮箱故障时脉冲成分往往淹没在其他频率分量中,早期故障特征难以有效提取的问题,提出一种自回归最小熵解卷积(AR-MED)共振特征增强的风电齿轮箱故障诊断方法,并结合一维卷积神经网络(1DCNN),实现齿轮箱高精度故障诊断。首先,使用共振稀疏分解算法(RSSD)将振动信号分解成含有噪声和谐波成分的高共振分量和含有故障冲击成分的低共振分量;其次,对低共振分量使用自回归最小熵解卷积运算,增强低共振分量中微弱的周期性冲击成分;最后,构建自回归最小熵解卷积共振特征增强的1DCNN模型,将分解得到的谐波分量和周期性冲击分量进行特征融合以及有针对的训练和分类。实验结果表明,与现有故障诊断模型相比,所提方法在提取风电齿轮箱的故障特征信息以及提高故障诊断精度方面具有有效性和优越性。

Reconstruction of pile-up events using a one-dimensional convolutional autoencoder for the NEDA detector array

Pulse pile-up is a problem in nuclear spectroscopy and nuclear reaction studies that occurs when two pulses overlap and distort each other,degrading the quality of energy and timing information.Different methods have been used for pile-up rejection,both digital and analogue,but some pile-up events may contain pulses of interest and need to be reconstructed.The paper proposes a new method for reconstructing pile-up events acquired with a neutron detector array(NEDA)using an one-dimensional convolutional autoencoder(1D-CAE).The datasets for training and testing the 1D-CAE are created from data acquired from the NEDA.The new pile-up signal reconstruction method is evaluated from the point of view of how similar the reconstructed signals are to the original ones.Furthermore,it is analysed considering the result of the neutron-gamma discrimination based on charge comparison,comparing the result obtained from original and reconstructed signals.

3D brain glioma segmentation in MRI through integrating multiple densely connected 2D convolutional neural networks

To overcome the computational burden of processing three-dimensional(3 D)medical scans and the lack of spatial information in two-dimensional(2 D)medical scans,a novel segmentation method was proposed that integrates the segmentation results of three densely connected 2 D convolutional neural networks(2 D-CNNs).In order to combine the lowlevel features and high-level features,we added densely connected blocks in the network structure design so that the low-level features will not be missed as the network layer increases during the learning process.Further,in order to resolve the problems of the blurred boundary of the glioma edema area,we superimposed and fused the T2-weighted fluid-attenuated inversion recovery(FLAIR)modal image and the T2-weighted(T2)modal image to enhance the edema section.For the loss function of network training,we improved the cross-entropy loss function to effectively avoid network over-fitting.On the Multimodal Brain Tumor Image Segmentation Challenge(BraTS)datasets,our method achieves dice similarity coefficient values of 0.84,0.82,and 0.83 on the BraTS2018 training;0.82,0.85,and 0.83 on the BraTS2018 validation;and 0.81,0.78,and 0.83 on the BraTS2013 testing in terms of whole tumors,tumor cores,and enhancing cores,respectively.Experimental results showed that the proposed method achieved promising accuracy and fast processing,demonstrating good potential for clinical medicine.

基于1D-CNN和Bi-LSTM的航空发动机剩余寿命预测

剩余寿命预测对航空发动机的预防性维修有重要指导作用,是保障飞机安全运行,提高维修保障效率的重要手段。一维卷积神经网络(1-dimensional convolutional neural network,1D-CNN)和双向长短时记忆神经网络(Bidirectional long short memory, Bi-LSTM)被应用于航空发动机剩余寿命预测模型。首先,根据工程经验在多状态参数的主成分分析的基础上对退化过程进行随机分布拟合,得到综合性能退化量;然后将多变量时间序列样本和对应的性能退化量代入1D-CNN模型进行回归分析,从而得到性能退化分析模型;再通过Bi-LSTM对性能退化量进行时间序列预测,得到性能退化的未来趋势;最后通过设定性能退化阈值,得到剩余寿命预测结果,从而得到从多状态参数-性能退化分析-性能退化预测-剩余寿命预测的实时动态感知模型。实例分析结果表明,提出的混合模型与其他单一深度学习和传统模型相比,有更低的回归分析误差和退化预测误差,能够得到更准确可靠的剩余寿命预测结果。

改进1D-CNN和LSTM的涡扇发动机剩余寿命预测

针对单一深度学习网络对涡扇发动机退化特征提取不足、超参数选择困难的问题,提出一种改进一维卷积神经网络(1-Dimensional Convolutional Neural Network,1D-CNN)和长短时记忆网络(Long Short-Term Memory,LSTM)的涡扇发动机剩余寿命预测方法。首先,利用相关性、单调性和离散性一系列评价指标对涡扇发动机的多维传感器特征参数进行评价和选择,将综合评价指标高的优选特征参数作为1D-CNN的原始输入特征;然后,通过改进激活函数和Dropout函数来提升1D-CNN的特征提取能力,构建表征发动机退化趋势的一维复合健康指标;最后,利用贝叶斯优化(Bayesian Optimization,BO)的LSTM挖掘一维复合健康指标的时间特征,并实现剩余寿命预测。为验证此方法的预测效果,采用美国国家航空航天局提供的涡扇发动机退化数据集进行剩余寿命预测,实验的均方根误差为14.0402,评分函数值为314.6078。结果表明:相比于单一深度学习方法和传统机器学习方法,该方法不仅能获得较高的剩余寿命预测精度,还能有效解决深度学习模型超参数选择困难的问题。

A Hybrid Deep Learning Scheme for Multi-Channel Sleep Stage Classification

Sleep stage classification plays a significant role in the accurate diagnosis and treatment of sleep-related diseases.This study aims to develop an efficient deep learning based scheme for correctly identifying sleep stages using multi-biological signals such as electroencephalography(EEG),electrocardiogram(ECG),electromyogram(EMG),and electrooculogram(EOG).Most of the prior studies in sleep stage classification focus on hand-crafted feature extraction methods.Traditional hand-crafted feature extraction methods choose features manually from raw data,which is tedious,and these features are limited in their ability to balance efficiency and accuracy.Moreover,most of the existing works on sleep staging are either single channel(a single-lead EEG may not contain enough information)or only EEG signal based which can not reveal more complicated physical features for reliable classification of various sleep stages.This study proposes an approach to combine Convolutional Neural Networks(CNNs)and Gated Recurrent Units(GRUs)that can discover hidden features from multi-biological signal data to recognize the different sleep stages efficiently.In the proposed scheme,the CNN is designed to extract concealed features from the multi-biological signals,and the GRU is employed to automatically learn the transition rules among different sleep stages.After that,the softmax layers are used to classify various sleep stages.The proposed method was tested on two publicly available databases:Sleep Heart Health Study(SHHS)and St.Vincent’s University Hospital/University College Dublin Sleep Apnoea(UCDDB).The experimental results reveal that the proposed model yields better performance compared to state-of-the-art works.Our proposed scheme will assist in building a new system to deal with multi-channel or multi-modal signal processing tasks in various applications.