Audiovisual speech recognition based on a deep convolutional neural network

Audiovisual speech recognition is an emerging research topic.Lipreading is the recognition of what someone is saying using visual information,primarily lip movements.In this study,we created a custom dataset for Indian English linguistics and categorized it into three main categories:(1)audio recognition,(2)visual feature extraction,and(3)combined audio and visual recognition.Audio features were extracted using the mel-frequency cepstral coefficient,and classification was performed using a one-dimension convolutional neural network.Visual feature extraction uses Dlib and then classifies visual speech using a long short-term memory type of recurrent neural networks.Finally,integration was performed using a deep convolutional network.The audio speech of Indian English was successfully recognized with accuracies of 93.67%and 91.53%,respectively,using testing data from 200 epochs.The training accuracy for visual speech recognition using the Indian English dataset was 77.48%and the test accuracy was 76.19%using 60 epochs.After integration,the accuracies of audiovisual speech recognition using the Indian English dataset for training and testing were 94.67%and 91.75%,respectively.

Comparison of enhancement techniques based on neural networks for attenuated voice signal captured by flexible vibration sensors on throats

Wearable flexible sensors attached on the neck have been developed to measure the vibration of vocal cords during speech.However,highfrequency attenuation caused by the frequency response of the flexible sensors and absorption of high-frequency sound by the skin are obstacles to the practical application of these sensors in speech capture based on bone conduction.In this paper,speech enhancement techniques for enhancing the intelligibility of sensor signals are developed and compared.Four kinds of speech enhancement algorithms based on a fully connected neural network(FCNN),a long short-term memory(LSTM),a bidirectional long short-term memory(BLSTM),and a convolutional-recurrent neural network(CRNN)are adopted to enhance the sensor signals,and their performance after deployment on four kinds of edge and cloud platforms is also investigated.Experimental results show that the BLSTM performs best in improving speech quality,but is poorest with regard to hardware deployment.It improves short-time objective intelligibility(STOI)by 0.18 to nearly 0.80,which corresponds to a good intelligibility level,but it introduces latency as well as being a large model.The CRNN,which improves STOI to about 0.75,ranks second among the four neural networks.It is also the only model that is able to achieves real-time processing with all four hardware platforms,demonstrating its great potential for deployment on mobile platforms.To the best of our knowledge,this is one of the first trials to systematically and specifically develop processing techniques for bone-conduction speed signals captured by flexible sensors.The results demonstrate the possibility of realizing a wearable lightweight speech collection system based on flexible vibration sensors and real-time speech enhancement to compensate for high-frequency attenuation.

Speech Enhancement via Residual Dense Generative Adversarial Network

Generative adversarial networks(GANs)are paid more attention to dealing with the end-to-end speech enhancement in recent years.Various GANbased enhancement methods are presented to improve the quality of reconstructed speech.However,the performance of these GAN-based methods is worse than those of masking-based methods.To tackle this problem,we propose speech enhancement method with a residual dense generative adversarial network(RDGAN)contributing to map the log-power spectrum(LPS)of degraded speech to the clean one.In detail,a residual dense block(RDB)architecture is designed to better estimate the LPS of clean speech,which can extract rich local features of LPS through densely connected convolution layers.Meanwhile,sequential RDB connections are incorporated on various scales of LPS.It significantly increases the feature learning flexibility and robustness in the time-frequency domain.Simulations show that the proposed method achieves attractive speech enhancement performance in various acoustic environments.Specifically,in the untrained acoustic test with limited priors,e.g.,unmatched signal-to-noise ratio(SNR)and unmatched noise category,RDGAN can still outperform the existing GAN-based methods and masking-based method in the measures of PESQ and other evaluation indexes.It indicates that our method is more generalized in untrained conditions.

Mobile Communication Voice Enhancement Under Convolutional Neural Networks and the Internet of Things

This study aims to reduce the interference of ambient noise in mobile communication,improve the accuracy and authenticity of information transmitted by sound,and guarantee the accuracy of voice information deliv-ered by mobile communication.First,the principles and techniques of speech enhancement are analyzed,and a fast lateral recursive least square method(FLRLS method)is adopted to process sound data.Then,the convolutional neural networks(CNNs)-based noise recognition CNN(NR-CNN)algorithm and speech enhancement model are proposed.Finally,related experiments are designed to verify the performance of the proposed algorithm and model.The experimental results show that the noise classification accuracy of the NR-CNN noise recognition algorithm is higher than 99.82%,and the recall rate and F1 value are also higher than 99.92.The proposed sound enhance-ment model can effectively enhance the original sound in the case of noise interference.After the CNN is incorporated,the average value of all noisy sound perception quality evaluation system values is improved by over 21%compared with that of the traditional noise reduction method.The proposed algorithm can adapt to a variety of voice environments and can simultaneously enhance and reduce noise processing on a variety of different types of voice signals,and the processing effect is better than that of traditional sound enhancement models.In addition,the sound distortion index of the proposed speech enhancement model is inferior to that of the control group,indicating that the addition of the CNN neural network is less likely to cause sound signal distortion in various sound environments and shows superior robustness.In summary,the proposed CNN-based speech enhancement model shows significant sound enhancement effects,stable performance,and strong adapt-ability.This study provides a reference and basis for research applying neural networks in speech enhancement.

Using Hybrid Penalty and Gated Linear Units to Improve Wasserstein Generative Adversarial Networks for Single-Channel Speech Enhancement

Recently,speech enhancement methods based on Generative Adversarial Networks have achieved good performance in time-domain noisy signals.However,the training of Generative Adversarial Networks has such problems as convergence difficulty,model collapse,etc.In this work,an end-to-end speech enhancement model based on Wasserstein Generative Adversarial Networks is proposed,and some improvements have been made in order to get faster convergence speed and better generated speech quality.Specifically,in the generator coding part,each convolution layer adopts different convolution kernel sizes to conduct convolution operations for obtaining speech coding information from multiple scales;a gated linear unit is introduced to alleviate the vanishing gradient problem with the increase of network depth;the gradient penalty of the discriminator is replaced with spectral normalization to accelerate the convergence rate of themodel;a hybrid penalty termcomposed of L1 regularization and a scale-invariant signal-to-distortion ratio is introduced into the loss function of the generator to improve the quality of generated speech.The experimental results on both TIMIT corpus and Tibetan corpus show that the proposed model improves the speech quality significantly and accelerates the convergence speed of the model.

Enhanced Marathi Speech Recognition Facilitated by Grasshopper Optimisation-Based Recurrent Neural Network

Communication is a significant part of being human and living in the world.Diverse kinds of languages and their variations are there;thus,one person can speak any language and cannot effectively communicate with one who speaks that language in a different accent.Numerous application fields such as education,mobility,smart systems,security,and health care systems utilize the speech or voice recognition models abundantly.Though,various studies are focused on the Arabic or Asian and English languages by ignoring other significant languages like Marathi that leads to the broader research motivations in regional languages.It is necessary to understand the speech recognition field,in which the major concentrated stages are feature extraction and classification.This paper emphasis developing a Speech Recognition model for the Marathi language by optimizing Recurrent Neural Network(RNN).Here,the preprocessing of the input signal is performed by smoothing and median filtering.After preprocessing the feature extraction is carried out using MFCC and Spectral features to get precise features from the input Marathi Speech corpus.The optimized RNN classifier is used for speech recognition after completing the feature extraction task,where the optimization of hidden neurons in RNN is performed by the Grasshopper Optimization Algorithm(GOA).Finally,the comparison with the conventional techniques has shown that the proposed model outperforms most competing models on a benchmark dataset.

Application of graph neural network and feature information enhancement in relation inference of sparse knowledge graph

At present,knowledge embedding methods are widely used in the field of knowledge graph(KG)reasoning,and have been successfully applied to those with large entities and relationships.However,in research and production environments,there are a large number of KGs with a small number of entities and relations,which are called sparse KGs.Limited by the performance of knowledge extraction methods or some other reasons(some common-sense information does not appear in the natural corpus),the relation between entities is often incomplete.To solve this problem,a method of the graph neural network and information enhancement is proposed.The improved method increases the mean reciprocal rank(MRR)and Hit@3 by 1.6%and 1.7%,respectively,when the sparsity of the FB15K-237 dataset is 10%.When the sparsity is 50%,the evaluation indexes MRR and Hit@10 are increased by 0.8%and 1.8%,respectively.

Structural reliability analysis using enhanced cuckoo search algorithm and artificial neural network

The present study proposed an enhanced cuckoo search(ECS) algorithm combined with artificial neural network(ANN) as the surrogate model to solve structural reliability problems. In order to enhance the accuracy and convergence rate of the original cuckoo search(CS) algorithm, the main parameters namely, abandon probability of worst nests paand search step sizeα0 are dynamically adjusted via nonlinear control equations. In addition, a global-best guided equation incorporating the information of global best nest is introduced to the ECS to enhance its exploitation. Then, the proposed ECS is linked to the well-trained ANN model for structural reliability analysis. The computational capability of the proposed algorithm is validated using five typical structural reliability problems and an engineering application. The comparison results show the efficiency and accuracy of the proposed algorithm.

A Study on Enhancing Chip Detection Efficiency Using the Lightweight Van-YOLOv8 Network

In pursuit of cost-effective manufacturing,enterprises are increasingly adopting the practice of utilizing recycled semiconductor chips.To ensure consistent chip orientation during packaging,a circular marker on the front side is employed for pin alignment following successful functional testing.However,recycled chips often exhibit substantial surface wear,and the identification of the relatively small marker proves challenging.Moreover,the complexity of generic target detection algorithms hampers seamless deployment.Addressing these issues,this paper introduces a lightweight YOLOv8s-based network tailored for detecting markings on recycled chips,termed Van-YOLOv8.Initially,to alleviate the influence of diminutive,low-resolution markings on the precision of deep learning models,we utilize an upscaling approach for enhanced resolution.This technique relies on the Super-Resolution Generative Adversarial Network with Extended Training(SRGANext)network,facilitating the reconstruction of high-fidelity images that align with input specifications.Subsequently,we replace the original YOLOv8smodel’s backbone feature extraction network with the lightweight VanillaNetwork(VanillaNet),simplifying the branch structure to reduce network parameters.Finally,a Hybrid Attention Mechanism(HAM)is implemented to capture essential details from input images,improving feature representation while concurrently expediting model inference speed.Experimental results demonstrate that the Van-YOLOv8 network outperforms the original YOLOv8s on a recycled chip dataset in various aspects.Significantly,it demonstrates superiority in parameter count,computational intricacy,precision in identifying targets,and speed when compared to certain prevalent algorithms in the current landscape.The proposed approach proves promising for real-time detection of recycled chips in practical factory settings.

Enhancing Human Action Recognition with Adaptive Hybrid Deep Attentive Networks and Archerfish Optimization

In recent years,wearable devices-based Human Activity Recognition(HAR)models have received significant attention.Previously developed HAR models use hand-crafted features to recognize human activities,leading to the extraction of basic features.The images captured by wearable sensors contain advanced features,allowing them to be analyzed by deep learning algorithms to enhance the detection and recognition of human actions.Poor lighting and limited sensor capabilities can impact data quality,making the recognition of human actions a challenging task.The unimodal-based HAR approaches are not suitable in a real-time environment.Therefore,an updated HAR model is developed using multiple types of data and an advanced deep-learning approach.Firstly,the required signals and sensor data are accumulated from the standard databases.From these signals,the wave features are retrieved.Then the extracted wave features and sensor data are given as the input to recognize the human activity.An Adaptive Hybrid Deep Attentive Network(AHDAN)is developed by incorporating a“1D Convolutional Neural Network(1DCNN)”with a“Gated Recurrent Unit(GRU)”for the human activity recognition process.Additionally,the Enhanced Archerfish Hunting Optimizer(EAHO)is suggested to fine-tune the network parameters for enhancing the recognition process.An experimental evaluation is performed on various deep learning networks and heuristic algorithms to confirm the effectiveness of the proposed HAR model.The EAHO-based HAR model outperforms traditional deep learning networks with an accuracy of 95.36,95.25 for recall,95.48 for specificity,and 95.47 for precision,respectively.The result proved that the developed model is effective in recognizing human action by taking less time.Additionally,it reduces the computation complexity and overfitting issue through using an optimization approach.

Contrast-enhanced ultrasonography parameters in neural network diagnosis of liver tumors

AIM:To study the role of time-intensity curve(TIC) analysis parameters in a complex system of neural networks designed to classify liver tumors.METHODS:We prospectively included 112 patients with hepatocellular carcinoma(HCC)(n = 41),hypervascular(n = 20) and hypovascular(n = 12) liver metastases,hepatic hemangiomas(n = 16) or focal fatty changes(n = 23) who underwent contrast-enhanced ultrasonography in the Research Center of Gastroenterology and Hepatology,Craiova,Romania.We recorded full length movies of all contrast uptake phases and post-processed them offline by selecting two areas of interest(one for the tumor and one for the healthy surrounding parenchyma) and consecutive TIC analysis.The difference in maximum intensities,the time to reaching them and the aspect of the late/portal phase,as quantified by the neural network and a ratio between median intensities of the central and peripheral areas were analyzed by a feed forward back propagation multi-layer neural network which was trained to classify data into five distinct classes,corresponding to each type of liver lesion.RESULTS:The neural network had 94.45% training accuracy(95% CI:89.31%-97.21%) and 87.12% testing accuracy(95% CI:86.83%-93.17%).The automatic classification process registered 93.2% sensitivity,89.7% specificity,94.42% positive predictive value and 87.57% negative predictive value.The artificial neural networks(ANN) incorrectly classified as hemangyomas three HCC cases and two hypervascular metastases,while in turn misclassifying four liver hemangyomas as HCC(one case) and hypervascular metastases(three cases).Comparatively,human interpretation of TICs showed 94.1% sensitivity,90.7% specificity,95.11% positive predictive value and 88.89% negative predictive value.The accuracy and specificity of the ANN diagnosis system was similar to that of human interpretation of the TICs(P = 0.225 and P = 0.451,respectively).Hepatocellular carcinoma cases showed contrast uptake during the arterial phase followed by wash-out in the portal and first seconds of the late phases.For the hypovascular metastases did not show significant contrast uptake during the arterial phase,which resulted in negative differences between the maximum intensities.We registered wash-out in the late phase for most of the hypervascular metastases.Liver hemangiomas had contrast uptake in the arterial phase without agent wash-out in the portallate phases.The focal fatty changes did not show any differences from surrounding liver parenchyma,resulting in similar TIC patterns and extracted parameters.CONCLUSION:Neural network analysis of contrastenhanced ultrasonography-obtained TICs seems a promising field of development for future techniques,providing fast and reliable diagnostic aid for the clinician.

Artificial Intelligence for Speech Recognition Based on Neural Networks

Speech recognition or speech to text includes capturing and digitizing the sound waves, transformation of basic linguistic units or phonemes, constructing words from phonemes and contextually analyzing the words to ensure the correct spelling of words that sounds the same. Approach: Studying the possibility of designing a software system using one of the techniques of artificial intelligence applications neuron networks where this system is able to distinguish the sound signals and neural networks of irregular users. Fixed weights are trained on those forms first and then the system gives the output match for each of these formats and high speed. The proposed neural network study is based on solutions of speech recognition tasks, detecting signals using angular modulation and detection of modulated techniques.

DNN-Based Speech Enhancement Using Soft Audible Noise Masking for Wind Noise Reduction

This paper presents a deep neural network(DNN)-based speech enhancement algorithm based on the soft audible noise masking for the single-channel wind noise reduction. To reduce the low-frequency residual noise, the psychoacoustic model is adopted to calculate the masking threshold from the estimated clean speech spectrum. The gain for noise suppression is obtained based on soft audible noise masking by comparing the estimated wind noise spectrum with the masking threshold. To deal with the abruptly time-varying noisy signals, two separate DNN models are utilized to estimate the spectra of clean speech and wind noise components. Experimental results on the subjective and objective quality tests show that the proposed algorithm achieves the better performance compared with the conventional DNN-based wind noise reduction method.

Facial Expression Recognition Using Enhanced Convolution Neural Network with Attention Mechanism

Facial Expression Recognition(FER)has been an interesting area of research in places where there is human-computer interaction.Human psychol-ogy,emotions and behaviors can be analyzed in FER.Classifiers used in FER have been perfect on normal faces but have been found to be constrained in occluded faces.Recently,Deep Learning Techniques(DLT)have gained popular-ity in applications of real-world problems including recognition of human emo-tions.The human face reflects emotional states and human intentions.An expression is the most natural and powerful way of communicating non-verbally.Systems which form communications between the two are termed Human Machine Interaction(HMI)systems.FER can improve HMI systems as human expressions convey useful information to an observer.This paper proposes a FER scheme called EECNN(Enhanced Convolution Neural Network with Atten-tion mechanism)to recognize seven types of human emotions with satisfying results in its experiments.Proposed EECNN achieved 89.8%accuracy in classi-fying the images.

Diagnosis of Leukemia Disease Based on Enhanced Virtual Neural Network

White Blood Cell(WBC)cancer or leukemia is one of the serious cancers that threaten the existence of human beings.In spite of its prevalence and serious consequences,it is mostly diagnosed through manual practices.The risks of inappropriate,sub-standard and wrong or biased diagnosis are high in manual methods.So,there is a need exists for automatic diagnosis and classification method that can replace the manual process.Leukemia is mainly classified into acute and chronic types.The current research work proposed a computer-based application to classify the disease.In the feature extraction stage,we use excellent physical properties to improve the diagnostic system’s accuracy,based on Enhanced Color Co-Occurrence Matrix.The study is aimed at identification and classification of chronic lymphocytic leukemia using microscopic images of WBCs based on Enhanced Virtual Neural Network(EVNN)classification.The proposed method achieved optimum accuracy in detection and classification of leukemia from WBC images.Thus,the study results establish the superiority of the proposed method in automated diagnosis of leukemia.The values achieved by the proposed method in terms of sensitivity,specificity,accuracy,and error rate were 97.8%,89.9%,76.6%,and 2.2%,respectively.Furthermore,the system could predict the disease in prior through images,and the probabilities of disease detection are also highly optimistic.

A Novel Forgery Detection in Image Frames of the Videos Using Enhanced Convolutional Neural Network in Face Images

Different devices in the recent era generated a vast amount of digital video.Generally,it has been seen in recent years that people are forging the video to use it as proof of evidence in the court of justice.Many kinds of researches on forensic detection have been presented,and it provides less accuracy.This paper proposed a novel forgery detection technique in image frames of the videos using enhanced Convolutional Neural Network(CNN).In the initial stage,the input video is taken as of the dataset and then converts the videos into image frames.Next,perform pre-sampling using the Adaptive Rood Pattern Search(ARPS)algorithm intended for reducing the useless frames.In the next stage,perform preprocessing for enhancing the image frames.Then,face detection is done as of the image utilizing the Viola-Jones algorithm.Finally,the improved Crow Search Algorithm(ICSA)has been used to select the extorted features and inputted to the Enhanced Convolutional Neural Network(ECNN)classifier for detecting the forged image frames.The experimental outcome of the proposed system has achieved 97.21%accuracy compared to other existing methods.

ENDPOINT DETECTOR OF NOISY SPEECH SIGNAL USING A RECURRENT NEURAL NETWORK

ＩｎｔｒｏｄｕｃｔｉｏｎＥｎｄｐｏｉｎｔｄｅｔｅｃｔｉｏｎｏｆｓｐｅｅｃｈｓｉｇｎａｌｉｓｉｍｐｏｒｔａｎｔｉｎｍａｎｙａｒｅａｓｏｆｓｐｅｅｃｈｐｒｏｃｅｓｓｉｎｇｔｅｃｈｎｏｌｏｇｙ，ｓｕｃｈａｓｓｐｅｅｃｈｅｎｈａｎｃｅｍｅｎｔ，ｓｐｅｅｃｈｒ...

A SPEECH RECOGNITION METHOD USING COMPETITIVE AND SELECTIVE LEARNING NEURAL NETWORKS

On the basis of asymptotic theory of Gersho, the isodistortion principle of vector clustering was discussed and a kind of competitive and selective learning method (CSL) which may avoid local optimization and have excellent result in application to clusters of HMM model was also proposed. In combining the parallel, self organizational hierarchical neural networks (PSHNN) to reclassify the scores of every form output by HMM, the CSL speech recognition rate is obviously elevated.

Adversarial Examples Protect Your Privacy on Speech Enhancement System

Speech is easily leaked imperceptibly.When people use their phones,the personal voice assistant is constantly listening and waiting to be activated.Private content in speech may be maliciously extracted through automatic speech recognition(ASR)technology by some applications on phone devices.To guarantee that the recognized speech content is accurate,speech enhancement technology is used to denoise the input speech.Speech enhancement technology has developed rapidly along with deep neural networks(DNNs),but adversarial examples can cause DNNs to fail.Considering that the vulnerability of DNN can be used to protect the privacy in speech.In this work,we propose an adversarial method to degrade speech enhancement systems,which can prevent the malicious extraction of private information in speech.Experimental results show that the generated enhanced adversarial examples can be removed most content of the target speech or replaced with target speech content by speech enhancement.The word error rate(WER)between the enhanced original example and enhanced adversarial example recognition result can reach 89.0%.WER of target attack between enhanced adversarial example and target example is low at 33.75%.The adversarial perturbation in the adversarial example can bring much more change than itself.The rate of difference between two enhanced examples and adversarial perturbation can reach more than 1.4430.Meanwhile,the transferability between different speech enhancement models is also investigated.The low transferability of the method can be used to ensure the content in the adversarial example is not damaged,the useful information can be extracted by the friendly ASR.This work can prevent the malicious extraction of speech.

An Optimal Method for Speech Recognition Based on Neural Network

Natural language processing technologies have become more widely available in recent years,making them more useful in everyday situations.Machine learning systems that employ accessible datasets and corporate work to serve the whole spectrum of problems addressed in computational linguistics have lately yielded a number of promising breakthroughs.These methods were particularly advantageous for regional languages,as they were provided with cut-ting-edge language processing tools as soon as the requisite corporate information was generated.The bulk of modern people are unconcerned about the importance of reading.Reading aloud,on the other hand,is an effective technique for nour-ishing feelings as well as a necessary skill in the learning process.This paper pro-posed a novel approach for speech recognition based on neural networks.The attention mechanism isfirst utilized to determine the speech accuracy andfluency assessments,with the spectrum map as the feature extraction input.To increase phoneme identification accuracy,reading precision,for example,employs a new type of deep speech.It makes use of the exportchapter tool,which provides a corpus,as well as the TensorFlow framework in the experimental setting.The experimentalfindings reveal that the suggested model can more effectively assess spoken speech accuracy and readingfluency than the old model,and its evalua-tion model’s score outcomes are more accurate.