Japanese Sign Language Recognition by Combining Joint Skeleton-Based Handcrafted and Pixel-Based Deep Learning Features with Machine Learning Classification

Sign language recognition is vital for enhancing communication accessibility among the Deaf and hard-of-hearing communities.In Japan,approximately 360,000 individualswith hearing and speech disabilities rely on Japanese Sign Language(JSL)for communication.However,existing JSL recognition systems have faced significant performance limitations due to inherent complexities.In response to these challenges,we present a novel JSL recognition system that employs a strategic fusion approach,combining joint skeleton-based handcrafted features and pixel-based deep learning features.Our system incorporates two distinct streams:the first stream extracts crucial handcrafted features,emphasizing the capture of hand and body movements within JSL gestures.Simultaneously,a deep learning-based transfer learning stream captures hierarchical representations of JSL gestures in the second stream.Then,we concatenated the critical information of the first stream and the hierarchy of the second stream features to produce the multiple levels of the fusion features,aiming to create a comprehensive representation of the JSL gestures.After reducing the dimensionality of the feature,a feature selection approach and a kernel-based support vector machine(SVM)were used for the classification.To assess the effectiveness of our approach,we conducted extensive experiments on our Lab JSL dataset and a publicly available Arabic sign language(ArSL)dataset.Our results unequivocally demonstrate that our fusion approach significantly enhances JSL recognition accuracy and robustness compared to individual feature sets or traditional recognition methods.

Virtual Keyboard:A Real-Time Hand Gesture Recognition-Based Character Input System Using LSTM and Mediapipe Holistic

In the digital age,non-touch communication technologies are reshaping human-device interactions and raising security concerns.A major challenge in current technology is the misinterpretation of gestures by sensors and cameras,often caused by environmental factors.This issue has spurred the need for advanced data processing methods to achieve more accurate gesture recognition and predictions.Our study presents a novel virtual keyboard allowing character input via distinct hand gestures,focusing on two key aspects:hand gesture recognition and character input mechanisms.We developed a novel model with LSTM and fully connected layers for enhanced sequential data processing and hand gesture recognition.We also integrated CNN,max-pooling,and dropout layers for improved spatial feature extraction.This model architecture processes both temporal and spatial aspects of hand gestures,using LSTM to extract complex patterns from frame sequences for a comprehensive understanding of input data.Our unique dataset,essential for training the model,includes 1,662 landmarks from dynamic hand gestures,33 postures,and 468 face landmarks,all captured in real-time using advanced pose estimation.The model demonstrated high accuracy,achieving 98.52%in hand gesture recognition and over 97%in character input across different scenarios.Its excellent performance in real-time testing underlines its practicality and effectiveness,marking a significant advancement in enhancing human-device interactions in the digital age.

Design and Implementation of Hand Gesture Detection System Using HM Model for Sign Language Recognition Development

Gesture detection is the primary and most significant step for sign language detection and sign language is the communication medium for people with speaking and hearing disabilities. This paper presents a novel method for dynamic hand gesture detection using Hidden Markov Models (HMMs) where we detect different English alphabet letters by tracing hand movements. The process involves skin color-based segmentation for hand isolation in video frames, followed by morphological operations to enhance image trajectories. Our system employs hand tracking and trajectory smoothing techniques, such as the Kalman filter, to monitor hand movements and refine gesture paths. Quantized sequences are then analyzed using the Baum-Welch Re-estimation Algorithm, an HMM-based approach. A maximum likelihood classifier is used to identify the most probable letter from the test sequences. Our method demonstrates significant improvements over traditional recognition techniques in real-time, automatic hand gesture recognition, particularly in its ability to distinguish complex gestures. The experimental results confirm the effectiveness of our approach in enhancing gesture-based sign language detection to alleviate the barrier between the deaf and hard-of-hearing community and general people.

A Novel SE-CNN Attention Architecture for sEMG-Based Hand Gesture Recognition

In this article,to reduce the complexity and improve the generalization ability of current gesture recognition systems,we propose a novel SE-CNN attention architecture for sEMG-based hand gesture recognition.The proposed algorithm introduces a temporal squeeze-and-excite block into a simple CNN architecture and then utilizes it to recalibrate the weights of the feature outputs from the convolutional layer.By enhancing important features while suppressing useless ones,the model realizes gesture recognition efficiently.The last procedure of the proposed algorithm is utilizing a simple attention mechanism to enhance the learned representations of sEMG signals to performmulti-channel sEMG-based gesture recognition tasks.To evaluate the effectiveness and accuracy of the proposed algorithm,we conduct experiments involving multi-gesture datasets Ninapro DB4 and Ninapro DB5 for both inter-session validation and subject-wise cross-validation.After a series of comparisons with the previous models,the proposed algorithm effectively increases the robustness with improved gesture recognition performance and generalization ability.

Appearance Based Dynamic Hand Gesture Recognition Using 3D Separable Convolutional Neural Network

Appearance-based dynamic Hand Gesture Recognition(HGR)remains a prominent area of research in Human-Computer Interaction(HCI).Numerous environmental and computational constraints limit its real-time deployment.In addition,the performance of a model decreases as the subject’s distance from the camera increases.This study proposes a 3D separable Convolutional Neural Network(CNN),considering the model’s computa-tional complexity and recognition accuracy.The 20BN-Jester dataset was used to train the model for six gesture classes.After achieving the best offline recognition accuracy of 94.39%,the model was deployed in real-time while considering the subject’s attention,the instant of performing a gesture,and the subject’s distance from the camera.Despite being discussed in numerous research articles,the distance factor remains unresolved in real-time deployment,which leads to degraded recognition results.In the proposed approach,the distance calculation substantially improves the classification performance by reducing the impact of the subject’s distance from the camera.Additionally,the capability of feature extraction,degree of relevance,and statistical significance of the proposed model against other state-of-the-art models were validated using t-distributed Stochastic Neighbor Embedding(t-SNE),Mathew’s Correlation Coefficient(MCC),and the McNemar test,respectively.We observed that the proposed model exhibits state-of-the-art outcomes and a comparatively high significance level.

Hand Gesture Recognition for Disabled People Using Bayesian Optimization with Transfer Learning

Sign language recognition can be treated as one of the efficient solu-tions for disabled people to communicate with others.It helps them to convey the required data by the use of sign language with no issues.The latest develop-ments in computer vision and image processing techniques can be accurately uti-lized for the sign recognition process by disabled people.American Sign Language(ASL)detection was challenging because of the enhancing intraclass similarity and higher complexity.This article develops a new Bayesian Optimiza-tion with Deep Learning-Driven Hand Gesture Recognition Based Sign Language Communication(BODL-HGRSLC)for Disabled People.The BODL-HGRSLC technique aims to recognize the hand gestures for disabled people’s communica-tion.The presented BODL-HGRSLC technique integrates the concepts of compu-ter vision(CV)and DL models.In the presented BODL-HGRSLC technique,a deep convolutional neural network-based residual network(ResNet)model is applied for feature extraction.Besides,the presented BODL-HGRSLC model uses Bayesian optimization for the hyperparameter tuning process.At last,a bidir-ectional gated recurrent unit(BiGRU)model is exploited for the HGR procedure.A wide range of experiments was conducted to demonstrate the enhanced perfor-mance of the presented BODL-HGRSLC model.The comprehensive comparison study reported the improvements of the BODL-HGRSLC model over other DL models with maximum accuracy of 99.75%.

An Efficient and Robust Hand Gesture Recognition System of Sign Language Employing Finetuned Inception-V3 and Efficientnet-B0 Network

Hand Gesture Recognition(HGR)is a promising research area with an extensive range of applications,such as surgery,video game techniques,and sign language translation,where sign language is a complicated structured form of hand gestures.The fundamental building blocks of structured expressions in sign language are the arrangement of the fingers,the orientation of the hand,and the hand’s position concerning the body.The importance of HGR has increased due to the increasing number of touchless applications and the rapid growth of the hearing-impaired population.Therefore,real-time HGR is one of the most effective interaction methods between computers and humans.Developing a user-free interface with good recognition performance should be the goal of real-time HGR systems.Nowadays,Convolutional Neural Network(CNN)shows great recognition rates for different image-level classification tasks.It is challenging to train deep CNN networks like VGG-16,VGG-19,Inception-v3,and Efficientnet-B0 from scratch because only some significant labeled image datasets are available for static hand gesture images.However,an efficient and robust hand gesture recognition system of sign language employing finetuned Inception-v3 and Efficientnet-Bo network is proposed to identify hand gestures using a comparative small HGR dataset.Experiments show that Inception-v3 achieved 90%accuracy and 0.93%precision,0.91%recall,and 0.90%f1-score,respectively,while EfficientNet-B0 achieved 99%accuracy and 0.98%,0.97%,0.98%,precision,recall,and f1-score respectively.

Gesture Recognition Based on Time-of-Flight Sensor and Residual Neural Network

With the advancement of technology and the increase in user demands, gesture recognition played a pivotal role in the field of human-computer interaction. Among various sensing devices, Time-of-Flight (ToF) sensors were widely applied due to their low cost. This paper explored the implementation of a human hand posture recognition system using ToF sensors and residual neural networks. Firstly, this paper reviewed the typical applications of human hand recognition. Secondly, this paper designed a hand gesture recognition system using a ToF sensor VL53L5. Subsequently, data preprocessing was conducted, followed by training the constructed residual neural network. Then, the recognition results were analyzed, indicating that gesture recognition based on the residual neural network achieved an accuracy of 98.5% in a 5-class classification scenario. Finally, the paper discussed existing issues and future research directions.

Dynamic Hand Gesture Recognition Based on Short-Term Sampling Neural Networks

Hand gestures are a natural way for human-robot interaction.Vision based dynamic hand gesture recognition has become a hot research topic due to its various applications.This paper presents a novel deep learning network for hand gesture recognition.The network integrates several well-proved modules together to learn both short-term and long-term features from video inputs and meanwhile avoid intensive computation.To learn short-term features,each video input is segmented into a fixed number of frame groups.A frame is randomly selected from each group and represented as an RGB image as well as an optical flow snapshot.These two entities are fused and fed into a convolutional neural network(Conv Net)for feature extraction.The Conv Nets for all groups share parameters.To learn longterm features,outputs from all Conv Nets are fed into a long short-term memory(LSTM)network,by which a final classification result is predicted.The new model has been tested with two popular hand gesture datasets,namely the Jester dataset and Nvidia dataset.Comparing with other models,our model produced very competitive results.The robustness of the new model has also been proved with an augmented dataset with enhanced diversity of hand gestures.

Vision Based Hand Gesture Recognition Using 3D Shape Context

Hand gesture recognition is a popular topic in computer vision and makes human-computer interaction more flexible and convenient.The representation of hand gestures is critical for recognition.In this paper,we propose a new method to measure the similarity between hand gestures and exploit it for hand gesture recognition.The depth maps of hand gestures captured via the Kinect sensors are used in our method,where the 3D hand shapes can be segmented from the cluttered backgrounds.To extract the pattern of salient 3D shape features,we propose a new descriptor-3D Shape Context,for 3D hand gesture representation.The 3D Shape Context information of each 3D point is obtained in multiple scales because both local shape context and global shape distribution are necessary for recognition.The description of all the 3D points constructs the hand gesture representation,and hand gesture recognition is explored via dynamic time warping algorithm.Extensive experiments are conducted on multiple benchmark datasets.The experimental results verify that the proposed method is robust to noise,articulated variations,and rigid transformations.Our method outperforms state-of-the-art methods in the comparisons of accuracy and efficiency.

Dynamic Hand Gesture Recognition Using 3D-CNN and LSTM Networks

Recognition of dynamic hand gestures in real-time is a difficult task because the system can never know when or from where the gesture starts and ends in a video stream.Many researchers have been working on visionbased gesture recognition due to its various applications.This paper proposes a deep learning architecture based on the combination of a 3D Convolutional Neural Network(3D-CNN)and a Long Short-Term Memory(LSTM)network.The proposed architecture extracts spatial-temporal information from video sequences input while avoiding extensive computation.The 3D-CNN is used for the extraction of spectral and spatial features which are then given to the LSTM network through which classification is carried out.The proposed model is a light-weight architecture with only 3.7 million training parameters.The model has been evaluated on 15 classes from the 20BN-jester dataset available publicly.The model was trained on 2000 video-clips per class which were separated into 80%training and 20%validation sets.An accuracy of 99%and 97%was achieved on training and testing data,respectively.We further show that the combination of 3D-CNN with LSTM gives superior results as compared to MobileNetv2+LSTM.

Robust Interactive Method for Hand Gestures Recognition Using Machine Learning

The Hand Gestures Recognition(HGR)System can be employed to facilitate communication between humans and computers instead of using special input and output devices.These devices may complicate communication with computers especially for people with disabilities.Hand gestures can be defined as a natural human-to-human communication method,which also can be used in human-computer interaction.Many researchers developed various techniques and methods that aimed to understand and recognize specific hand gestures by employing one or two machine learning algorithms with a reasonable accuracy.Thiswork aims to develop a powerful hand gesture recognition model with a 100%recognition rate.We proposed an ensemble classification model that combines the most powerful machine learning classifiers to obtain diversity and improve accuracy.The majority voting method was used to aggregate accuracies produced by each classifier and get the final classification result.Our model was trained using a self-constructed dataset containing 1600 images of ten different hand gestures.The employing of canny’s edge detector and histogram of oriented gradient method was a great combination with the ensemble classifier and the recognition rate.The experimental results had shown the robustness of our proposed model.Logistic Regression and Support Vector Machine have achieved 100%accuracy.The developed model was validated using two public datasets,and the findings have proved that our model outperformed other compared studies.

A Hand Gesture Recognition Method Based on SVM

A hand gesture recognition method is presented for human-computer interaction, which is based on fingertip localization. First, hand gesture is segmented from the background based on skin color characteristics. Second, feature vectors are selected with equal intervals on the boundary of the gesture, and then gestures＇ length normalization is accomplished. Third, the fingertip positions are determined by the feature vectors＇ parameters, and angles of feature vectors are normalized. Finally the gestures are classified by support vector machine. The experimental results demonstrate that the proposed method can recognize 9 gestures with an accuracy of 94.1%.

Home Automation-Based Health Assessment Along Gesture Recognition via Inertial Sensors

Hand gesture recognition (HGR) is used in a numerous applications,including medical health-care, industrial purpose and sports detection.We have developed a real-time hand gesture recognition system using inertialsensors for the smart home application. Developing such a model facilitatesthe medical health field (elders or disabled ones). Home automation has alsobeen proven to be a tremendous benefit for the elderly and disabled. Residentsare admitted to smart homes for comfort, luxury, improved quality of life,and protection against intrusion and burglars. This paper proposes a novelsystem that uses principal component analysis, linear discrimination analysisfeature extraction, and random forest as a classifier to improveHGRaccuracy.We have achieved an accuracy of 94% over the publicly benchmarked HGRdataset. The proposed system can be used to detect hand gestures in thehealthcare industry as well as in the industrial and educational sectors.

Extraction of intersecting palm-vein and palmprint features for cancellable identity verification

A novel method based on the cross-modality intersecting features of the palm-vein and the palmprint is proposed for identity verification.Capitalising on the unique geometrical relationship between the two biometric modalities,the cross-modality intersecting points provides a stable set of features for identity verification.To facilitate flexibility in template changes,a template transformation is proposed.While maintaining non-invertibility,the template transformation allows transformation sizes beyond that offered by the con-ventional means.Extensive experiments using three public palm databases are conducted to verify the effectiveness the proposed system for identity recognition.

Intelligent Sign Language Recognition System for E-Learning Context

In this research work,an efficient sign language recognition tool for e-learning has been proposed with a new type of feature set based on angle and lines.This feature set has the ability to increase the overall performance of machine learning algorithms in an efficient way.The hand gesture recognition based on these features has been implemented for usage in real-time.The feature set used hand landmarks,which were generated using media-pipe(MediaPipe)and open computer vision(openCV)on each frame of the incoming video.The overall algorithm has been tested on two well-known ASLalphabet(American Sign Language)and ISL-HS(Irish Sign Language)sign language datasets.Different machine learning classifiers including random forest,decision tree,and naïve Bayesian have been used to classify hand gestures using this unique feature set and their respective results have been compared.Since the random forest classifier performed better,it has been selected as the base classifier for the proposed system.It showed 96.7%accuracy with ISL-HS and 93.7%accuracy with ASL-alphabet dataset using the extracted features.

Performance Analysis of Machine Learning Algorithms for Classifying Hand Motion-Based EEG Brain Signals

Brain-computer interfaces (BCIs) records brain activity using electroencephalogram (EEG) headsets in the form of EEG signals;these signals can berecorded, processed and classified into different hand movements, which can beused to control other IoT devices. Classification of hand movements will beone step closer to applying these algorithms in real-life situations using EEGheadsets. This paper uses different feature extraction techniques and sophisticatedmachine learning algorithms to classify hand movements from EEG brain signalsto control prosthetic hands for amputated persons. To achieve good classificationaccuracy, denoising and feature extraction of EEG signals is a significant step. Wesaw a considerable increase in all the machine learning models when the movingaverage filter was applied to the raw EEG data. Feature extraction techniques likea fast fourier transform (FFT) and continuous wave transform (CWT) were usedin this study;three types of features were extracted, i.e., FFT Features, CWTCoefficients and CWT scalogram images. We trained and compared differentmachine learning (ML) models like logistic regression, random forest, k-nearestneighbors (KNN), light gradient boosting machine (GBM) and XG boost onFFT and CWT features and deep learning (DL) models like VGG-16, DenseNet201 and ResNet50 trained on CWT scalogram images. XG Boost with FFTfeatures gave the maximum accuracy of 88%.

End-to-End Multiview Gesture Recognition for Autonomous Car Parking System

The use of hand gestures can be the most intuitive human-machine interaction medium.The early approaches for hand gesture recognition used device-based methods.These methods use mechanical or optical sensors attached to a glove or markers,which hinder the natural human-machine communication.On the other hand,vision-based methods are less restrictive and allow for a more spontaneous communication without the need of an intermediary between human and machine.Therefore,vision gesture recognition has been a popular area of research for the past thirty years.Hand gesture recognition finds its application in many areas,particularly the automotive industry where advanced automotive human-machine interface(HMI)designers are using gesture recognition to improve driver and vehicle safety.However,technology advances go beyond active/passive safety and into convenience and comfort.In this context,one of America’s big three automakers has partnered with the Centre of Pattern Analysis and Machine Intelligence(CPAMI)at the University of Waterloo to investigate expanding their product segment through machine learning to provide an increased driver convenience and comfort with the particular application of hand gesture recognition for autonomous car parking.The present paper leverages the state-of-the-art deep learning and optimization techniques to develop a vision-based multiview dynamic hand gesture recognizer for a self-parking system.We propose a 3D-CNN gesture model architecture that we train on a publicly available hand gesture database.We apply transfer learning methods to fine-tune the pre-trained gesture model on custom-made data,which significantly improves the proposed system performance in a real world environment.We adapt the architecture of end-to-end solution to expand the state-of-the-art video classifier from a single image as input(fed by monocular camera)to a Multiview 360 feed,offered by a six cameras module.Finally,we optimize the proposed solution to work on a limited resource embedded platform(Nvidia Jetson TX2)that is used by automakers for vehicle-based features,without sacrificing the accuracy robustness and real time functionality of the system.

Integrated photonic convolution acceleration core for wearable devices

With the advancement of deep learning and neural networks,the computational demands for applications in wearable devices have grown exponentially.However,wearable devices also have strict requirements for long battery life,low power consumption,and compact size.In this work,we propose a scalable optoelectronic computing system based on an integrated optical convolution acceleration core.This system enables high-precision computation at the speed of light,achieving 7-bit accuracy while maintaining extremely low power consumption.It also demonstrates peak throughput of 3.2 TOPS(tera operations per second)in parallel processing.We have successfully demonstrated image convolution and the typical application of an interactive first-person perspective gesture recognition application based on depth information.The system achieves a comparable recognition accuracy to traditional electronic computation in all blind tests.

Multi-modal fusion for robust hand gesture recognition based on heterogeneous networks

Hand gesture recognition has become a vital subject in the fields of human-computer interaction and rehabilitation assessment.This paper presents a multi-modal fusion for hand gesture recognition(MFHG)model,which uses two heterogeneous networks to extract and fuse the features of the vision-based motion signals and the surface electromyography(s EMG)signals,respectively.To extract the features of the vision-based motion signals,a graph neural network,named the cumulation graph attention(CGAT)model,is first proposed to characterize the prior knowledge of motion coupling between finger joints.The CGAT model uses the cumulation mechanism to combine the early and late extracted features to improve motion-based hand gesture recognition.For the s EMG signals,a time-frequency convolutional neural network model,named TF-CNN,is proposed to extract both the signals'time-domain and frequency-domain information.To improve the performance of hand gesture recognition,the deep features from multiple modes are merged with an average layer,and then the regularization items containing center loss and the mutual information loss are employed to enhance the robustness of this multi-modal system.Finally,a data set containing the multi-modal signals from seven subjects on different days is built to verify the performance of the multi-modal model.The experimental results indicate that the MFHG can reach 99.96%and 92.46%accuracy on hand gesture recognition in the cases of within-session and cross-day,respectively.