Chemical simulation teaching system based on virtual reality and gesture interaction

Background Most existing chemical experiment teaching systems lack solid immersive experiences,making it difficult to engage students.To address these challenges,we propose a chemical simulation teaching system based on virtual reality and gesture interaction.Methods The parameters of the models were obtained through actual investigation,whereby Blender and 3DS MAX were used to model and import these parameters into a physics engine.By establishing an interface for the physics engine,gesture interaction hardware,and virtual reality(VR)helmet,a highly realistic chemical experiment environment was created.Using code script logic,particle systems,as well as other systems,chemical phenomena were simulated.Furthermore,we created an online teaching platform using streaming media and databases to address the problems of distance teaching.Results The proposed system was evaluated against two mainstream products in the market.In the experiments,the proposed system outperformed the other products in terms of fidelity and practicality.Conclusions The proposed system which offers realistic simulations and practicability,can help improve the high school chemistry experimental education.

Virtual Keyboard:A Real-Time Hand Gesture Recognition-Based Character Input System Using LSTM and Mediapipe Holistic

In the digital age,non-touch communication technologies are reshaping human-device interactions and raising security concerns.A major challenge in current technology is the misinterpretation of gestures by sensors and cameras,often caused by environmental factors.This issue has spurred the need for advanced data processing methods to achieve more accurate gesture recognition and predictions.Our study presents a novel virtual keyboard allowing character input via distinct hand gestures,focusing on two key aspects:hand gesture recognition and character input mechanisms.We developed a novel model with LSTM and fully connected layers for enhanced sequential data processing and hand gesture recognition.We also integrated CNN,max-pooling,and dropout layers for improved spatial feature extraction.This model architecture processes both temporal and spatial aspects of hand gestures,using LSTM to extract complex patterns from frame sequences for a comprehensive understanding of input data.Our unique dataset,essential for training the model,includes 1,662 landmarks from dynamic hand gestures,33 postures,and 468 face landmarks,all captured in real-time using advanced pose estimation.The model demonstrated high accuracy,achieving 98.52%in hand gesture recognition and over 97%in character input across different scenarios.Its excellent performance in real-time testing underlines its practicality and effectiveness,marking a significant advancement in enhancing human-device interactions in the digital age.

Magnetic Array Assisted Triboelectric Nanogenerator Sensor for Real‑Time Gesture Interaction

In human-machine interaction,robotic hands are useful in many scenarios.To operate robotic hands via gestures instead of handles will greatly improve the convenience and intuition of human-machine interaction.Here,we present a magnetic array assisted sliding triboelectric sensor for achieving a real-time gesture interaction between a human hand and robotic hand.With a finger’s traction movement of flexion or extension,the sensor can induce positive/negative pulse signals.Through counting the pulses in unit time,the degree,speed,and direction of finger motion can be judged in realtime.The magnetic array plays an important role in generating the quantifiable pulses.The designed two parts of magnetic array can transform sliding motion into contact-separation and constrain the sliding pathway,respectively,thus improve the durability,low speed signal amplitude,and stability of the system.This direct quantization approach and optimization of wearable gesture sensor provide a new strategy for achieving a natural,intuitive,and real-time human-robotic interaction.

Gesture interaction in virtual reality

With the development of virtual reality(VR)and human-computer interaction technology,how to use natural and efficient interaction methods in the virtual environment has become a hot topic of research.Gesture is one of the most important communication methods of human beings,which can effectively express users'demands.In the past few decades,gesture-based interaction has made significant progress.This article focuses on the gesture interaction technology and discusses the definition and classification of gestures,input devices for gesture interaction,and gesture interaction recognition technology.The application of gesture interaction technology in virtual reality is studied,the existing problems in the current gesture interaction are summarized,and the future development is prospected.

EyeScreen:A Vision-Based Gesture Interaction System

EyeScreen is a vision-based interaction system which provides a natural gesture interface for humancomputer interaction （HCI） by tracking human fingers and recognizing gestures. Multi-view video images are captured by two cameras facing a computer screen, which can be used to detect clicking actions of a fingertip and improve the recognition rate. The system enables users to directly interact with rendered objects on the screen. Robustness of the system has been verified by extensive experiments with different user scenarios. EyeScreen can be used in many applications such as intelligent interaction and digital entertainment.

Dynamic constraint and objective generation approach for real-time train rescheduling model under human-computer interaction

Real-time train rescheduling plays a vital role in railway transportation as it is crucial for maintaining punctuality and reliability in rail operations.In this paper,we propose a rescheduling model that incorporates constraints and objectives generated through human-computer interaction.This approach ensures that the model is aligned with practical requirements and daily operational tasks while facilitating iterative train rescheduling.The dispatcher’s empirical knowledge is integrated into the train rescheduling process using a human-computer interaction framework.We introduce six interfaces to dynamically construct constraints and objectives that capture human intentions.By summarizing rescheduling rules,we devise a rule-based conflict detection-resolution heuristic algorithm to effectively solve the formulated model.A series of numerical experiments are presented,demonstrating strong performance across the entire system.Furthermore,theflexibility of rescheduling is enhanced through secondary analysis-driven solutions derived from the outcomes of humancomputer interactions in the previous step.This proposed interaction method complements existing literature on rescheduling methods involving human-computer interactions.It serves as a tool to aid dispatchers in identifying more feasible solutions in accordance with their empirical rescheduling strategies.

Optimization and Performance Enhancement of Gesture Recognition Algorithm Based on FMCW Millimeter-Wave Radar

Gesture recognition plays an increasingly important role as the requirements of intelligent systems for human-computer interaction methods increase.To improve the accuracy of the millimeter-wave radar gesture detection algorithm with limited computational resources,this study improves the detection performance in terms of optimized features and interference filtering.The accuracy of the algorithm is improved by refining the combination of gesture features using a self-constructed dataset,and biometric filtering is introduced to reduce the interference of inanimate object motion.Finally,experiments demonstrate the effectiveness of the proposed algorithm in both mitigating interference from inanimate objects and accurately recognizing gestures.Results show a notable 93.29%average reduction in false detections achieved through the integration of biometric filtering into the algorithm’s interpretation of target movements.Additionally,the algorithm adeptly identifies the six gestures with an average accuracy of 96.84%on embedded systems.

Gesture Recognition Based on Time-of-Flight Sensor and Residual Neural Network

With the advancement of technology and the increase in user demands, gesture recognition played a pivotal role in the field of human-computer interaction. Among various sensing devices, Time-of-Flight (ToF) sensors were widely applied due to their low cost. This paper explored the implementation of a human hand posture recognition system using ToF sensors and residual neural networks. Firstly, this paper reviewed the typical applications of human hand recognition. Secondly, this paper designed a hand gesture recognition system using a ToF sensor VL53L5. Subsequently, data preprocessing was conducted, followed by training the constructed residual neural network. Then, the recognition results were analyzed, indicating that gesture recognition based on the residual neural network achieved an accuracy of 98.5% in a 5-class classification scenario. Finally, the paper discussed existing issues and future research directions.

A Survey of Gesture Recognition Using Frequency Modulated Continuous Wave Radar

With technology advances and human requirements increasing, human-computer interaction plays an important role in our daily lives. Among these interactions, gesture-based recognition offers a natural and intuitive user experience that does not require physical contact and is becoming increasingly prevalent across various fields. Gesture recognition systems based on Frequency Modulated Continuous Wave (FMCW) millimeter-wave radar are receiving widespread attention due to their ability to operate without wearable sensors, their robustness to environmental factors, and the excellent penetrative ability of radar signals. This paper first reviews the current main gesture recognition applications. Subsequently, we introduce the system of gesture recognition based on FMCW radar and provide a general framework for gesture recognition, including gesture data acquisition, data preprocessing, and classification methods. We then discuss typical applications of gesture recognition systems and summarize the performance of these systems in terms of experimental environment, signal acquisition, signal processing, and classification methods. Specifically, we focus our study on four typical gesture recognition systems, including air-writing recognition, gesture command recognition, sign language recognition, and text input recognition. Finally, this paper addresses the challenges and unresolved problems in FMCW radar-based gesture recognition and provides insights into potential future research directions.

Research of Natural Gesture Recognition and Interactive Technology Compatible with YCb Crand HSV Color Space

In view of the current gesture recognition algorithm based on skin color segmentation is not flexible and has weak resistance to the environment, this paper puts forward a new method of skin color modeling to improve the adaptability of gesture segmentation when it face to different states. The modeling built by double color space instead of only one is compatible both in YCbCr and HSV color space to training the Gaussian model which can update the threshold value for binarization. Finally, this paper designed a natural gesture recognition and interactive systems based on the double color space model. It has shown that the system has a good interactive experience in different environments.

Improving Students’Participation and Collaboration With Adjusting Cloud Education Platform During the Real-Time Interactive Class

The purpose of this study was to find a way to promote the collaboration and interaction of students and bring about the growth of learners through feedback while taking advantage of real-time interactive class via video conferencing tools.Although real-time interactive class with using video conferencing tools had great advantages,but there were also limitations of active interaction.To this end,real-time interactive tool and cloud-based educational platform were applied to create cases of learner participation classes and analyze the cases.The convergence of real-time interactive class tools and cloud tools has been able to draw students’participation and collaboration in non-face-to-face situations,and it can be seen that it is very helpful in creating learner-centered educational activities based on communication and interaction with students.Through this,the application of the cloud-based educational platform in real-time interactive class could lead students to participate and collaborate even in non-face-to-face situations.

Vision Based Hand Gesture Recognition Using 3D Shape Context

Hand gesture recognition is a popular topic in computer vision and makes human-computer interaction more flexible and convenient.The representation of hand gestures is critical for recognition.In this paper,we propose a new method to measure the similarity between hand gestures and exploit it for hand gesture recognition.The depth maps of hand gestures captured via the Kinect sensors are used in our method,where the 3D hand shapes can be segmented from the cluttered backgrounds.To extract the pattern of salient 3D shape features,we propose a new descriptor-3D Shape Context,for 3D hand gesture representation.The 3D Shape Context information of each 3D point is obtained in multiple scales because both local shape context and global shape distribution are necessary for recognition.The description of all the 3D points constructs the hand gesture representation,and hand gesture recognition is explored via dynamic time warping algorithm.Extensive experiments are conducted on multiple benchmark datasets.The experimental results verify that the proposed method is robust to noise,articulated variations,and rigid transformations.Our method outperforms state-of-the-art methods in the comparisons of accuracy and efficiency.

Dynamic Hand Gesture Recognition Using 3D-CNN and LSTM Networks

Recognition of dynamic hand gestures in real-time is a difficult task because the system can never know when or from where the gesture starts and ends in a video stream.Many researchers have been working on visionbased gesture recognition due to its various applications.This paper proposes a deep learning architecture based on the combination of a 3D Convolutional Neural Network(3D-CNN)and a Long Short-Term Memory(LSTM)network.The proposed architecture extracts spatial-temporal information from video sequences input while avoiding extensive computation.The 3D-CNN is used for the extraction of spectral and spatial features which are then given to the LSTM network through which classification is carried out.The proposed model is a light-weight architecture with only 3.7 million training parameters.The model has been evaluated on 15 classes from the 20BN-jester dataset available publicly.The model was trained on 2000 video-clips per class which were separated into 80%training and 20%validation sets.An accuracy of 99%and 97%was achieved on training and testing data,respectively.We further show that the combination of 3D-CNN with LSTM gives superior results as compared to MobileNetv2+LSTM.

A Visual-Based Gesture Prediction Framework Applied in Social Robots

In daily life,people use their hands in various ways for most daily activities.There are many applications based on the position,direction,and joints of the hand,including gesture recognition,gesture prediction,robotics and so on.This paper proposes a gesture prediction system that uses hand joint coordinate features collected by the Leap Motion to predict dynamic hand gestures.The model is applied to the NAO robot to verify the effectiveness of the proposed method.First of all,in order to reduce jitter or jump generated in the process of data acquisition by the Leap Motion,the Kalman filter is applied to the original data.Then some new feature descriptors are introduced.The length feature,angle feature and angular velocity feature are extracted from the filtered data.These features are fed into the long-short time memory recurrent neural network(LSTM-RNN)with different combinations.Experimental results show that the combination of coordinate,length and angle features achieves the highest accuracy of 99.31%,and it can also run in real time.Finally,the trained model is applied to the NAO robot to play the finger-guessing game.Based on the predicted gesture,the NAO robot can respond in advance.

Real-time forward modeling and inversion of logging-while-drilling electromagnetic measurements in horizontal wells

Based on the pseudo-analytical equation of electromagnetic log for layered formation,an optimal boundary match method is proposed to adaptively truncate the encountered formation structures.An efficient integral method is put forward to significantly accelerate the convergence of Sommerfeld integral.By asymptotically approximating and subtracting the first reflection/transmission waves from the scattered field,the new Sommerfeld integral method has addressed difficulties encountered by the traditional digital filtering method,such as low computational precision and limited operating range,and realized the acceleration of the computation speed of logging-while-drilling electromagnetic measurements(LWD EM).By making use of the priori information from the offset/pilot wells and interactively adjusting the formation model,the optimum initial guesses of the inversion model is determined in order to predict the nearby formation boundaries.The gradient optimization algorithm is developed and an interactive inversion system for the LWD EM data from the horizontal wells is established.The inverted results of field data demonstrated that the real-time interactive inversion method is capable of providing the accurate boundaries of layers around the wellbore from the LWD EM,and it will benefit the wellbore trajectory optimization and reservoir interpretation.

Active Appearance Model Based Hand Gesture Recognition

This paper addresses the application of hand gesture recognition in monocular image sequences using Active Appearance Model （AAM）, For this work, the proposed algorithm is composed of constricting AAMs and fitting the models to the interest region. In training stage, according to the manual labeled feature points, the relative AAM is constructed and the corresponding average feature is obtained. In recognition stage, the interesting hand gesture region is firstly segmented by skin and movement cues. Secondly, the models are fitted to the image that includes the hand gesture, and the relative features are extracted. Thirdly, the classification is done by comparing the extracted features and average features. 30 different gestures of Chinese sign language are applied for testing the effectiveness of the method. The Experimental results are given indicating good performance of the algorithm.

A Hand Gesture Recognition Method Based on SVM

A hand gesture recognition method is presented for human-computer interaction, which is based on fingertip localization. First, hand gesture is segmented from the background based on skin color characteristics. Second, feature vectors are selected with equal intervals on the boundary of the gesture, and then gestures＇ length normalization is accomplished. Third, the fingertip positions are determined by the feature vectors＇ parameters, and angles of feature vectors are normalized. Finally the gestures are classified by support vector machine. The experimental results demonstrate that the proposed method can recognize 9 gestures with an accuracy of 94.1%.

Multimodal interaction design and application in augmented reality for chemical experiment

Background Augmented reality classrooms have become an interesting research topic in the field of education,but there are some limitations.Firstly,most researchers use cards to operate experiments,and a large number of cards cause difficulty and inconvenience for users.Secondly,most users conduct experiments only in the visual modal,and such single-modal interaction greatly reduces the users'real sense of interaction.In order to solve these problems,we propose the Multimodal Interaction Algorithm based on Augmented Reality(ARGEV),which is based on visual and tactile feedback in Augmented Reality.In addition,we design a Virtual and Real Fusion Interactive Tool Suite(VRFITS)with gesture recognition and intelligent equipment.Methods The ARGVE method fuses gesture,intelligent equipment,and virtual models.We use a gesture recognition model trained by a convolutional neural network to recognize the gestures in AR,and to trigger a vibration feedback after a recognizing a five finger grasp gesture.We establish a coordinate mapping relationship between real hands and the virtual model to achieve the fusion of gestures and the virtual model.Results The average accuracy rate of gesture recognition was 99.04%.We verify and apply VRFITS in the Augmented Reality Chemistry Lab(ARCL),and the overall operation load of ARCL is thus reduced by 29.42%,in comparison to traditional simulation virtual experiments.Conclusions We achieve real-time fusion of the gesture,virtual model,and intelligent equipment in ARCL.Compared with the NOBOOK virtual simulation experiment,ARCL improves the users'real sense of operation and interaction efficiency.

On attaining user-friendly hand gesture interfaces to control existing GUIs

Background Hand ge stur e interfaces are dedicated programs that principally perform hand tracking and hand gesture prediction to provide alternative controls and interaction methods.They take advantage of one of the most natural ways of interaction and communication,proposing novel input and showing great potential in the field of the human-computer interaction.Developing a flexible and rich hand gesture interface is known to be a time-consuming and arduous task.Previously published studies have demonstrated the significance of the finite-state-machine(FSM)approach when mapping detected gestures to GUI actions.Methods In our hand gesture interface,we broadened the FSM approach by utilizing gesture-specific attributes,such as distance between hands,distance from the camera,and time of occurrences,to enable users to perform unique GUI actions.These attributes are obtained from hand gestures detected by the RealSense SDK employed in our hand gesture interface.By means of these gesture-specific attributes,users can activate static gestures and perform them as dynamic gestures.We also provided supplementary features to enhance the efficiency,convenience,and user-friendliness of our hand gesture interface.Moreover,we developed a complementary application for recording hand gestures by capturing hand keypoints in depth and color images to facilitate the generation of hand gesture datasets.Results We conducted a small-scale user survey with fifteen subjects to test and evaluate our hand gesture interface.Anonymous feedback obtained from the users indicates that our hand gesture interface is adequately facile and self-explanatory to use.In addition,we received constructive feedback about minor flaws regarding the responsiveness of the interface.Conclusions We proposed a hand gesture interface along with key concepts to attain use r-friendlines s and effectiveness in the control of existing GUIs.

Review of dynamic gesture recognition

In recent years,gesture recognition has been widely used in the fields of intelligent driving,virtual reality,and human-computer interaction.With the development of artificial intelligence,deep learning has achieved remarkable success in computer vision.To help researchers better understanding the development status of gesture recognition in video,this article provides a detailed survey of the latest developments in gesture recognition technology for videos based on deep learning.The reviewed methods are broadly categorized into three groups based on the type of neural networks used for recognition:two stream convolutional neural networks,3D convolutional neural networks,and Long-short Term Memory(LSTM)networks.In this review,we discuss the advantages and limitations of existing technologies,focusing on the feature extraction method of the spatiotemporal structure information in a video sequence,and consider future research directions.