A Hand Features Based Fusion Recognition Network with Enhancing Multi-Modal Correlation

Fusing hand-based features in multi-modal biometric recognition enhances anti-spoofing capabilities.Additionally,it leverages inter-modal correlation to enhance recognition performance.Concurrently,the robustness and recognition performance of the system can be enhanced through judiciously leveraging the correlation among multimodal features.Nevertheless,two issues persist in multi-modal feature fusion recognition:Firstly,the enhancement of recognition performance in fusion recognition has not comprehensively considered the inter-modality correlations among distinct modalities.Secondly,during modal fusion,improper weight selection diminishes the salience of crucial modal features,thereby diminishing the overall recognition performance.To address these two issues,we introduce an enhanced DenseNet multimodal recognition network founded on feature-level fusion.The information from the three modalities is fused akin to RGB,and the input network augments the correlation between modes through channel correlation.Within the enhanced DenseNet network,the Efficient Channel Attention Network(ECA-Net)dynamically adjusts the weight of each channel to amplify the salience of crucial information in each modal feature.Depthwise separable convolution markedly reduces the training parameters and further enhances the feature correlation.Experimental evaluations were conducted on four multimodal databases,comprising six unimodal databases,including multispectral palmprint and palm vein databases from the Chinese Academy of Sciences.The Equal Error Rates(EER)values were 0.0149%,0.0150%,0.0099%,and 0.0050%,correspondingly.In comparison to other network methods for palmprint,palm vein,and finger vein fusion recognition,this approach substantially enhances recognition performance,rendering it suitable for high-security environments with practical applicability.The experiments in this article utilized amodest sample database comprising 200 individuals.The subsequent phase involves preparing for the extension of the method to larger databases.

Research on Driver’s Fatigue Detection Based on Information Fusion

Driving fatigue is a physiological phenomenon that often occurs during driving.After the driver enters a fatigued state,the attentionis lax,the response is slow,and the ability todeal with emergencies is significantly reduced,which can easily cause traffic accidents.Therefore,studying driver fatigue detectionmethods is significant in ensuring safe driving.However,the fatigue state of actual drivers is easily interfered with by the external environment(glasses and light),which leads to many problems,such as weak reliability of fatigue driving detection.Moreover,fatigue is a slow process,first manifested in physiological signals and then reflected in human face images.To improve the accuracy and stability of fatigue detection,this paper proposed a driver fatigue detection method based on image information and physiological information,designed a fatigue driving detection device,built a simulation driving experiment platform,and collected facial as well as physiological information of drivers during driving.Finally,the effectiveness of the fatigue detection method was evaluated.Eye movement feature parameters and physiological signal features of drivers’fatigue levels were extracted.The driver fatigue detection model was trained to classify fatigue and non-fatigue states based on the extracted features.Accuracy rates of the image,electroencephalogram(EEG),and blood oxygen signals were 86%,82%,and 71%,separately.Information fusion theory was presented to facilitate the fatigue detection effect;the fatigue features were fused using multiple kernel learning and typical correlation analysis methods to increase the detection accuracy to 94%.It can be seen that the fatigue driving detectionmethod based onmulti-source feature fusion effectively detected driver fatigue state,and the accuracy rate was higher than that of a single information source.In summary,fatigue drivingmonitoring has broad development prospects and can be used in traffic accident prevention and wearable driver fatigue recognition.

PowerDetector:Malicious PowerShell Script Family Classification Based on Multi-Modal Semantic Fusion and Deep Learning

Power Shell has been widely deployed in fileless malware and advanced persistent threat(APT)attacks due to its high stealthiness and live-off-theland technique.However,existing works mainly focus on deobfuscation and malicious detection,lacking the malicious Power Shell families classification and behavior analysis.Moreover,the state-of-the-art methods fail to capture fine-grained features and semantic relationships,resulting in low robustness and accuracy.To this end,we propose Power Detector,a novel malicious Power Shell script detector based on multimodal semantic fusion and deep learning.Specifically,we design four feature extraction methods to extract key features from character,token,abstract syntax tree(AST),and semantic knowledge graph.Then,we intelligently design four embeddings(i.e.,Char2Vec,Token2Vec,AST2Vec,and Rela2Vec) and construct a multi-modal fusion algorithm to concatenate feature vectors from different views.Finally,we propose a combined model based on transformer and CNN-Bi LSTM to implement Power Shell family detection.Our experiments with five types of Power Shell attacks show that PowerDetector can accurately detect various obfuscated and stealth PowerShell scripts,with a 0.9402 precision,a 0.9358 recall,and a 0.9374 F1-score.Furthermore,through singlemodal and multi-modal comparison experiments,we demonstrate that PowerDetector’s multi-modal embedding and deep learning model can achieve better accuracy and even identify more unknown attacks.

Multi-Modal Military Event Extraction Based on Knowledge Fusion

Event extraction stands as a significant endeavor within the realm of information extraction,aspiring to automatically extract structured event information from vast volumes of unstructured text.Extracting event elements from multi-modal data remains a challenging task due to the presence of a large number of images and overlapping event elements in the data.Although researchers have proposed various methods to accomplish this task,most existing event extraction models cannot address these challenges because they are only applicable to text scenarios.To solve the above issues,this paper proposes a multi-modal event extraction method based on knowledge fusion.Specifically,for event-type recognition,we use a meticulous pipeline approach that integrates multiple pre-trained models.This approach enables a more comprehensive capture of the multidimensional event semantic features present in military texts,thereby enhancing the interconnectedness of information between trigger words and events.For event element extraction,we propose a method for constructing a priori templates that combine event types with corresponding trigger words.This approach facilitates the acquisition of fine-grained input samples containing event trigger words,thus enabling the model to understand the semantic relationships between elements in greater depth.Furthermore,a fusion method for spatial mapping of textual event elements and image elements is proposed to reduce the category number overload and effectively achieve multi-modal knowledge fusion.The experimental results based on the CCKS 2022 dataset show that our method has achieved competitive results,with a comprehensive evaluation value F1-score of 53.4%for the model.These results validate the effectiveness of our method in extracting event elements from multi-modal data.

An Analysis Model of Learners’ Online Learning Status Based on Deep Neural Network and Multi-Dimensional Information Fusion

The learning status of learners directly affects the quality of learning.Compared with offline teachers,it is difficult for online teachers to capture the learning status of students in the whole class,and it is even more difficult to continue to pay attention to studentswhile teaching.Therefore,this paper proposes an online learning state analysis model based on a convolutional neural network and multi-dimensional information fusion.Specifically,a facial expression recognition model and an eye state recognition model are constructed to detect students’emotions and fatigue,respectively.By integrating the detected data with the homework test score data after online learning,an analysis model of students’online learning status is constructed.According to the PAD model,the learning state is expressed as three dimensions of students’understanding,engagement and interest,and then analyzed from multiple perspectives.Finally,the proposed model is applied to actual teaching,and procedural analysis of 5 different types of online classroom learners is carried out,and the validity of the model is verified by comparing with the results of the manual analysis.

Robust Symmetry Prediction with Multi-Modal Feature Fusion for Partial Shapes

In geometry processing,symmetry research benefits from global geo-metric features of complete shapes,but the shape of an object captured in real-world applications is often incomplete due to the limited sensor resolution,single viewpoint,and occlusion.Different from the existing works predicting symmetry from the complete shape,we propose a learning approach for symmetry predic-tion based on a single RGB-D image.Instead of directly predicting the symmetry from incomplete shapes,our method consists of two modules,i.e.,the multi-mod-al feature fusion module and the detection-by-reconstruction module.Firstly,we build a channel-transformer network(CTN)to extract cross-fusion features from the RGB-D as the multi-modal feature fusion module,which helps us aggregate features from the color and the depth separately.Then,our self-reconstruction net-work based on a 3D variational auto-encoder(3D-VAE)takes the global geo-metric features as input,followed by a prediction symmetry network to detect the symmetry.Our experiments are conducted on three public datasets:ShapeNet,YCB,and ScanNet,we demonstrate that our method can produce reliable and accurate results.

A novel adaptive temporal-spatial information fusion model based on Dempster-Shafer evidence theory

In the field of target recognition based on the temporal-spatial information fusion,evidence the-ory has received extensive attention.To achieve accurate and efficient target recognition by the evi-dence theory,an adaptive temporal-spatial information fusion model is proposed.Firstly,an adaptive evaluation correction mechanism is constructed by the evidence distance and Deng entropy,which realizes the credibility discrimination and adaptive correction of the spatial evidence.Secondly,the credibility decay operator is introduced to obtain the dynamic credibility of temporal evidence.Finally,the sequential combination of temporal-spatial evidences is achieved by Shafer’s discount criterion and Dempster’s combination rule.The simulation results show that the proposed method not only considers the dynamic and sequential characteristics of the temporal-spatial evidences com-bination,but also has a strong conflict information processing capability,which provides a new refer-ence for the field of temporal-spatial information fusion.

A Heterogeneous Information Fusion Method for Maritime Radar and AIS Based on D-S Evidence Theory

Maritime radar and automatic identification systems (AIS), which are essential auxiliary equipment for navigation safety in the shipping industry, have played significant roles in maritime safety supervision. However, in practical applications, the information obtained by a single device is limited, and it is necessary to integrate the information of maritime radar and AIS messages to achieve better recognition effects. In this study, the D-S evidence theory is used to fusion the two kinds of heterogeneous information: maritime radar images and AIS messages. Firstly, the radar image and AIS message are processed to get the targets of interest in the same coordinate system. Then, the coordinate position and heading of targets are chosen as the indicators for judging target similarity. Finally, a piece of D-S evidence theory based on the information fusion method is proposed to match the radar target and the AIS target of the same ship. Particularly, the effectiveness of the proposed method has been validated and evaluated through several experiments, which proves that such a method is practical in maritime safety supervision.

Information perception and feedback mechanism and key techniques of multi-modality human-robot interaction for service robots

With the increasing of the elderly population and the growing hearth care cost, the role of service robots in aiding the disabled and the elderly is becoming important. Many researchers in the world have paid much attention to heaRthcare robots and rehabilitation robots. To get natural and harmonious communication between the user and a service robot, the information perception/feedback ability, and interaction ability for service robots become more important in many key issues.

Monitoring Computer Numerical Control Machining Progress Based on Information Fusion

To cope with the market demand dynamically,enterprise needs to obtain the production status of work in process real-timely,but the information of machining progress has feature of uncertainty and can not reflect the status of production field effectively.In this work,to overcome the ineffectiveness of computer numerical control(CNC) machining progress information extraction and its application restriction in practice because of heterogeneous system of CNC machine,based on information fusion by analyzing multi-sources information,estimating CNC machining status and predicting the machining progress through tracking tool coordinates,a CNC machining progress monitoring method is presented.The multi-sources heterogeneous information includes machining path,real-time spindle power information,manual input data and tool position.On the method of obtaining this multi-sources heterogeneous information,the method which helps explore numerical control(NC) program,monitor spindle power of CNC,collect human-computer interaction(HCI) information,obtain real-time tool coordinates and express the knowledge concerned in this field is analyzed; The decision rule of CNC machining status in the way of fusing multi-sources information in manufacturing process is summarized,as well as the machining progress tracking method in accordance with real-time tool coordinates and machining path is presented.Finally,the method discussed is proved feasible by the verification of machining progress tracking through simulation experiment.The proposed research realizes the effective integration of CNC machining progress information,and enables enterprises an efficient way to share CNC information and configure CNC resources optimally.

Hierarchical hybrid testability modeling and evaluation method based on information fusion

In order to meet the demand of testability analysis and evaluation for complex equipment under a small sample test in the equipment life cycle, the hierarchical hybrid testability model- ing and evaluation method （HHTME）, which combines the testabi- lity structure model （TSM） with the testability Bayesian networks model （TBNM）, is presented. Firstly, the testability network topo- logy of complex equipment is built by using the hierarchical hybrid testability modeling method. Secondly, the prior conditional prob- ability distribution between network nodes is determined through expert experience. Then the Bayesian method is used to update the conditional probability distribution, according to history test information, virtual simulation information and similar product in- formation. Finally, the learned hierarchical hybrid testability model （HHTM） is used to estimate the testability of equipment. Compared with the results of other modeling methods, the relative deviation of the HHTM is only 0.52%, and the evaluation result is the most accu rate.

Uncertain information fusion with robust adaptive neural networks-fuzzy reasoning

In practical multi-sensor information fusion systems, there exists uncertainty about the network structure, active state of sensors, and information itself （including fuzziness, randomness, incompleteness as well as roughness, etc）. Hence it requires investigating the problem of uncertain information fusion. Robust learning algorithm which adapts to complex environment and the fuzzy inference algorithm which disposes fuzzy information are explored to solve the problem. Based on the fusion technology of neural networks and fuzzy inference algorithm, a multi-sensor uncertain information fusion system is modeled. Also RANFIS learning algorithm and fusing weight synthesized inference algorithm are developed from the ANFIS algorithm according to the concept of robust neural networks. This fusion system mainly consists of RANFIS confidence estimator, fusing weight synthesized inference knowledge base and weighted fusion section. The simulation result demonstrates that the proposed fusion model and algorithm have the capability of uncertain information fusion, thus is obviously advantageous compared with the conventional Kalman weighted fusion algorithm.

An Intelligent Fault Diagnosis Method of Multi-Scale Deep Feature Fusion Based on Information Entropy

For a single-structure deep learning fault diagnosis model,its disadvantages are an insufficient feature extraction and weak fault classification capability.This paper proposes a multi-scale deep feature fusion intelligent fault diagnosis method based on information entropy.First,a normal autoencoder,denoising autoencoder,sparse autoencoder,and contractive autoencoder are used in parallel to construct a multi-scale deep neural network feature extraction structure.A deep feature fusion strategy based on information entropy is proposed to obtain low-dimensional features and ensure the robustness of the model and the quality of deep features.Finally,the advantage of the deep belief network probability model is used as the fault classifier to identify the faults.The effectiveness of the proposed method was verified by a gearbox test-bed.Experimental results show that,compared with traditional and existing intelligent fault diagnosis methods,the proposed method can obtain representative information and features from the raw data with higher classification accuracy.

CONDITION MONITOR OF DEEP-HOLE DRILLING BASED ON MULTI-SENSOR INFORMATION FUSION

A condition monitoring method of deep-hole drilling based on multi-sensor information fusion is discussed. The signal of vibration and cutting force are collected when the condition of deep-hole drilling on stainless steel 0Cr17Ni4Cu4Nb is normal or abnormal. Four eigenvectors are extracted on time-domain and frequency-domain analysis of the signals. Then the four eigenvectors are combined and sent to neural networks to dispose. The fusion results indicate that multi-sensor information fusion is superior to single-sensor information, and that cutting force signal can reflect the condition of cutting tool better than vibration signal.

Sensor Fusion with Square-Root Cubature Information Filtering

This paper derives a square-root information-type filtering algorithm for nonlinear multi-sensor fusion problems using the cubature Kalman filter theory. The resulting filter is called the square-root cubature Information filter (SCIF). The SCIF propagates the square-root information matrices derived from numerically stable matrix operations and is therefore numerically robust. The SCIF is applied to a highly maneuvering target tracking problem in a distributed sensor network with feedback. The SCIF’s performance is finally compared with the regular cubature information filter and the traditional extended information filter. The results, presented herein, indicate that the SCIF is the most reliable of all three filters and yields a more accurate estimate than the extended information filter.

Neural Machine Translation by Fusing Key Information of Text

When the Transformer proposed by Google in 2017,it was first used for machine translation tasks and achieved the state of the art at that time.Although the current neural machine translation model can generate high quality translation results,there are still mistranslations and omissions in the translation of key information of long sentences.On the other hand,the most important part in traditional translation tasks is the translation of key information.In the translation results,as long as the key information is translated accurately and completely,even if other parts of the results are translated incorrect,the final translation results’quality can still be guaranteed.In order to solve the problem of mistranslation and missed translation effectively,and improve the accuracy and completeness of long sentence translation in machine translation,this paper proposes a key information fused neural machine translation model based on Transformer.The model proposed in this paper extracts the keywords of the source language text separately as the input of the encoder.After the same encoding as the source language text,it is fused with the output of the source language text encoded by the encoder,then the key information is processed and input into the decoder.With incorporating keyword information from the source language sentence,the model’s performance in the task of translating long sentences is very reliable.In order to verify the effectiveness of the method of fusion of key information proposed in this paper,a series of experiments were carried out on the verification set.The experimental results show that the Bilingual Evaluation Understudy(BLEU)score of the model proposed in this paper on theWorkshop on Machine Translation(WMT)2017 test dataset is higher than the BLEU score of Transformer proposed by Google on the WMT2017 test dataset.The experimental results show the advantages of the model proposed in this paper.

Altitude information fusion method and experiment for UAV

Altitude regulation is a fundamental problem in UAV(unmanned aerial vehicles) control to ensure hovering and autonomous navigation performance.However,data from altitude sensors may be unstable by interference.A digital-filter-based improved adaptive Kalman method is proposed to improve accuracy and reliability of the altitude measurement information.A unique sensor data fusion structure is designed to make different sensors switch automatically in different environment.Simulation and experimental results show that an improved Sage-Husa adaptive extended Kalman filter(SHAEKF) is adopted in altitude data fusion which means that altitude error is limited to 1.5m in high altitude and 1.2m near the ground.This method is proved feasible and effective through hovering flight test and three-dimensional track flight experiment.

Evaluation Model for Capability of Enterprise Agent Coalition Based on Information Fusion and Attribute Reduction

For the issue of evaluation of capability of enterprise agent coalition,an evaluation model based on information fusion and entropy weighting method is presented. The attribute reduction method is utilized to reduce indicators of the capability according to the theory of rough set. The new indicator system can be determined. Attribute reduction can also reduce the workload and remove the redundant information,when there are too many indicators or the indicators have strong correlation. The research complexity can be reduced and the efficiency can be improved. Entropy weighting method is used to determine the weights of the remaining indicators,and the importance of indicators is analyzed. The information fusion model based on nearest neighbor method is developed and utilized to evaluate the capability of multiple agent coalitions,compared to cloud evaluation model and D-S evidence method. Simulation results are reasonable and with obvious distinction. Thus they verify the effectiveness and feasibility of the model. The information fusion model can provide more scientific,rational decision support for choosing the best agent coalition,and provide innovative steps for the evaluation process of capability of agent coalitions.

Recognition of robot-soccer in point of information fusion

Summary of typical information fusion systems, synthesis analysis of Robot Soccer’s architecture, recognition of its characteristic and key technique are given. The result is prompted that Robot Soccer can be treated as a platform of the information fusion.

Bayesian Assessment for Reliability of Binomial Components Based on Information Fusion of Similar Products

A modified Bayesian reliability assessment method of binomial components was proposed by fusing prior information of similar products.The traditional Bayesian method usually directly used all the prior data,ignoring the differences between them,which might decrease the credibility level of reliability evaluation and result in data submergence.To solve the problem,a revised approach was derived to calculate groups of prior data's quantitative credibility,used for weighted data fusion.Then inheritance factor was introduced to build a mixed beta distribution to illustrate the innovation of new products.However,in many cases,inheritance factor was determined by Chi-square test that could not give out exact result with respect to rare failures.To make the model more precise,Barnard's exact test was suggested being used to calculate the inheritance factor.A numerical example is given to demonstrate that the modified method is successful and rational,while the classical method is too conservative and the traditional Bayesian method is too risky.