A Comprehensive Survey on Deep Learning Multi-Modal Fusion:Methods,Technologies and Applications

Multi-modal fusion technology gradually become a fundamental task in many fields,such as autonomous driving,smart healthcare,sentiment analysis,and human-computer interaction.It is rapidly becoming the dominant research due to its powerful perception and judgment capabilities.Under complex scenes,multi-modal fusion technology utilizes the complementary characteristics of multiple data streams to fuse different data types and achieve more accurate predictions.However,achieving outstanding performance is challenging because of equipment performance limitations,missing information,and data noise.This paper comprehensively reviews existing methods based onmulti-modal fusion techniques and completes a detailed and in-depth analysis.According to the data fusion stage,multi-modal fusion has four primary methods:early fusion,deep fusion,late fusion,and hybrid fusion.The paper surveys the three majormulti-modal fusion technologies that can significantly enhance the effect of data fusion and further explore the applications of multi-modal fusion technology in various fields.Finally,it discusses the challenges and explores potential research opportunities.Multi-modal tasks still need intensive study because of data heterogeneity and quality.Preserving complementary information and eliminating redundant information between modalities is critical in multi-modal technology.Invalid data fusion methods may introduce extra noise and lead to worse results.This paper provides a comprehensive and detailed summary in response to these challenges.

Sparse Adversarial Learning for FDIA Attack Sample Generation in Distributed Smart

False data injection attack(FDIA)is an attack that affects the stability of grid cyber-physical system(GCPS)by evading the detecting mechanism of bad data.Existing FDIA detection methods usually employ complex neural networkmodels to detect FDIA attacks.However,they overlook the fact that FDIA attack samples at public-private network edges are extremely sparse,making it difficult for neural network models to obtain sufficient samples to construct a robust detection model.To address this problem,this paper designs an efficient sample generative adversarial model of FDIA attack in public-private network edge,which can effectively bypass the detectionmodel to threaten the power grid system.A generative adversarial network(GAN)framework is first constructed by combining residual networks(ResNet)with fully connected networks(FCN).Then,a sparse adversarial learning model is built by integrating the time-aligned data and normal data,which is used to learn the distribution characteristics between normal data and attack data through iterative confrontation.Furthermore,we introduce a Gaussian hybrid distributionmatrix by aggregating the network structure of attack data characteristics and normal data characteristics,which can connect and calculate FDIA data with normal characteristics.Finally,efficient FDIA attack samples can be sequentially generated through interactive adversarial learning.Extensive simulation experiments are conducted with IEEE 14-bus and IEEE 118-bus system data,and the results demonstrate that the generated attack samples of the proposed model can present superior performance compared to state-of-the-art models in terms of attack strength,robustness,and covert capability.

Generative adversarial networks based motion learning towards robotic calligraphy synthesis

Robot calligraphy visually reflects the motion capability of robotic manipulators.While traditional researches mainly focus on image generation and the writing of simple calligraphic strokes or characters,this article presents a generative adversarial network(GAN)-based motion learning method for robotic calligraphy synthesis(Gan2CS)that can enhance the efficiency in writing complex calligraphy words and reproducing classic calligraphy works.The key technologies in the proposed approach include:(1)adopting the GAN to learn the motion parameters from the robot writing operation;(2)converting the learnt motion data into the style font and realising the transition from static calligraphy images to dynamic writing demonstration;(3)reproducing high-precision calligraphy works by synthesising the writing motion data hierarchically.In this study,the motion trajectories of sample calligraphy images are firstly extracted and converted into the robot module.The robot performs the writing with motion planning,and the writing motion parameters of calligraphy strokes are learnt with GANs.Then the motion data of basic strokes is synthesised based on the hierarchical process of‘stroke-radicalpart-character’.And the robot re-writes the synthesised characters whose similarity with the original calligraphy characters is evaluated.Regular calligraphy characters have been tested in the experiments for method validation and the results validated that the robot can actualise the robotic calligraphy synthesis of writing motion data with GAN.

PowerDetector:Malicious PowerShell Script Family Classification Based on Multi-Modal Semantic Fusion and Deep Learning

Power Shell has been widely deployed in fileless malware and advanced persistent threat(APT)attacks due to its high stealthiness and live-off-theland technique.However,existing works mainly focus on deobfuscation and malicious detection,lacking the malicious Power Shell families classification and behavior analysis.Moreover,the state-of-the-art methods fail to capture fine-grained features and semantic relationships,resulting in low robustness and accuracy.To this end,we propose Power Detector,a novel malicious Power Shell script detector based on multimodal semantic fusion and deep learning.Specifically,we design four feature extraction methods to extract key features from character,token,abstract syntax tree(AST),and semantic knowledge graph.Then,we intelligently design four embeddings(i.e.,Char2Vec,Token2Vec,AST2Vec,and Rela2Vec) and construct a multi-modal fusion algorithm to concatenate feature vectors from different views.Finally,we propose a combined model based on transformer and CNN-Bi LSTM to implement Power Shell family detection.Our experiments with five types of Power Shell attacks show that PowerDetector can accurately detect various obfuscated and stealth PowerShell scripts,with a 0.9402 precision,a 0.9358 recall,and a 0.9374 F1-score.Furthermore,through singlemodal and multi-modal comparison experiments,we demonstrate that PowerDetector’s multi-modal embedding and deep learning model can achieve better accuracy and even identify more unknown attacks.

A survey of multi-modal learning theory

Deep multi-modal learning,a rapidly growing field with a wide range of practical applications,aims to effectively utilize and integrate information from multiple sources,known as modalities.Despite its impressive empirical performance,the theoretical foundations of deep multi-modal learning have yet to be fully explored.In this paper,we will undertake a comprehensive survey of recent developments in multi-modal learning theories,focusing on the fundamental properties that govern this field.Our goal is to provide a thorough collection of current theoretical tools for analyzing multi-modal learning,to clarify their implications for practitioners,and to suggest future directions for the establishment of a solid theoretical foundation for deep multi-modal learning.

Transfer Learning-Based Semi-Supervised Generative Adversarial Network for Malaria Classification

Malaria is a lethal disease responsible for thousands of deaths worldwide every year.Manual methods of malaria diagnosis are timeconsuming that require a great deal of human expertise and efforts.Computerbased automated diagnosis of diseases is progressively becoming popular.Although deep learning models show high performance in the medical field,it demands a large volume of data for training which is hard to acquire for medical problems.Similarly,labeling of medical images can be done with the help of medical experts only.Several recent studies have utilized deep learning models to develop efficient malaria diagnostic system,which showed promising results.However,the most common problem with these models is that they need a large amount of data for training.This paper presents a computer-aided malaria diagnosis system that combines a semi-supervised generative adversarial network and transfer learning.The proposed model is trained in a semi-supervised manner and requires less training data than conventional deep learning models.Performance of the proposed model is evaluated on a publicly available dataset of blood smear images(with malariainfected and normal class)and achieved a classification accuracy of 96.6%.

Alpha Fusion Adversarial Attack Analysis Using Deep Learning

The deep learning model encompasses a powerful learning ability that integrates the feature extraction,and classification method to improve accuracy.Convolutional Neural Networks(CNN)perform well in machine learning and image processing tasks like segmentation,classification,detection,identification,etc.The CNN models are still sensitive to noise and attack.The smallest change in training images as in an adversarial attack can greatly decrease the accuracy of the CNN model.This paper presents an alpha fusion attack analysis and generates defense against adversarial attacks.The proposed work is divided into three phases:firstly,an MLSTM-based CNN classification model is developed for classifying COVID-CT images.Secondly,an alpha fusion attack is generated to fool the classification model.The alpha fusion attack is tested in the last phase on a modified LSTM-based CNN(CNN-MLSTM)model and other pre-trained models.The results of CNN models show that the accuracy of these models dropped greatly after the alpha-fusion attack.The highest F1 score before the attack was achieved is 97.45 And after the attack lowest F1 score recorded is 22%.Results elucidate the performance in terms of accuracy,precision,F1 score and Recall.

Correcting Climate Model Sea Surface Temperature Simulations with Generative Adversarial Networks:Climatology,Interannual Variability,and Extremes

Climate models are vital for understanding and projecting global climate change and its associated impacts.However,these models suffer from biases that limit their accuracy in historical simulations and the trustworthiness of future projections.Addressing these challenges requires addressing internal variability,hindering the direct alignment between model simulations and observations,and thwarting conventional supervised learning methods.Here,we employ an unsupervised Cycle-consistent Generative Adversarial Network(CycleGAN),to correct daily Sea Surface Temperature(SST)simulations from the Community Earth System Model 2(CESM2).Our results reveal that the CycleGAN not only corrects climatological biases but also improves the simulation of major dynamic modes including the El Niño-Southern Oscillation(ENSO)and the Indian Ocean Dipole mode,as well as SST extremes.Notably,it substantially corrects climatological SST biases,decreasing the globally averaged Root-Mean-Square Error(RMSE)by 58%.Intriguingly,the CycleGAN effectively addresses the well-known excessive westward bias in ENSO SST anomalies,a common issue in climate models that traditional methods,like quantile mapping,struggle to rectify.Additionally,it substantially improves the simulation of SST extremes,raising the pattern correlation coefficient(PCC)from 0.56 to 0.88 and lowering the RMSE from 0.5 to 0.32.This enhancement is attributed to better representations of interannual,intraseasonal,and synoptic scales variabilities.Our study offers a novel approach to correct global SST simulations and underscores its effectiveness across different time scales and primary dynamical modes.

Generative Multi-Modal Mutual Enhancement Video Semantic Communications

Recently,there have been significant advancements in the study of semantic communication in single-modal scenarios.However,the ability to process information in multi-modal environments remains limited.Inspired by the research and applications of natural language processing across different modalities,our goal is to accurately extract frame-level semantic information from videos and ultimately transmit high-quality videos.Specifically,we propose a deep learning-basedMulti-ModalMutual Enhancement Video Semantic Communication system,called M3E-VSC.Built upon a VectorQuantized Generative AdversarialNetwork(VQGAN),our systemaims to leverage mutual enhancement among different modalities by using text as the main carrier of transmission.With it,the semantic information can be extracted fromkey-frame images and audio of the video and performdifferential value to ensure that the extracted text conveys accurate semantic information with fewer bits,thus improving the capacity of the system.Furthermore,a multi-frame semantic detection module is designed to facilitate semantic transitions during video generation.Simulation results demonstrate that our proposed model maintains high robustness in complex noise environments,particularly in low signal-to-noise ratio conditions,significantly improving the accuracy and speed of semantic transmission in video communication by approximately 50 percent.

A Hand Features Based Fusion Recognition Network with Enhancing Multi-Modal Correlation

Fusing hand-based features in multi-modal biometric recognition enhances anti-spoofing capabilities.Additionally,it leverages inter-modal correlation to enhance recognition performance.Concurrently,the robustness and recognition performance of the system can be enhanced through judiciously leveraging the correlation among multimodal features.Nevertheless,two issues persist in multi-modal feature fusion recognition:Firstly,the enhancement of recognition performance in fusion recognition has not comprehensively considered the inter-modality correlations among distinct modalities.Secondly,during modal fusion,improper weight selection diminishes the salience of crucial modal features,thereby diminishing the overall recognition performance.To address these two issues,we introduce an enhanced DenseNet multimodal recognition network founded on feature-level fusion.The information from the three modalities is fused akin to RGB,and the input network augments the correlation between modes through channel correlation.Within the enhanced DenseNet network,the Efficient Channel Attention Network(ECA-Net)dynamically adjusts the weight of each channel to amplify the salience of crucial information in each modal feature.Depthwise separable convolution markedly reduces the training parameters and further enhances the feature correlation.Experimental evaluations were conducted on four multimodal databases,comprising six unimodal databases,including multispectral palmprint and palm vein databases from the Chinese Academy of Sciences.The Equal Error Rates(EER)values were 0.0149%,0.0150%,0.0099%,and 0.0050%,correspondingly.In comparison to other network methods for palmprint,palm vein,and finger vein fusion recognition,this approach substantially enhances recognition performance,rendering it suitable for high-security environments with practical applicability.The experiments in this article utilized amodest sample database comprising 200 individuals.The subsequent phase involves preparing for the extension of the method to larger databases.

Toward Trustworthy Decision-Making for Autonomous Vehicles:A Robust Reinforcement Learning Approach with Safety Guarantees

While autonomous vehicles are vital components of intelligent transportation systems,ensuring the trustworthiness of decision-making remains a substantial challenge in realizing autonomous driving.Therefore,we present a novel robust reinforcement learning approach with safety guarantees to attain trustworthy decision-making for autonomous vehicles.The proposed technique ensures decision trustworthiness in terms of policy robustness and collision safety.Specifically,an adversary model is learned online to simulate the worst-case uncertainty by approximating the optimal adversarial perturbations on the observed states and environmental dynamics.In addition,an adversarial robust actor-critic algorithm is developed to enable the agent to learn robust policies against perturbations in observations and dynamics.Moreover,we devise a safety mask to guarantee the collision safety of the autonomous driving agent during both the training and testing processes using an interpretable knowledge model known as the Responsibility-Sensitive Safety Model.Finally,the proposed approach is evaluated through both simulations and experiments.These results indicate that the autonomous driving agent can make trustworthy decisions and drastically reduce the number of collisions through robust safety policies.

An Empirical Study on the Effectiveness of Adversarial Examples in Malware Detection

Antivirus vendors and the research community employ Machine Learning(ML)or Deep Learning(DL)-based static analysis techniques for efficient identification of new threats,given the continual emergence of novel malware variants.On the other hand,numerous researchers have reported that Adversarial Examples(AEs),generated by manipulating previously detected malware,can successfully evade ML/DL-based classifiers.Commercial antivirus systems,in particular,have been identified as vulnerable to such AEs.This paper firstly focuses on conducting black-box attacks to circumvent ML/DL-based malware classifiers.Our attack method utilizes seven different perturbations,including Overlay Append,Section Append,and Break Checksum,capitalizing on the ambiguities present in the PE format,as previously employed in evasion attack research.By directly applying the perturbation techniques to PE binaries,our attack method eliminates the need to grapple with the problem-feature space dilemma,a persistent challenge in many evasion attack studies.Being a black-box attack,our method can generate AEs that successfully evade both DL-based and ML-based classifiers.Also,AEs generated by the attack method retain their executability and malicious behavior,eliminating the need for functionality verification.Through thorogh evaluations,we confirmed that the attack method achieves an evasion rate of 65.6%against well-known ML-based malware detectors and can reach a remarkable 99%evasion rate against well-known DL-based malware detectors.Furthermore,our AEs demonstrated the capability to bypass detection by 17%of vendors out of the 64 on VirusTotal(VT).In addition,we propose a defensive approach that utilizes Trend Locality Sensitive Hashing(TLSH)to construct a similarity-based defense model.Through several experiments on the approach,we verified that our defense model can effectively counter AEs generated by the perturbation techniques.In conclusion,our defense model alleviates the limitation of the most promising defense method,adversarial training,which is only effective against the AEs that are included in the training classifiers.

Image segmentation of exfoliated two-dimensional materials by generative adversarial network-based data augmentation

Mechanically cleaved two-dimensional materials are random in size and thickness.Recognizing atomically thin flakes by human experts is inefficient and unsuitable for scalable production.Deep learning algorithms have been adopted as an alternative,nevertheless a major challenge is a lack of sufficient actual training images.Here we report the generation of synthetic two-dimensional materials images using StyleGAN3 to complement the dataset.DeepLabv3Plus network is trained with the synthetic images which reduces overfitting and improves recognition accuracy to over 90%.A semi-supervisory technique for labeling images is introduced to reduce manual efforts.The sharper edges recognized by this method facilitate material stacking with precise edge alignment,which benefits exploring novel properties of layered-material devices that crucially depend on the interlayer twist-angle.This feasible and efficient method allows for the rapid and high-quality manufacturing of atomically thin materials and devices.

Multi-modal knowledge graph inference via media convergence and logic rule

Media convergence works by processing information from different modalities and applying them to different domains.It is difficult for the conventional knowledge graph to utilise multi-media features because the introduction of a large amount of information from other modalities reduces the effectiveness of representation learning and makes knowledge graph inference less effective.To address the issue,an inference method based on Media Convergence and Rule-guided Joint Inference model(MCRJI)has been pro-posed.The authors not only converge multi-media features of entities but also introduce logic rules to improve the accuracy and interpretability of link prediction.First,a multi-headed self-attention approach is used to obtain the attention of different media features of entities during semantic synthesis.Second,logic rules of different lengths are mined from knowledge graph to learn new entity representations.Finally,knowledge graph inference is performed based on representing entities that converge multi-media features.Numerous experimental results show that MCRJI outperforms other advanced baselines in using multi-media features and knowledge graph inference,demonstrating that MCRJI provides an excellent approach for knowledge graph inference with converged multi-media features.

Unified deep learning model for predicting fundus fluorescein angiography image from fundus structure image

The prediction of fundus fluorescein angiography(FFA)images from fundus structural images is a cutting-edge research topic in ophthalmological image processing.Prediction comprises estimating FFA from fundus camera imaging,single-phase FFA from scanning laser ophthalmoscopy(SLO),and three-phase FFA also from SLO.Although many deep learning models are available,a single model can only perform one or two of these prediction tasks.To accomplish three prediction tasks using a unified method,we propose a unified deep learning model for predicting FFA images from fundus structure images using a supervised generative adversarial network.The three prediction tasks are processed as follows:data preparation,network training under FFA supervision,and FFA image prediction from fundus structure images on a test set.By comparing the FFA images predicted by our model,pix2pix,and CycleGAN,we demonstrate the remarkable progress achieved by our proposal.The high performance of our model is validated in terms of the peak signal-to-noise ratio,structural similarity index,and mean squared error.

Toward Improved Accuracy in Quasi-Static Elastography Using Deep Learning

Elastography is a non-invasive medical imaging technique to map the spatial variation of elastic properties of soft tissues.The quality of reconstruction results in elastography is highly sensitive to the noise induced by imaging measurements and processing.To address this issue,we propose a deep learning(DL)model based on conditional Generative Adversarial Networks(cGANs)to improve the quality of nonhomogeneous shear modulus reconstruction.To train this model,we generated a synthetic displacement field with finite element simulation under known nonhomogeneous shear modulus distribution.Both the simulated and experimental displacement fields are used to validate the proposed method.The reconstructed results demonstrate that the DL model with synthetic training data is able to improve the quality of the reconstruction compared with the well-established optimization method.Moreover,we emphasize that our DL model is only trained on synthetic data.This might provide a way to alleviate the challenge of obtaining clinical or experimental data in elastography.Overall,this work addresses several fatal issues in applying the DL technique into elastography,and the proposed method has shown great potential in improving the accuracy of the disease diagnosis in clinical medicine.

Slope displacement prediction based on multisource domain transfer learning for insufficient sample data

Accurate displacement prediction is critical for the early warning of landslides.The complexity of the coupling relationship between multiple influencing factors and displacement makes the accurate prediction of displacement difficult.Moreover,in engineering practice,insufficient monitoring data limit the performance of prediction models.To alleviate this problem,a displacement prediction method based on multisource domain transfer learning,which helps accurately predict data in the target domain through the knowledge of one or more source domains,is proposed.First,an optimized variational mode decomposition model based on the minimum sample entropy is used to decompose the cumulative displacement into the trend,periodic,and stochastic components.The trend component is predicted by an autoregressive model,and the periodic component is predicted by the long short-term memory.For the stochastic component,because it is affected by uncertainties,it is predicted by a combination of a Wasserstein generative adversarial network and multisource domain transfer learning for improved prediction accuracy.Considering a real mine slope as a case study,the proposed prediction method was validated.Therefore,this study provides new insights that can be applied to scenarios lacking sample data.

Unsupervised multi-modal image translation based on the squeeze-and-excitation mechanism and feature attention module

The unsupervised multi-modal image translation is an emerging domain of computer vision whose goal is to transform an image from the source domain into many diverse styles in the target domain.However,the multi-generator mechanism is employed among the advanced approaches available to model different domain mappings,which results in inefficient training of neural networks and pattern collapse,leading to inefficient generation of image diversity.To address this issue,this paper introduces a multi-modal unsupervised image translation framework that uses a generator to perform multi-modal image translation.Specifically,firstly,the domain code is introduced in this paper to explicitly control the different generation tasks.Secondly,this paper brings in the squeeze-and-excitation(SE)mechanism and feature attention(FA)module.Finally,the model integrates multiple optimization objectives to ensure efficient multi-modal translation.This paper performs qualitative and quantitative experiments on multiple non-paired benchmark image translation datasets while demonstrating the benefits of the proposed method over existing technologies.Overall,experimental results have shown that the proposed method is versatile and scalable.

Fetal MRI Artifacts: Semi-Supervised Generative Adversarial Neural Network for Motion Artifacts Reducing in Fetal Magnetic Resonance Images

This study addresses challenges in fetal magnetic resonance imaging (MRI) related to motion artifacts, maternal respiration, and hardware limitations. To enhance MRI quality, we employ deep learning techniques, specifically utilizing Cycle GAN. Synthetic pairs of images, simulating artifacts in fetal MRI, are generated to train the model. Our primary contribution is the use of Cycle GAN for fetal MRI restoration, augmented by artificially corrupted data. We compare three approaches (supervised Cycle GAN, Pix2Pix, and Mobile Unet) for artifact removal. Experimental results demonstrate that the proposed supervised Cycle GAN effectively removes artifacts while preserving image details, as validated through Structural Similarity Index Measure (SSIM) and normalized Mean Absolute Error (MAE). The method proves comparable to alternatives but avoids the generation of spurious regions, which is crucial for medical accuracy.

Enhancing Pneumonia Detection in Pediatric Chest X-Rays Using CGAN-Augmented Datasets and Lightweight Deep Transfer Learning Models

Pneumonia ranks as a leading cause of mortality, particularly in children aged five and under. Detecting this disease typically requires radiologists to examine chest X-rays and report their findings to physicians, a task susceptible to human error. The application of Deep Transfer Learning (DTL) for the identification of pneumonia through chest X-rays is hindered by a shortage of available images, which has led to less than optimal DTL performance and issues with overfitting. Overfitting is characterized by a model’s learning that is too closely fitted to the training data, reducing its effectiveness on unseen data. The problem of overfitting is especially prevalent in medical image processing due to the high costs and extensive time required for image annotation, as well as the challenge of collecting substantial datasets that also respect patient privacy concerning infectious diseases such as pneumonia. To mitigate these challenges, this paper introduces the use of conditional generative adversarial networks (CGAN) to enrich the pneumonia dataset with 2690 synthesized X-ray images of the minority class, aiming to even out the dataset distribution for improved diagnostic performance. Subsequently, we applied four modified lightweight deep transfer learning models such as Xception, MobileNetV2, MobileNet, and EfficientNetB0. These models have been fine-tuned and evaluated, demonstrating remarkable detection accuracies of 99.26%, 98.23%, 97.06%, and 94.55%, respectively, across fifty epochs. The experimental results validate that the models we have proposed achieve high detection accuracy rates, with the best model reaching up to 99.26% effectiveness, outperforming other models in the diagnosis of pneumonia from X-ray images.