A Comprehensive Survey on Deep Learning Multi-Modal Fusion:Methods,Technologies and Applications

Multi-modal fusion technology gradually become a fundamental task in many fields,such as autonomous driving,smart healthcare,sentiment analysis,and human-computer interaction.It is rapidly becoming the dominant research due to its powerful perception and judgment capabilities.Under complex scenes,multi-modal fusion technology utilizes the complementary characteristics of multiple data streams to fuse different data types and achieve more accurate predictions.However,achieving outstanding performance is challenging because of equipment performance limitations,missing information,and data noise.This paper comprehensively reviews existing methods based onmulti-modal fusion techniques and completes a detailed and in-depth analysis.According to the data fusion stage,multi-modal fusion has four primary methods:early fusion,deep fusion,late fusion,and hybrid fusion.The paper surveys the three majormulti-modal fusion technologies that can significantly enhance the effect of data fusion and further explore the applications of multi-modal fusion technology in various fields.Finally,it discusses the challenges and explores potential research opportunities.Multi-modal tasks still need intensive study because of data heterogeneity and quality.Preserving complementary information and eliminating redundant information between modalities is critical in multi-modal technology.Invalid data fusion methods may introduce extra noise and lead to worse results.This paper provides a comprehensive and detailed summary in response to these challenges.

Human Gait Recognition for Biometrics Application Based on Deep Learning Fusion Assisted Framework

The demand for a non-contact biometric approach for candidate identification has grown over the past ten years.Based on the most important biometric application,human gait analysis is a significant research topic in computer vision.Researchers have paid a lot of attention to gait recognition,specifically the identification of people based on their walking patterns,due to its potential to correctly identify people far away.Gait recognition systems have been used in a variety of applications,including security,medical examinations,identity management,and access control.These systems require a complex combination of technical,operational,and definitional considerations.The employment of gait recognition techniques and technologies has produced a number of beneficial and well-liked applications.Thiswork proposes a novel deep learning-based framework for human gait classification in video sequences.This framework’smain challenge is improving the accuracy of accuracy gait classification under varying conditions,such as carrying a bag and changing clothes.The proposed method’s first step is selecting two pre-trained deep learningmodels and training fromscratch using deep transfer learning.Next,deepmodels have been trained using static hyperparameters;however,the learning rate is calculated using the particle swarmoptimization(PSO)algorithm.Then,the best features are selected from both trained models using the Harris Hawks controlled Sine-Cosine optimization algorithm.This algorithm chooses the best features,combined in a novel correlation-based fusion technique.Finally,the fused best features are categorized using medium,bi-layer,and tri-layered neural networks.On the publicly accessible dataset known as the CASIA-B dataset,the experimental process of the suggested technique was carried out,and an improved accuracy of 94.14% was achieved.The achieved accuracy of the proposed method is improved by the recent state-of-the-art techniques that show the significance of this work.

Olive Leaf Disease Detection via Wavelet Transform and Feature Fusion of Pre-Trained Deep Learning Models

Olive trees are susceptible to a variety of diseases that can cause significant crop damage and economic losses.Early detection of these diseases is essential for effective management.We propose a novel transformed wavelet,feature-fused,pre-trained deep learning model for detecting olive leaf diseases.The proposed model combines wavelet transforms with pre-trained deep-learning models to extract discriminative features from olive leaf images.The model has four main phases:preprocessing using data augmentation,three-level wavelet transformation,learning using pre-trained deep learning models,and a fused deep learning model.In the preprocessing phase,the image dataset is augmented using techniques such as resizing,rescaling,flipping,rotation,zooming,and contrasting.In wavelet transformation,the augmented images are decomposed into three frequency levels.Three pre-trained deep learning models,EfficientNet-B7,DenseNet-201,and ResNet-152-V2,are used in the learning phase.The models were trained using the approximate images of the third-level sub-band of the wavelet transform.In the fused phase,the fused model consists of a merge layer,three dense layers,and two dropout layers.The proposed model was evaluated using a dataset of images of healthy and infected olive leaves.It achieved an accuracy of 99.72%in the diagnosis of olive leaf diseases,which exceeds the accuracy of other methods reported in the literature.This finding suggests that our proposed method is a promising tool for the early detection of olive leaf diseases.

A Power Data Anomaly Detection Model Based on Deep Learning with Adaptive Feature Fusion

With the popularisation of intelligent power,power devices have different shapes,numbers and specifications.This means that the power data has distributional variability,the model learning process cannot achieve sufficient extraction of data features,which seriously affects the accuracy and performance of anomaly detection.Therefore,this paper proposes a deep learning-based anomaly detection model for power data,which integrates a data alignment enhancement technique based on random sampling and an adaptive feature fusion method leveraging dimension reduction.Aiming at the distribution variability of power data,this paper developed a sliding window-based data adjustment method for this model,which solves the problem of high-dimensional feature noise and low-dimensional missing data.To address the problem of insufficient feature fusion,an adaptive feature fusion method based on feature dimension reduction and dictionary learning is proposed to improve the anomaly data detection accuracy of the model.In order to verify the effectiveness of the proposed method,we conducted effectiveness comparisons through elimination experiments.The experimental results show that compared with the traditional anomaly detection methods,the method proposed in this paper not only has an advantage in model accuracy,but also reduces the amount of parameter calculation of the model in the process of feature matching and improves the detection speed.

Prediction model for corrosion rate of low-alloy steels under atmospheric conditions using machine learning algorithms

This work constructed a machine learning(ML)model to predict the atmospheric corrosion rate of low-alloy steels(LAS).The material properties of LAS,environmental factors,and exposure time were used as the input,while the corrosion rate as the output.6 dif-ferent ML algorithms were used to construct the proposed model.Through optimization and filtering,the eXtreme gradient boosting(XG-Boost)model exhibited good corrosion rate prediction accuracy.The features of material properties were then transformed into atomic and physical features using the proposed property transformation approach,and the dominant descriptors that affected the corrosion rate were filtered using the recursive feature elimination(RFE)as well as XGBoost methods.The established ML models exhibited better predic-tion performance and generalization ability via property transformation descriptors.In addition,the SHapley additive exPlanations(SHAP)method was applied to analyze the relationship between the descriptors and corrosion rate.The results showed that the property transformation model could effectively help with analyzing the corrosion behavior,thereby significantly improving the generalization ability of corrosion rate prediction models.

Sleep Posture Classification Using RGB and Thermal Cameras Based on Deep Learning Model

Sleep posture surveillance is crucial for patient comfort,yet current systems face difficulties in providing compre-hensive studies due to the obstruction caused by blankets.Precise posture assessment remains challenging because of the complex nature of the human body and variations in sleep patterns.Consequently,this study introduces an innovative method utilizing RGB and thermal cameras for comprehensive posture classification,thereby enhancing the analysis of body position and comfort.This method begins by capturing a dataset of sleep postures in the form of videos using RGB and thermal cameras,which depict six commonly adopted postures:supine,left log,right log,prone head,prone left,and prone right.The study involves 10 participants under two conditions:with and without blankets.Initially,the database is normalized into a video frame.The subsequent step entails training a fine-tuned,pretrained Visual Geometry Group(VGG16)and ResNet50 model.In the third phase,the extracted features are utilized for classification.The fourth step of the proposed approach employs a serial fusion technique based on the normal distribution to merge the vectors derived from both the RGB and thermal datasets.Finally,the fused vectors are passed to machine learning classifiers for final classification.The dataset,which includes human sleep postures used in this study’s experiments,achieved a 96.7%accuracy rate using the Quadratic Support Vector Machine(QSVM)without the blanket.Moreover,the Linear SVM,when utilized with a blanket,attained an accuracy of 96%.When normal distribution serial fusion was applied to the blanket features,it resulted in a remarkable average accuracy of 99%.

A Dual Discriminator Method for Generalized Zero-Shot Learning

Zero-shot learning enables the recognition of new class samples by migrating models learned from semanticfeatures and existing sample features to things that have never been seen before. The problems of consistencyof different types of features and domain shift problems are two of the critical issues in zero-shot learning. Toaddress both of these issues, this paper proposes a new modeling structure. The traditional approach mappedsemantic features and visual features into the same feature space;based on this, a dual discriminator approachis used in the proposed model. This dual discriminator approach can further enhance the consistency betweensemantic and visual features. At the same time, this approach can also align unseen class semantic features andtraining set samples, providing a portion of information about the unseen classes. In addition, a new feature fusionmethod is proposed in the model. This method is equivalent to adding perturbation to the seen class features,which can reduce the degree to which the classification results in the model are biased towards the seen classes.At the same time, this feature fusion method can provide part of the information of the unseen classes, improvingits classification accuracy in generalized zero-shot learning and reducing domain bias. The proposed method isvalidated and compared with othermethods on four datasets, and fromthe experimental results, it can be seen thatthe method proposed in this paper achieves promising results.

An Artificial Intelligence-Based Framework for Fruits Disease Recognition Using Deep Learning

Fruit infections have an impact on both the yield and the quality of the crop.As a result,an automated recognition system for fruit leaf diseases is important.In artificial intelligence(AI)applications,especially in agriculture,deep learning shows promising disease detection and classification results.The recent AI-based techniques have a few challenges for fruit disease recognition,such as low-resolution images,small datasets for learning models,and irrelevant feature extraction.This work proposed a new fruit leaf leaf leaf disease recognition framework using deep learning features and improved pathfinder optimization.Three fruit types have been employed in this work for the validation process,such as apple,grape,and Citrus.In the first step,a noisy dataset is prepared by employing the original images to learn the designed framework better.The EfficientNet-B0 deep model is fine-tuned on the next step and trained separately on the original and noisy data.After that,features are fused using a serial concatenation approach that is later optimized in the next step using an improved Path Finder Algorithm(PFA).This algorithm aims to select the best features based on the fitness score and ignore redundant information.The selected features are finally classified using machine learning classifiers such as Medium Neural Network,Wide Neural Network,and Support Vector Machine.The experimental process was conducted on each fruit dataset separately and obtained an accuracy of 100%,99.7%,99.7%,and 93.4%for apple,grape,Citrus fruit,and citrus plant leaves,respectively.A detailed analysis is conducted and also compared with the recent techniques,and the proposed framework shows improved accuracy.

HybridHR-Net:Action Recognition in Video Sequences Using Optimal Deep Learning Fusion Assisted Framework

The combination of spatiotemporal videos and essential features can improve the performance of human action recognition(HAR);however,the individual type of features usually degrades the performance due to similar actions and complex backgrounds.The deep convolutional neural network has improved performance in recent years for several computer vision applications due to its spatial information.This article proposes a new framework called for video surveillance human action recognition dubbed HybridHR-Net.On a few selected datasets,deep transfer learning is used to pre-trained the EfficientNet-b0 deep learning model.Bayesian optimization is employed for the tuning of hyperparameters of the fine-tuned deep model.Instead of fully connected layer features,we considered the average pooling layer features and performed two feature selection techniques-an improved artificial bee colony and an entropy-based approach.Using a serial nature technique,the features that were selected are combined into a single vector,and then the results are categorized by machine learning classifiers.Five publically accessible datasets have been utilized for the experimental approach and obtained notable accuracy of 97%,98.7%,100%,99.7%,and 96.8%,respectively.Additionally,a comparison of the proposed framework with contemporarymethods is done to demonstrate the increase in accuracy.

Multi-Model Fusion Framework Using Deep Learning for Visual-Textual Sentiment Classification

Multimodal Sentiment Analysis(SA)is gaining popularity due to its broad application potential.The existing studies have focused on the SA of single modalities,such as texts or photos,posing challenges in effectively handling social media data with multiple modalities.Moreover,most multimodal research has concentrated on merely combining the two modalities rather than exploring their complex correlations,leading to unsatisfactory sentiment classification results.Motivated by this,we propose a new visualtextual sentiment classification model named Multi-Model Fusion(MMF),which uses a mixed fusion framework for SA to effectively capture the essential information and the intrinsic relationship between the visual and textual content.The proposed model comprises three deep neural networks.Two different neural networks are proposed to extract the most emotionally relevant aspects of image and text data.Thus,more discriminative features are gathered for accurate sentiment classification.Then,a multichannel joint fusion modelwith a self-attention technique is proposed to exploit the intrinsic correlation between visual and textual characteristics and obtain emotionally rich information for joint sentiment classification.Finally,the results of the three classifiers are integrated using a decision fusion scheme to improve the robustness and generalizability of the proposed model.An interpretable visual-textual sentiment classification model is further developed using the Local Interpretable Model-agnostic Explanation model(LIME)to ensure the model’s explainability and resilience.The proposed MMF model has been tested on four real-world sentiment datasets,achieving(99.78%)accuracy on Binary_Getty(BG),(99.12%)on Binary_iStock(BIS),(95.70%)on Twitter,and(79.06%)on the Multi-View Sentiment Analysis(MVSA)dataset.These results demonstrate the superior performance of our MMF model compared to single-model approaches and current state-of-the-art techniques based on model evaluation criteria.

Machine Learning for Data Fusion:A Fuzzy AHP Approach for Open Issues

Data fusion generates fused data by combining multiple sources,resulting in information that is more consistent,accurate,and useful than any individual source and more reliable and consistent than the raw original data,which are often imperfect,inconsistent,complex,and uncertain.Traditional data fusion methods like probabilistic fusion,set-based fusion,and evidential belief reasoning fusion methods are computationally complex and require accurate classification and proper handling of raw data.Data fusion is the process of integrating multiple data sources.Data filtering means examining a dataset to exclude,rearrange,or apportion data according to the criteria.Different sensors generate a large amount of data,requiring the development of machine learning(ML)algorithms to overcome the challenges of traditional methods.The advancement in hardware acceleration and the abundance of data from various sensors have led to the development of machine learning(ML)algorithms,expected to address the limitations of traditional methods.However,many open issues still exist as machine learning algorithms are used for data fusion.From the literature,nine issues have been identified irrespective of any application.The decision-makers should pay attention to these issues as data fusion becomes more applicable and successful.A fuzzy analytical hierarchical process(FAHP)enables us to handle these issues.It helps to get the weights for each corresponding issue and rank issues based on these calculated weights.The most significant issue identified is the lack of deep learning models used for data fusion that improve accuracy and learning quality weighted 0.141.The least significant one is the cross-domain multimodal data fusion weighted 0.076 because the whole semantic knowledge for multimodal data cannot be captured.

Fusion-Based Deep Learning Model for Automated Forest Fire Detection

Earth resource and environmental monitoring are essential areas that can be used to investigate the environmental conditions and natural resources supporting sustainable policy development,regulatory measures,and their implementation elevating the environment.Large-scale forest fire is considered a major harmful hazard that affects climate change and life over the globe.Therefore,the early identification of forest fires using automated tools is essential to avoid the spread of fire to a large extent.Therefore,this paper focuses on the design of automated forest fire detection using a fusion-based deep learning(AFFD-FDL)model for environmental monitoring.The AFFDFDL technique involves the design of an entropy-based fusion model for feature extraction.The combination of the handcrafted features using histogram of gradients(HOG)with deep features using SqueezeNet and Inception v3 models.Besides,an optimal extreme learning machine(ELM)based classifier is used to identify the existence of fire or not.In order to properly tune the parameters of the ELM model,the oppositional glowworm swarm optimization(OGSO)algorithm is employed and thereby improves the forest fire detection performance.A wide range of simulation analyses takes place on a benchmark dataset and the results are inspected under several aspects.The experimental results highlighted the betterment of the AFFD-FDL technique over the recent state of art techniques.

PowerDetector:Malicious PowerShell Script Family Classification Based on Multi-Modal Semantic Fusion and Deep Learning

Power Shell has been widely deployed in fileless malware and advanced persistent threat(APT)attacks due to its high stealthiness and live-off-theland technique.However,existing works mainly focus on deobfuscation and malicious detection,lacking the malicious Power Shell families classification and behavior analysis.Moreover,the state-of-the-art methods fail to capture fine-grained features and semantic relationships,resulting in low robustness and accuracy.To this end,we propose Power Detector,a novel malicious Power Shell script detector based on multimodal semantic fusion and deep learning.Specifically,we design four feature extraction methods to extract key features from character,token,abstract syntax tree(AST),and semantic knowledge graph.Then,we intelligently design four embeddings(i.e.,Char2Vec,Token2Vec,AST2Vec,and Rela2Vec) and construct a multi-modal fusion algorithm to concatenate feature vectors from different views.Finally,we propose a combined model based on transformer and CNN-Bi LSTM to implement Power Shell family detection.Our experiments with five types of Power Shell attacks show that PowerDetector can accurately detect various obfuscated and stealth PowerShell scripts,with a 0.9402 precision,a 0.9358 recall,and a 0.9374 F1-score.Furthermore,through singlemodal and multi-modal comparison experiments,we demonstrate that PowerDetector’s multi-modal embedding and deep learning model can achieve better accuracy and even identify more unknown attacks.

An Analysis Model of Learners’ Online Learning Status Based on Deep Neural Network and Multi-Dimensional Information Fusion

The learning status of learners directly affects the quality of learning.Compared with offline teachers,it is difficult for online teachers to capture the learning status of students in the whole class,and it is even more difficult to continue to pay attention to studentswhile teaching.Therefore,this paper proposes an online learning state analysis model based on a convolutional neural network and multi-dimensional information fusion.Specifically,a facial expression recognition model and an eye state recognition model are constructed to detect students’emotions and fatigue,respectively.By integrating the detected data with the homework test score data after online learning,an analysis model of students’online learning status is constructed.According to the PAD model,the learning state is expressed as three dimensions of students’understanding,engagement and interest,and then analyzed from multiple perspectives.Finally,the proposed model is applied to actual teaching,and procedural analysis of 5 different types of online classroom learners is carried out,and the validity of the model is verified by comparing with the results of the manual analysis.

Feature Fusion-Based Deep Learning Network to Recognize Table Tennis Actions

A system for classifying four basic table tennis strokes using wearable devices and deep learning networks is proposed in this study.The wearable device consisted of a six-axis sensor,Raspberry Pi 3,and a power bank.Multiple kernel sizes were used in convolutional neural network(CNN)to evaluate their performance for extracting features.Moreover,a multiscale CNN with two kernel sizes was used to perform feature fusion at different scales in a concatenated manner.The CNN achieved recognition of the four table tennis strokes.Experimental data were obtained from20 research participants who wore sensors on the back of their hands while performing the four table tennis strokes in a laboratory environment.The data were collected to verify the performance of the proposed models for wearable devices.Finally,the sensor and multi-scale CNN designed in this study achieved accuracy and F1 scores of 99.58%and 99.16%,respectively,for the four strokes.The accuracy for five-fold cross validation was 99.87%.This result also shows that the multi-scale convolutional neural network has better robustness after fivefold cross validation.

Data and Ensemble Machine Learning Fusion Based Intelligent Software Defect Prediction System

The software engineering field has long focused on creating high-quality software despite limited resources.Detecting defects before the testing stage of software development can enable quality assurance engineers to con-centrate on problematic modules rather than all the modules.This approach can enhance the quality of the final product while lowering development costs.Identifying defective modules early on can allow for early corrections and ensure the timely delivery of a high-quality product that satisfies customers and instills greater confidence in the development team.This process is known as software defect prediction,and it can improve end-product quality while reducing the cost of testing and maintenance.This study proposes a software defect prediction system that utilizes data fusion,feature selection,and ensemble machine learning fusion techniques.A novel filter-based metric selection technique is proposed in the framework to select the optimum features.A three-step nested approach is presented for predicting defective modules to achieve high accuracy.In the first step,three supervised machine learning techniques,including Decision Tree,Support Vector Machines,and Naïve Bayes,are used to detect faulty modules.The second step involves integrating the predictive accuracy of these classification techniques through three ensemble machine-learning methods:Bagging,Voting,and Stacking.Finally,in the third step,a fuzzy logic technique is employed to integrate the predictive accuracy of the ensemble machine learning techniques.The experiments are performed on a fused software defect dataset to ensure that the developed fused ensemble model can perform effectively on diverse datasets.Five NASA datasets are integrated to create the fused dataset:MW1,PC1,PC3,PC4,and CM1.According to the results,the proposed system exhibited superior performance to other advanced techniques for predicting software defects,achieving a remarkable accuracy rate of 92.08%.

Entropy Based Feature Fusion Using Deep Learning for Waste Object Detection and Classification Model

Object Detection is the task of localization and classification of objects in a video or image.In recent times,because of its widespread applications,it has obtained more importance.In the modern world,waste pollution is one significant environmental problem.The prominence of recycling is known very well for both ecological and economic reasons,and the industry needs higher efficiency.Waste object detection utilizing deep learning(DL)involves training a machine-learning method to classify and detect various types of waste in videos or images.This technology is utilized for several purposes recycling and sorting waste,enhancing waste management and reducing environmental pollution.Recent studies of automatic waste detection are difficult to compare because of the need for benchmarks and broadly accepted standards concerning the employed data andmetrics.Therefore,this study designs an Entropy-based Feature Fusion using Deep Learning forWasteObject Detection and Classification(EFFDL-WODC)algorithm.The presented EFFDL-WODC system inherits the concepts of feature fusion and DL techniques for the effectual recognition and classification of various kinds of waste objects.In the presented EFFDL-WODC system,two major procedures can be contained,such as waste object detection and waste object classification.For object detection,the EFFDL-WODC technique uses a YOLOv7 object detector with a fusionbased backbone network.In addition,entropy feature fusion-based models such as VGG-16,SqueezeNet,and NASNetmodels are used.Finally,the EFFDL-WODC technique uses a graph convolutional network(GCN)model performed for the classification of detected waste objects.The performance validation of the EFFDL-WODC approach was validated on the benchmark database.The comprehensive comparative results demonstrated the improved performance of the EFFDL-WODC technique over recent approaches.

Leveraging Multimodal Ensemble Fusion-Based Deep Learning for COVID-19 on Chest Radiographs

Recently,COVID-19 has posed a challenging threat to researchers,scientists,healthcare professionals,and administrations over the globe,from its diagnosis to its treatment.The researchers are making persistent efforts to derive probable solutions formanaging the pandemic in their areas.One of the widespread and effective ways to detect COVID-19 is to utilize radiological images comprising X-rays and computed tomography(CT)scans.At the same time,the recent advances in machine learning(ML)and deep learning(DL)models show promising results in medical imaging.Particularly,the convolutional neural network(CNN)model can be applied to identifying abnormalities on chest radiographs.While the epidemic of COVID-19,much research is led on processing the data compared with DL techniques,particularly CNN.This study develops an improved fruit fly optimization with a deep learning-enabled fusion(IFFO-DLEF)model for COVID-19 detection and classification.The major intention of the IFFO-DLEF model is to investigate the presence or absence of COVID-19.To do so,the presented IFFODLEF model applies image pre-processing at the initial stage.In addition,the ensemble of three DL models such as DenseNet169,EfficientNet,and ResNet50,are used for feature extraction.Moreover,the IFFO algorithm with a multilayer perceptron(MLP)classification model is utilized to identify and classify COVID-19.The parameter optimization of the MLP approach utilizing the IFFO technique helps in accomplishing enhanced classification performance.The experimental result analysis of the IFFO-DLEF model carried out on the CXR image database portrayed the better performance of the presented IFFO-DLEF model over recent approaches.

Feature Fusion Based Deep Transfer Learning Based Human Gait Classification Model

Gait is a biological typical that defines the method by that people walk.Walking is the most significant performance which keeps our day-to-day life and physical condition.Surface electromyography(sEMG)is a weak bioelectric signal that portrays the functional state between the human muscles and nervous system to any extent.Gait classifiers dependent upon sEMG signals are extremely utilized in analysing muscle diseases and as a guide path for recovery treatment.Several approaches are established in the works for gait recognition utilizing conventional and deep learning(DL)approaches.This study designs an Enhanced Artificial Algae Algorithm with Hybrid Deep Learning based Human Gait Classification(EAAA-HDLGR)technique on sEMG signals.The EAAA-HDLGR technique extracts the time domain(TD)and frequency domain(FD)features from the sEMG signals and is fused.In addition,the EAAA-HDLGR technique exploits the hybrid deep learning(HDL)model for gait recognition.At last,an EAAA-based hyperparameter optimizer is applied for the HDL model,which is mainly derived from the quasi-oppositional based learning(QOBL)concept,showing the novelty of the work.A brief classifier outcome of the EAAA-HDLGR technique is examined under diverse aspects,and the results indicate improving the EAAA-HDLGR technique.The results imply that the EAAA-HDLGR technique accomplishes improved results with the inclusion of EAAA on gait recognition.

Hyperspectral Image Super-Resolution Meets Deep Learning:A Survey and Perspective

Hyperspectral image super-resolution,which refers to reconstructing the high-resolution hyperspectral image from the input low-resolution observation,aims to improve the spatial resolution of the hyperspectral image,which is beneficial for subsequent applications.The development of deep learning has promoted significant progress in hyperspectral image super-resolution,and the powerful expression capabilities of deep neural networks make the predicted results more reliable.Recently,several latest deep learning technologies have made the hyperspectral image super-resolution method explode.However,a comprehensive review and analysis of the latest deep learning methods from the hyperspectral image super-resolution perspective is absent.To this end,in this survey,we first introduce the concept of hyperspectral image super-resolution and classify the methods from the perspectives with or without auxiliary information.Then,we review the learning-based methods in three categories,including single hyperspectral image super-resolution,panchromatic-based hyperspectral image super-resolution,and multispectral-based hyperspectral image super-resolution.Subsequently,we summarize the commonly used hyperspectral dataset,and the evaluations for some representative methods in three categories are performed qualitatively and quantitatively.Moreover,we briefly introduce several typical applications of hyperspectral image super-resolution,including ground object classification,urban change detection,and ecosystem monitoring.Finally,we provide the conclusion and challenges in existing learning-based methods,looking forward to potential future research directions.