CAEFusion: A New Convolutional Autoencoder-Based Infrared and Visible Light Image Fusion Algorithm

To address the issues of incomplete information,blurred details,loss of details,and insufficient contrast in infrared and visible image fusion,an image fusion algorithm based on a convolutional autoencoder is proposed.The region attention module is meant to extract the background feature map based on the distinct properties of the background feature map and the detail feature map.A multi-scale convolution attention module is suggested to enhance the communication of feature information.At the same time,the feature transformation module is introduced to learn more robust feature representations,aiming to preserve the integrity of image information.This study uses three available datasets from TNO,FLIR,and NIR to perform thorough quantitative and qualitative trials with five additional algorithms.The methods are assessed based on four indicators:information entropy(EN),standard deviation(SD),spatial frequency(SF),and average gradient(AG).Object detection experiments were done on the M3FD dataset to further verify the algorithm’s performance in comparison with five other algorithms.The algorithm’s accuracy was evaluated using the mean average precision at a threshold of 0.5(mAP@0.5)index.Comprehensive experimental findings show that CAEFusion performs well in subjective visual and objective evaluation criteria and has promising potential in downstream object detection tasks.

Infrared and Visible Image Fusion Based on Res2Net-Transformer Automatic Encoding and Decoding

A novel image fusion network framework with an autonomous encoder and decoder is suggested to increase thevisual impression of fused images by improving the quality of infrared and visible light picture fusion. The networkcomprises an encoder module, fusion layer, decoder module, and edge improvementmodule. The encoder moduleutilizes an enhanced Inception module for shallow feature extraction, then combines Res2Net and Transformerto achieve deep-level co-extraction of local and global features from the original picture. An edge enhancementmodule (EEM) is created to extract significant edge features. A modal maximum difference fusion strategy isintroduced to enhance the adaptive representation of information in various regions of the source image, therebyenhancing the contrast of the fused image. The encoder and the EEM module extract features, which are thencombined in the fusion layer to create a fused picture using the decoder. Three datasets were chosen to test thealgorithmproposed in this paper. The results of the experiments demonstrate that the network effectively preservesbackground and detail information in both infrared and visible images, yielding superior outcomes in subjectiveand objective evaluations.

Fusion SST from Infrared and Microwave Measurement of FY-3D Meteorological Satellite

Sea surface temperature(SST)is one of the important parameters of global ocean and climate research,which can be retrieved by satellite infrared and passive microwave remote sensing instruments.While satellite infrared SST offers high spatial resolution,it is limited by cloud cover.On the other hand,passive microwave SST provides all-weather observation but suffers from poor spatial resolution and susceptibility to environmental factors such as rainfall,coastal effects,and high wind speeds.To achieve high-precision,comprehensive,and high-resolution SST data,it is essential to fuse infrared and microwave SST measurements.In this study,data from the Fengyun-3D(FY-3D)medium resolution spectral imager II(MERSI-II)SST and microwave imager(MWRI)SST were fused.Firstly,the accuracy of both MERSIII SST and MWRI SST was verified,and the latter was bilinearly interpolated to match the 5km resolution grid of MERSI SST.After pretreatment and quality control of MERSI SST and MWRI SST,a Piece-Wise Regression method was employed to correct biases in MWRI SST.Subsequently,SST data were selected based on spatial resolution and accuracy within a 3-day window of the analysis date.Finally,an optimal interpolation method was applied to fuse the FY-3D MERSI-II SST and MWRI SST.The results demonstrated a significant improvement in spatial coverage compared to MERSI-II SST and MWRI SST.Furthermore,the fusion SST retained true spatial distribution details and exhibited an accuracy of–0.12±0.74℃compared to OSTIA SST.This study has improved the accuracy of FY satellite fusion SST products in China.

Disparity estimation for multi-scale multi-sensor fusion

The perception module of advanced driver assistance systems plays a vital role.Perception schemes often use a single sensor for data processing and environmental perception or adopt the information processing results of various sensors for the fusion of the detection layer.This paper proposes a multi-scale and multi-sensor data fusion strategy in the front end of perception and accomplishes a multi-sensor function disparity map generation scheme.A binocular stereo vision sensor composed of two cameras and a light deterction and ranging(LiDAR)sensor is used to jointly perceive the environment,and a multi-scale fusion scheme is employed to improve the accuracy of the disparity map.This solution not only has the advantages of dense perception of binocular stereo vision sensors but also considers the perception accuracy of LiDAR sensors.Experiments demonstrate that the multi-scale multi-sensor scheme proposed in this paper significantly improves disparity map estimation.

Research on the Mechanism of Multi-Sensor Fusion Configuration Based on the Optimal Principle of the Vehicle

In order to address the issue of sensor configuration redundancy in intelligent driving,this paper constructs a multi-objective optimization model that considers cost,coverage ability,and perception performance.And then,combining a specific set of parameters,the NSGA-II algorithm is used to solve the multi-objective model established in this paper,and a Pareto front containing 24 typical configuration schemes is extracted after considering empirical constraints.Finally,using the decision preference method proposed in this paper that combines subjective and objective factors,decision scores are calculated and ranked for various configuration schemes from both cost and performance preferences.The research results indicate that the multi-objective optimization model established in this paper can screen and optimize various configuration schemes from the optimal principle of the vehicle,and the optimized configuration schemes can be quantitatively ranked to obtain the decision results for the vehicle under different preference tendencies.

State Estimation Method for GNSS/INS/Visual Multi-sensor Fusion Based on Factor Graph Optimization for Unmanned System

With the development of unmanned driving technology,intelligent robots and drones,high-precision localization,navigation and state estimation technologies have also made great progress.Traditional global navigation satellite system/inertial navigation system(GNSS/INS)integrated navigation systems can provide high-precision navigation information continuously.However,when this system is applied to indoor or GNSS-denied environments,such as outdoor substations with strong electromagnetic interference and complex dense spaces,it is often unable to obtain high-precision GNSS positioning data.The positioning and orientation errors will diverge and accumulate rapidly,which cannot meet the high-precision localization requirements in large-scale and long-distance navigation scenarios.This paper proposes a method of high-precision state estimation with fusion of GNSS/INS/Vision using a nonlinear optimizer factor graph optimization as the basis for multi-source optimization.Through the collected experimental data and simulation results,this system shows good performance in the indoor environment and the environment with partial GNSS signal loss.

Research on Infrared Image Fusion Technology Based on Road Crack Detection

This study aimed to propose road crack detection method based on infrared image fusion technology.By analyzing the characteristics of road crack images,this method uses a variety of infrared image fusion methods to process different types of images.The use of this method allows the detection of road cracks,which not only reduces the professional requirements for inspectors,but also improves the accuracy of road crack detection.Based on infrared image processing technology,on the basis of in-depth analysis of infrared image features,a road crack detection method is proposed,which can accurately identify the road crack location,direction,length,and other characteristic information.Experiments showed that this method has a good effect,and can meet the requirement of road crack detection.

Infrared polarization image fusion based on combination of NSST and improved PCA

In view of the problem that current mainstream fusion method of infrared polarization image—Multiscale Geometry Analysis method only focuses on a certain characteristic to image representation.And spatial domain fusion method,Principal Component Analysis(PCA)method has the shortcoming of losing small target,this paper presents a new fusion method of infrared polarization images based on combination of Nonsubsampled Shearlet Transformation(NSST)and improved PCA.This method can make full use of the effectiveness to image details expressed by NSST and the characteristics that PCA can highlight the main features of images.The combination of the two methods can integrate the complementary features of themselves to retain features of targets and image details fully.Firstly,intensity and polarization images are decomposed into low frequency and high frequency components with different directions by NSST.Secondly,the low frequency components are fused with improved PCA,while the high frequency components are fused by joint decision making rule with local energy and local variance.Finally,the fused image is reconstructed with the inverse NSST to obtain the final fused image of infrared polarization.The experiment results show that the method proposed has higher advantages than other methods in terms of detail preservation and visual effect.

An infrared target intrusion detection method based on feature fusion and enhancement

Infrared target intrusion detection has significant applications in the fields of military defence and intelligent warning.In view of the characteristics of intrusion targets as well as inspection difficulties,an infrared target intrusion detection algorithm based on feature fusion and enhancement was proposed.This algorithm combines static target mode analysis and dynamic multi-frame correlation detection to extract infrared target features at different levels.Among them,LBP texture analysis can be used to effectively identify the posterior feature patterns which have been contained in the target library,while motion frame difference method can detect the moving regions of the image,improve the integrity of target regions such as camouflage,sheltering and deformation.In order to integrate the advantages of the two methods,the enhanced convolutional neural network was designed and the feature images obtained by the two methods were fused and enhanced.The enhancement module of the network strengthened and screened the targets,and realized the background suppression of infrared images.Based on the experiments,the effect of the proposed method and the comparison method on the background suppression and detection performance was evaluated,and the results showed that the SCRG and BSF values of the method in this paper had a better performance in multiple data sets,and it’s detection performance was far better than the comparison algorithm.The experiment results indicated that,compared with traditional infrared target detection methods,the proposed method could detect the infrared invasion target more accurately,and suppress the background noise more effectively.

Feature-Based Fusion of Dual Band Infrared Image Using Multiple Pulse Coupled Neural Network

To improve the quality of the infrared image and enhance the information of the object,a dual band infrared image fusion method based on feature extraction and a novel multiple pulse coupled neural network(multi-PCNN)is proposed.In this multi-PCNN fusion scheme,the auxiliary PCNN which captures the characteristics of feature image extracting from the infrared image is used to modulate the main PCNN,whose input could be original infrared image.Meanwhile,to make the PCNN fusion effect consistent with the human vision system,Laplacian energy is adopted to obtain the value of adaptive linking strength in PCNN.After that,the original dual band infrared images are reconstructed by using a weight fusion rule with the fire mapping images generated by the main PCNNs to obtain the fused image.Compared to wavelet transforms,Laplacian pyramids and traditional multi-PCNNs,fusion images based on our method have more information,rich details and clear edges.

Intelligent Fusion of Infrared and Visible Image Data Based on Convolutional Sparse Representation and Improved Pulse-Coupled Neural Network

Multi-source information can be obtained through the fusion of infrared images and visible light images,which have the characteristics of complementary information.However,the existing acquisition methods of fusion images have disadvantages such as blurred edges,low contrast,and loss of details.Based on convolution sparse representation and improved pulse-coupled neural network this paper proposes an image fusion algorithm that decompose the source images into high-frequency and low-frequency subbands by non-subsampled Shearlet Transform(NSST).Furthermore,the low-frequency subbands were fused by convolutional sparse representation(CSR),and the high-frequency subbands were fused by an improved pulse coupled neural network(IPCNN)algorithm,which can effectively solve the problem of difficulty in setting parameters of the traditional PCNN algorithm,improving the performance of sparse representation with details injection.The result reveals that the proposed method in this paper has more advantages than the existing mainstream fusion algorithms in terms of visual effects and objective indicators.

An infrared and visible image fusion method based upon multi-scale and top-hat transforms

The high-frequency components in the traditional multi-scale transform method are approximately sparse, which can represent different information of the details. But in the low-frequency component, the coefficients around the zero value are very few, so we cannot sparsely represent low-frequency image information. The low-frequency component contains the main energy of the image and depicts the profile of the image. Direct fusion of the low-frequency component will not be conducive to obtain highly accurate fusion result. Therefore, this paper presents an infrared and visible image fusion method combining the multi-scale and top-hat transforms. On one hand, the new top-hat-transform can effectively extract the salient features of the low-frequency component. On the other hand, the multi-scale transform can extract highfrequency detailed information in multiple scales and from diverse directions. The combination of the two methods is conducive to the acquisition of more characteristics and more accurate fusion results. Among them, for the low-frequency component, a new type of top-hat transform is used to extract low-frequency features, and then different fusion rules are applied to fuse the low-frequency features and low-frequency background; for high-frequency components, the product of characteristics method is used to integrate the detailed information in high-frequency. Experimental results show that the proposed algorithm can obtain more detailed information and clearer infrared target fusion results than the traditional multiscale transform methods. Compared with the state-of-the-art fusion methods based on sparse representation, the proposed algorithm is simple and efficacious, and the time consumption is significantly reduced.

Progress and Achievements of Multi-sensor Fusion Navigation in China during 2019—2023

Global Navigation Satellite System(GNSS)can provide all-weather,all-time,high-precision positioning,navigation and timing services,which plays an important role in national security,national economy,public life and other aspects.However,in environments with limited satellite signals such as urban canyons,tunnels,and indoor spaces,it is difficult to provide accurate and reliable positioning services only by satellite navigation.Multi-source sensor integrated navigation can effectively overcome the limitations of single-sensor navigation through the fusion of different types of sensor data such as Inertial Measurement Unit(IMU),vision sensor,and LiDAR,and provide more accurate,stable and robust navigation information in complex environments.We summarizes the research status of multi-source sensor integrated navigation technology,and focuses on the representative innovations and applications of integrated navigation and positioning technology by major domestic scientific research institutions in China during 2019—2023.

Fusion of Infrared and Visible Images Using Fuzzy Based Siamese Convolutional Network

Traditional techniques based on image fusion are arduous in integrating complementary or heterogeneous infrared(IR)/visible(VS)images.Dissimilarities in various kind of features in these images are vital to preserve in the single fused image.Hence,simultaneous preservation of both the aspects at the same time is a challenging task.However,most of the existing methods utilize the manual extraction of features;and manual complicated designing of fusion rules resulted in a blurry artifact in the fused image.Therefore,this study has proposed a hybrid algorithm for the integration of multi-features among two heterogeneous images.Firstly,fuzzification of two IR/VS images has been done by feeding it to the fuzzy sets to remove the uncertainty present in the background and object of interest of the image.Secondly,images have been learned by two parallel branches of the siamese convolutional neural network(CNN)to extract prominent features from the images as well as high-frequency information to produce focus maps containing source image information.Finally,the obtained focused maps which contained the detailed integrated information are directly mapped with the source image via pixelwise strategy to result in fused image.Different parameters have been used to evaluate the performance of the proposed image fusion by achieving 1.008 for mutual information(MI),0.841 for entropy(EG),0.655 for edge information(EI),0.652 for human perception(HP),and 0.980 for image structural similarity(ISS).Experimental results have shown that the proposed technique has attained the best qualitative and quantitative results using 78 publically available images in comparison to the existing discrete cosine transform(DCT),anisotropic diffusion&karhunen-loeve(ADKL),guided filter(GF),random walk(RW),principal component analysis(PCA),and convolutional neural network(CNN)methods.

PF-YOLOv4-Tiny: Towards Infrared Target Detection on Embedded Platform

Infrared target detection models are more required than ever before to be deployed on embedded platforms,which requires models with less memory consumption and better real-time performance while considering accuracy.To address the above challenges,we propose a modified You Only Look Once(YOLO)algorithm PF-YOLOv4-Tiny.The algorithm incorpo-rates spatial pyramidal pooling(SPP)and squeeze-and-excitation(SE)visual attention modules to enhance the target localization capability.The PANet-based-feature pyramid networks(P-FPN)are proposed to transfer semantic information and location information simultaneously to ameliorate detection accuracy.To lighten the network,the standard convolutions other than the backbone network are replaced with depthwise separable convolutions.In post-processing the images,the soft-non-maximum suppression(soft-NMS)algorithm is employed to subside the missed and false detection problems caused by the occlusion between targets.The accuracy of our model can finally reach 61.75%,while the total Params is only 9.3 M and GFLOPs is 11.At the same time,the inference speed reaches 87 FPS on NVIDIA GeForce GTX 1650 Ti,which can meet the requirements of the infrared target detection algorithm for the embedded deployments.

Sub-Regional Infrared-Visible Image Fusion Using Multi-Scale Transformation

Infrared-visible image fusion plays an important role in multi-source data fusion,which has the advantage of integrating useful information from multi-source sensors.However,there are still challenges in target enhancement and visual improvement.To deal with these problems,a sub-regional infrared-visible image fusion method(SRF)is proposed.First,morphology and threshold segmentation is applied to extract targets interested in infrared images.Second,the infrared back-ground is reconstructed based on extracted targets and the visible image.Finally,target and back-ground regions are fused using a multi-scale transform.Experimental results are obtained using public data for comparison and evaluation,which demonstrate that the proposed SRF has poten-tial benefits over other methods.

Driver Fatigue Detection System Based on Colored and Infrared Eye Features Fusion

Real-time detection of driver fatigue status is of great significance for road traffic safety.In this paper,a proposed novel driver fatigue detection method is able to detect the driver’s fatigue status around the clock.The driver’s face images were captured by a camera with a colored lens and an infrared lens mounted above the dashboard.The landmarks of the driver’s face were labeled and the eye-area was segmented.By calculating the aspect ratios of the eyes,the duration of eye closure,frequency of blinks and PERCLOS of both colored and infrared,fatigue can be detected.Based on the change of light intensity detected by a photosensitive device,the weight matrix of the colored features and the infrared features was adjusted adaptively to reduce the impact of lighting on fatigue detection.Video samples of the driver’s face were recorded in the test vehicle.After training the classification model,the results showed that our method has high accuracy on driver fatigue detection in both daytime and nighttime.

Research on Optimal Preload Method of Controllable Rolling Bearing Based on Multisensor Fusion

Angular contact ball bearings have been widely used in machine tool spindles,and the bearing preload plays an important role in the performance of the spindle.In order to solve the problems of the traditional optimal preload prediction method limited by actual conditions and uncertainties,a roller bearing preload test method based on the improved D-S evidence theorymulti-sensor fusion method was proposed.First,a novel controllable preload system is proposed and evaluated.Subsequently,multiple sensors are employed to collect data on the bearing parameters during preload application.Finally,a multisensor fusion algorithm is used to make predictions,and a neural network is used to optimize the fitting of the preload data.The limitations of conventional preload testing methods are identified,and the integration of complementary information frommultiple sensors is used to achieve accurate predictions,offering valuable insights into the optimal preload force.Experimental results demonstrate that the multi-sensor fusion approach outperforms traditional methods in accurately measuring the optimal preload for rolling bearings.

Automated Registration for Infrared Image Based on Wavelet Analysis

To develop a quick, accurate and antinoise automated image registration technique for infrared images, the wavelet analysis technique was used to extract the feature points in two images followed by the compensation for input image with angle difference between them. A hi erarchical feature matching algorithm was adopted to get the final transform parameters between the two images. The simulation results for two infrared images show that the method can effectively, quickly and accurately register images and be antinoise to some extent.

Multiscale feature learning and attention mechanism for infrared and visible image fusion

Current fusion methods for infrared and visible images tend to extract features at a single scale,which results in insufficient detail and incomplete feature preservation.To address these issues,we propose an infrared and visible image fusion network based on a multiscale feature learning and attention mechanism(MsAFusion).A multiscale dilation convolution framework is employed to capture image features across various scales and broaden the perceptual scope.Furthermore,an attention network is introduced to enhance the focus on salient targets in infrared images and detailed textures in visible images.To compensate for information loss during convolution,jump connections are utilized during the image reconstruction phase.The fusion process utilizes a combined loss function consisting of pixel loss and gradient loss for unsupervised fusion of infrared and visible images.Extensive experiments on the dataset of electricity facilities demonstrate that our proposed method outperforms nine state-of-theart methods in terms of visual perception and four objective evaluation metrics.