Infrared and Visible Image Fusion Based on Res2Net-Transformer Automatic Encoding and Decoding

A novel image fusion network framework with an autonomous encoder and decoder is suggested to increase thevisual impression of fused images by improving the quality of infrared and visible light picture fusion. The networkcomprises an encoder module, fusion layer, decoder module, and edge improvementmodule. The encoder moduleutilizes an enhanced Inception module for shallow feature extraction, then combines Res2Net and Transformerto achieve deep-level co-extraction of local and global features from the original picture. An edge enhancementmodule (EEM) is created to extract significant edge features. A modal maximum difference fusion strategy isintroduced to enhance the adaptive representation of information in various regions of the source image, therebyenhancing the contrast of the fused image. The encoder and the EEM module extract features, which are thencombined in the fusion layer to create a fused picture using the decoder. Three datasets were chosen to test thealgorithmproposed in this paper. The results of the experiments demonstrate that the network effectively preservesbackground and detail information in both infrared and visible images, yielding superior outcomes in subjectiveand objective evaluations.

CAEFusion: A New Convolutional Autoencoder-Based Infrared and Visible Light Image Fusion Algorithm

To address the issues of incomplete information,blurred details,loss of details,and insufficient contrast in infrared and visible image fusion,an image fusion algorithm based on a convolutional autoencoder is proposed.The region attention module is meant to extract the background feature map based on the distinct properties of the background feature map and the detail feature map.A multi-scale convolution attention module is suggested to enhance the communication of feature information.At the same time,the feature transformation module is introduced to learn more robust feature representations,aiming to preserve the integrity of image information.This study uses three available datasets from TNO,FLIR,and NIR to perform thorough quantitative and qualitative trials with five additional algorithms.The methods are assessed based on four indicators:information entropy(EN),standard deviation(SD),spatial frequency(SF),and average gradient(AG).Object detection experiments were done on the M3FD dataset to further verify the algorithm’s performance in comparison with five other algorithms.The algorithm’s accuracy was evaluated using the mean average precision at a threshold of 0.5(mAP@0.5)index.Comprehensive experimental findings show that CAEFusion performs well in subjective visual and objective evaluation criteria and has promising potential in downstream object detection tasks.

Fusion of Infrared and Visible Images Using Fuzzy Based Siamese Convolutional Network

Traditional techniques based on image fusion are arduous in integrating complementary or heterogeneous infrared(IR)/visible(VS)images.Dissimilarities in various kind of features in these images are vital to preserve in the single fused image.Hence,simultaneous preservation of both the aspects at the same time is a challenging task.However,most of the existing methods utilize the manual extraction of features;and manual complicated designing of fusion rules resulted in a blurry artifact in the fused image.Therefore,this study has proposed a hybrid algorithm for the integration of multi-features among two heterogeneous images.Firstly,fuzzification of two IR/VS images has been done by feeding it to the fuzzy sets to remove the uncertainty present in the background and object of interest of the image.Secondly,images have been learned by two parallel branches of the siamese convolutional neural network(CNN)to extract prominent features from the images as well as high-frequency information to produce focus maps containing source image information.Finally,the obtained focused maps which contained the detailed integrated information are directly mapped with the source image via pixelwise strategy to result in fused image.Different parameters have been used to evaluate the performance of the proposed image fusion by achieving 1.008 for mutual information(MI),0.841 for entropy(EG),0.655 for edge information(EI),0.652 for human perception(HP),and 0.980 for image structural similarity(ISS).Experimental results have shown that the proposed technique has attained the best qualitative and quantitative results using 78 publically available images in comparison to the existing discrete cosine transform(DCT),anisotropic diffusion&karhunen-loeve(ADKL),guided filter(GF),random walk(RW),principal component analysis(PCA),and convolutional neural network(CNN)methods.

Intelligent Fusion of Infrared and Visible Image Data Based on Convolutional Sparse Representation and Improved Pulse-Coupled Neural Network

Multi-source information can be obtained through the fusion of infrared images and visible light images,which have the characteristics of complementary information.However,the existing acquisition methods of fusion images have disadvantages such as blurred edges,low contrast,and loss of details.Based on convolution sparse representation and improved pulse-coupled neural network this paper proposes an image fusion algorithm that decompose the source images into high-frequency and low-frequency subbands by non-subsampled Shearlet Transform(NSST).Furthermore,the low-frequency subbands were fused by convolutional sparse representation(CSR),and the high-frequency subbands were fused by an improved pulse coupled neural network(IPCNN)algorithm,which can effectively solve the problem of difficulty in setting parameters of the traditional PCNN algorithm,improving the performance of sparse representation with details injection.The result reveals that the proposed method in this paper has more advantages than the existing mainstream fusion algorithms in terms of visual effects and objective indicators.

An infrared and visible image fusion method based upon multi-scale and top-hat transforms

The high-frequency components in the traditional multi-scale transform method are approximately sparse, which can represent different information of the details. But in the low-frequency component, the coefficients around the zero value are very few, so we cannot sparsely represent low-frequency image information. The low-frequency component contains the main energy of the image and depicts the profile of the image. Direct fusion of the low-frequency component will not be conducive to obtain highly accurate fusion result. Therefore, this paper presents an infrared and visible image fusion method combining the multi-scale and top-hat transforms. On one hand, the new top-hat-transform can effectively extract the salient features of the low-frequency component. On the other hand, the multi-scale transform can extract highfrequency detailed information in multiple scales and from diverse directions. The combination of the two methods is conducive to the acquisition of more characteristics and more accurate fusion results. Among them, for the low-frequency component, a new type of top-hat transform is used to extract low-frequency features, and then different fusion rules are applied to fuse the low-frequency features and low-frequency background; for high-frequency components, the product of characteristics method is used to integrate the detailed information in high-frequency. Experimental results show that the proposed algorithm can obtain more detailed information and clearer infrared target fusion results than the traditional multiscale transform methods. Compared with the state-of-the-art fusion methods based on sparse representation, the proposed algorithm is simple and efficacious, and the time consumption is significantly reduced.

Infrared polarization image fusion based on combination of NSST and improved PCA

In view of the problem that current mainstream fusion method of infrared polarization image—Multiscale Geometry Analysis method only focuses on a certain characteristic to image representation.And spatial domain fusion method,Principal Component Analysis(PCA)method has the shortcoming of losing small target,this paper presents a new fusion method of infrared polarization images based on combination of Nonsubsampled Shearlet Transformation(NSST)and improved PCA.This method can make full use of the effectiveness to image details expressed by NSST and the characteristics that PCA can highlight the main features of images.The combination of the two methods can integrate the complementary features of themselves to retain features of targets and image details fully.Firstly,intensity and polarization images are decomposed into low frequency and high frequency components with different directions by NSST.Secondly,the low frequency components are fused with improved PCA,while the high frequency components are fused by joint decision making rule with local energy and local variance.Finally,the fused image is reconstructed with the inverse NSST to obtain the final fused image of infrared polarization.The experiment results show that the method proposed has higher advantages than other methods in terms of detail preservation and visual effect.

Research on Infrared Image Fusion Technology Based on Road Crack Detection

This study aimed to propose road crack detection method based on infrared image fusion technology.By analyzing the characteristics of road crack images,this method uses a variety of infrared image fusion methods to process different types of images.The use of this method allows the detection of road cracks,which not only reduces the professional requirements for inspectors,but also improves the accuracy of road crack detection.Based on infrared image processing technology,on the basis of in-depth analysis of infrared image features,a road crack detection method is proposed,which can accurately identify the road crack location,direction,length,and other characteristic information.Experiments showed that this method has a good effect,and can meet the requirement of road crack detection.

Feature-Based Fusion of Dual Band Infrared Image Using Multiple Pulse Coupled Neural Network

To improve the quality of the infrared image and enhance the information of the object,a dual band infrared image fusion method based on feature extraction and a novel multiple pulse coupled neural network(multi-PCNN)is proposed.In this multi-PCNN fusion scheme,the auxiliary PCNN which captures the characteristics of feature image extracting from the infrared image is used to modulate the main PCNN,whose input could be original infrared image.Meanwhile,to make the PCNN fusion effect consistent with the human vision system,Laplacian energy is adopted to obtain the value of adaptive linking strength in PCNN.After that,the original dual band infrared images are reconstructed by using a weight fusion rule with the fire mapping images generated by the main PCNNs to obtain the fused image.Compared to wavelet transforms,Laplacian pyramids and traditional multi-PCNNs,fusion images based on our method have more information,rich details and clear edges.

Sub-Regional Infrared-Visible Image Fusion Using Multi-Scale Transformation

Infrared-visible image fusion plays an important role in multi-source data fusion,which has the advantage of integrating useful information from multi-source sensors.However,there are still challenges in target enhancement and visual improvement.To deal with these problems,a sub-regional infrared-visible image fusion method(SRF)is proposed.First,morphology and threshold segmentation is applied to extract targets interested in infrared images.Second,the infrared back-ground is reconstructed based on extracted targets and the visible image.Finally,target and back-ground regions are fused using a multi-scale transform.Experimental results are obtained using public data for comparison and evaluation,which demonstrate that the proposed SRF has poten-tial benefits over other methods.

Multiscale feature learning and attention mechanism for infrared and visible image fusion

Current fusion methods for infrared and visible images tend to extract features at a single scale,which results in insufficient detail and incomplete feature preservation.To address these issues,we propose an infrared and visible image fusion network based on a multiscale feature learning and attention mechanism(MsAFusion).A multiscale dilation convolution framework is employed to capture image features across various scales and broaden the perceptual scope.Furthermore,an attention network is introduced to enhance the focus on salient targets in infrared images and detailed textures in visible images.To compensate for information loss during convolution,jump connections are utilized during the image reconstruction phase.The fusion process utilizes a combined loss function consisting of pixel loss and gradient loss for unsupervised fusion of infrared and visible images.Extensive experiments on the dataset of electricity facilities demonstrate that our proposed method outperforms nine state-of-theart methods in terms of visual perception and four objective evaluation metrics.

Multi-sensors Image Fusion via NSCT and GoogLeNet

In order to improve the detail preservation and target information integrity of different sensor fusion images,an image fusion method of different sensors based on non-subsampling contourlet transform(NSCT)and GoogLeNet neural network model is proposed. First,the different sensors images,i. e.,infrared and visible images,are transformed by NSCT to obtain a low frequency sub-band and a series of high frequency sub-bands respectively.Then,the high frequency sub-bands are fused with the max regional energy selection strategy,the low frequency subbands are input into GoogLeNet neural network model to extract feature maps,and the fusion weight matrices are adaptively calculated from the feature maps. Next,the fused low frequency sub-band is obtained with weighted summation. Finally,the fused image is obtained by inverse NSCT. The experimental results demonstrate that the proposed method improves the image visual effect and achieves better performance in both edge retention and mutual information.

Fusion of Infrared and Visible Light Images Based on Region Segmentation

This article proposes a novel method to fuse infrared and visible light images based on region segmentation. Region segmen-tation is used to determine important regions and background information in the input image. The non-subsampled contourlet transform (NSCT) provides a flexible multiresolution,local and directional image expansion,and also a sparse representation for two-dimensional (2-D) piecewise smooth signal building images,and then different fusion rules are applied to fuse the NSCT coefficients fo...

Automated Registration for Infrared Image Based on Wavelet Analysis

To develop a quick, accurate and antinoise automated image registration technique for infrared images, the wavelet analysis technique was used to extract the feature points in two images followed by the compensation for input image with angle difference between them. A hi erarchical feature matching algorithm was adopted to get the final transform parameters between the two images. The simulation results for two infrared images show that the method can effectively, quickly and accurately register images and be antinoise to some extent.

Enhanced target tracking algorithm for autonomous driving based on visible and infrared image fusion

In autonomous driving,target tracking is essential to environmental perception.The study of target tracking algorithms can improve the accuracy of an autonomous driving vehicle’s perception,which is of great significance in ensuring the safety of autonomous driving and promoting the landing of technical applications.This study focuses on the fusion tracking algorithm based on visible and infrared images.The proposed approach utilizes a feature-level image fusion method,dividing the tracking process into two components:image fusion and target tracking.An unsupervised network,Visible and Infrared image Fusion Network(VIF-net),is employed for visible and infrared image fusion in the image fusion part.In the target tracking part,Siamese Region Proposal Network(SiamRPN),based on deep learning,tracks the target with fused images.The fusion tracking algorithm is trained and evaluated on the visible infrared image dataset RGBT234.Experimental results demonstrate that the algorithm outperforms training networks solely based on visible images,proving that the fusion of visible and infrared images in the target tracking algorithm can improve the accuracy of the target tracking even if it is like tracking-based visual images.This improvement is also attributed to the algorithm’s ability to extract infrared image features,augmenting the target tracking accuracy.

Image Fusion Based on Bioinspired Rattlesnake Visual Mechanism Under Lighting Environments of Day and Night Two Levels

This study,grounded in Waxman fusion method,introduces an algorithm for the fusion of visible and infrared images,tailored to a two-level lighting environment,inspired by the mathematical model of the visual receptive field of rattlesnakes and the two-mode cells'mechanism.The research presented here is segmented into three components.In the first segment,we design a preprocessing module to judge the ambient light intensity and divide the lighting environment into two levels:day and night.The second segment proposes two distinct network structures designed specifically for these daytime and nighttime images.For the daytime images,where visible light information is predominant,we feed the ON-VIS signal and the IR-enhanced visual signal into the central excitation and surrounding suppression regions of the ON-center receptive field in the B channel,respectively.Conversely,for nighttime images where infrared information takes precedence,the ON-IR signal and the Visual-enhanced IR signal are separately input into the central excitation and surrounding suppression regions of the ON-center receptive field in the B channel.The outcome is a pseudo-color fused image.The third segment employs five different no-reference image quality assessment methods to evaluate the quality of thirteen sets of pseudo-color images produced by fusing infrared and visible information.These images are then compared with those obtained by six other methods cited in the relevant reference.The empirical results indicate that this study's outcomes surpass the comparative results in terms of average gradient and spatial frequency.Only one or two sets of fused images underperformed in terms of standard deviation and entropy when compared to the control results.Four sets of fused images did not perform as well as the comparison in the QAB/F index.In conclusion,the fused images generated through the proposed method show superior performance in terms of scene detail,visual perception,and image sharpness when compared with their counterparts from other methods.

Pseudo Color Fusion of Infrared and Visible Images Based on the Rattlesnake Vision Imaging System

Image fusion is a key technology in the field of digital image processing.In the present study,an effect-based pseudo color fusion model of infrared and visible images based on the rattlesnake vision imaging system(the rattlesnake bimodal cell fusion mechanism and the visual receptive field model)is proposed.The innovation point of the proposed model lies in the following three features:first,the introduction of a simple mathematical model of the visual receptive field reduce computational complexity;second,the enhanced image is obtained by extracting the common information and unique information of source images,which improves fusion image quality;and third,the Waxman typical fusion structure is improved for the pseudo color image fusion model.The performance of the image fusion model is verified through comparative experiments.In the subjective visual evaluation,we find that the color of the fusion image obtained through the proposed model is natural and can highlight the target and scene details.In the objective quantitative evaluation,we observe that the best values on the four indicators,namely standard deviation,average gradient,entropy,and spatial frequency,accounts for 90%,100%,90%,and 100%,respectively,indicating that the fusion image exhibits superior contrast,image clarity,information content,and overall activity.Experimental results reveal that the performance of the proposed model is superior to that of other models and thus verified the validity and reliability of the model.

Vision Enhancement Technology of Drivers Based on Image Fusion

The rise of urban traffic flow highlights the growing importance of traffic safety.In order to reduce the occurrence rate of traffic accidents,and improve front vision information of vehicle drivers,the method to improve visual information of the vehicle driver in low visibility conditions is put forward based on infrared and visible image fusion technique.The wavelet image confusion algorithm is adopted to decompose the image into low-frequency approximation components and high-frequency detail components.Low-frequency component contains information representing gray value differences.High-frequency component contains the detail information of the image,which is frequently represented by gray standard deviation to assess image quality.To extract feature information of low-frequency component and high-frequency component with different emphases,different fusion operators are used separately by low-frequency and high-frequency components.In the processing of low-frequency component,the fusion rule of weighted regional energy proportion is adopted to improve the brightness of the image,and the fusion rule of weighted regional proportion of standard deviation is used in all the three high-frequency components to enhance the image contrast.The experiments on image fusion of infrared and visible light demonstrate that this image fusion method can effectively improve the image brightness and contrast,and it is suitable for vision enhancement of the low-visibility images.

Fusion of the low-light-level visible and infrared images for night-vision context enhancement

For better night-vision applications using the low-light-level visible and infrared imaging, a fusion framework for night-vision context enhancement（FNCE） method is proposed. An adaptive brightness stretching method is first proposed for enhancing the visible image. Then, a hybrid multi-scale decomposition with edge-preserving filtering is proposed to decompose the source images. Finally, the fused result is obtained via a combination of the decomposed images in three different rules. Experimental results demonstrate that the FNCE method has better performance on the details（edges）, the contrast, the sharpness, and the human visual perception. Therefore,better results for the night-vision context enhancement can be achieved.

Research on Face Anti-Spoofing Algorithm Based on Image Fusion

Along with the rapid development of biometric authentication technology,face recognition has been commercially used in many industries in recent years.However,it cannot be ignored that face recognition-based authentication techniques can be easily spoofed using various types of attacks such photographs,videos or forged 3D masks.In order to solve this problem,this work proposed a face anti-fraud algorithm based on the fusion of thermal infrared images and visible light images.The normal temperature distribution of the human face is stable and characteristic,and the important physiological information of the human body can be observed by the infrared thermal images.Therefore,based on the thermal infrared image,the pixel value of the pulse sensitive area of the human face is collected,and the human heart rate signal is detected to distinguish between real faces and spoofing faces.In order to better obtain the texture features of the face,an image fusion algorithm based on DTCWT and the improved Roberts algorithm is proposed.Firstly,DTCWT is used to decompose the thermal infrared image and visible light image of the face to obtain high-and low-frequency subbands.Then,the method based on region energy and the improved Roberts algorithm are then used to fuse the coefficients of the high-and low-frequency subbands.Finally,the DTCWT inverse transform is used to obtain the fused image containing the facial texture features.Face recognition is carried out on the fused image to realize identity authentication.Experimental results show that this algorithm can effectively resist attacks from photos,videos or masks.Compared with the use of visible light images alone for face recognition,this algorithm has higher recognition accuracy and better robustness.

Fuzzy Based Hybrid Focus Value Estimation for Multi Focus Image Fusion

Due to limited depth-of-field of digital single-lens reflex cameras,the scene content within a limited distance from the imaging plane remains in focus while other objects closer to or further away from the point of focus appear as blurred(out-of-focus)in the image.Multi-Focus Image Fusion can be used to reconstruct a fully focused image from two or more partially focused images of the same scene.In this paper,a new Fuzzy Based Hybrid Focus Measure(FBHFM)for multi-focus image fusion has been proposed.Optimal block size is very critical step for multi-focus image fusion.Particle Swarm Optimization(PSO)algorithm has been used to find optimal size of the block of the images for extraction of focus measure features.After finding optimal blocks,three focus measures Sum of Modified Laplacian,Gray Level Variance and Contrast Visibility has been extracted and combined these focus measures by using intelligent fuzzy technique.Fuzzy based hybrid intelligent focus values were estimated using contrast visibility measure to generate focused image.Different sets of multi-focus images have been used in detailed experimentation and compared the results with state-of-the-art existing techniques such as Genetic Algorithm(GA),Principal Component Analysis(PCA),Laplacian Pyramid discrete wavelet transform(DWT),and aDWT for image fusion.It has been found that proposed method performs well as compare to existing methods.