Infrared and Visible Image Fusion Based on Res2Net-Transformer Automatic Encoding and Decoding

A novel image fusion network framework with an autonomous encoder and decoder is suggested to increase thevisual impression of fused images by improving the quality of infrared and visible light picture fusion. The networkcomprises an encoder module, fusion layer, decoder module, and edge improvementmodule. The encoder moduleutilizes an enhanced Inception module for shallow feature extraction, then combines Res2Net and Transformerto achieve deep-level co-extraction of local and global features from the original picture. An edge enhancementmodule (EEM) is created to extract significant edge features. A modal maximum difference fusion strategy isintroduced to enhance the adaptive representation of information in various regions of the source image, therebyenhancing the contrast of the fused image. The encoder and the EEM module extract features, which are thencombined in the fusion layer to create a fused picture using the decoder. Three datasets were chosen to test thealgorithmproposed in this paper. The results of the experiments demonstrate that the network effectively preservesbackground and detail information in both infrared and visible images, yielding superior outcomes in subjectiveand objective evaluations.

CAEFusion: A New Convolutional Autoencoder-Based Infrared and Visible Light Image Fusion Algorithm

To address the issues of incomplete information,blurred details,loss of details,and insufficient contrast in infrared and visible image fusion,an image fusion algorithm based on a convolutional autoencoder is proposed.The region attention module is meant to extract the background feature map based on the distinct properties of the background feature map and the detail feature map.A multi-scale convolution attention module is suggested to enhance the communication of feature information.At the same time,the feature transformation module is introduced to learn more robust feature representations,aiming to preserve the integrity of image information.This study uses three available datasets from TNO,FLIR,and NIR to perform thorough quantitative and qualitative trials with five additional algorithms.The methods are assessed based on four indicators:information entropy(EN),standard deviation(SD),spatial frequency(SF),and average gradient(AG).Object detection experiments were done on the M3FD dataset to further verify the algorithm’s performance in comparison with five other algorithms.The algorithm’s accuracy was evaluated using the mean average precision at a threshold of 0.5(mAP@0.5)index.Comprehensive experimental findings show that CAEFusion performs well in subjective visual and objective evaluation criteria and has promising potential in downstream object detection tasks.

Intelligent Fusion of Infrared and Visible Image Data Based on Convolutional Sparse Representation and Improved Pulse-Coupled Neural Network

Multi-source information can be obtained through the fusion of infrared images and visible light images,which have the characteristics of complementary information.However,the existing acquisition methods of fusion images have disadvantages such as blurred edges,low contrast,and loss of details.Based on convolution sparse representation and improved pulse-coupled neural network this paper proposes an image fusion algorithm that decompose the source images into high-frequency and low-frequency subbands by non-subsampled Shearlet Transform(NSST).Furthermore,the low-frequency subbands were fused by convolutional sparse representation(CSR),and the high-frequency subbands were fused by an improved pulse coupled neural network(IPCNN)algorithm,which can effectively solve the problem of difficulty in setting parameters of the traditional PCNN algorithm,improving the performance of sparse representation with details injection.The result reveals that the proposed method in this paper has more advantages than the existing mainstream fusion algorithms in terms of visual effects and objective indicators.

Fusion of Infrared and Visible Images Using Fuzzy Based Siamese Convolutional Network

Traditional techniques based on image fusion are arduous in integrating complementary or heterogeneous infrared(IR)/visible(VS)images.Dissimilarities in various kind of features in these images are vital to preserve in the single fused image.Hence,simultaneous preservation of both the aspects at the same time is a challenging task.However,most of the existing methods utilize the manual extraction of features;and manual complicated designing of fusion rules resulted in a blurry artifact in the fused image.Therefore,this study has proposed a hybrid algorithm for the integration of multi-features among two heterogeneous images.Firstly,fuzzification of two IR/VS images has been done by feeding it to the fuzzy sets to remove the uncertainty present in the background and object of interest of the image.Secondly,images have been learned by two parallel branches of the siamese convolutional neural network(CNN)to extract prominent features from the images as well as high-frequency information to produce focus maps containing source image information.Finally,the obtained focused maps which contained the detailed integrated information are directly mapped with the source image via pixelwise strategy to result in fused image.Different parameters have been used to evaluate the performance of the proposed image fusion by achieving 1.008 for mutual information(MI),0.841 for entropy(EG),0.655 for edge information(EI),0.652 for human perception(HP),and 0.980 for image structural similarity(ISS).Experimental results have shown that the proposed technique has attained the best qualitative and quantitative results using 78 publically available images in comparison to the existing discrete cosine transform(DCT),anisotropic diffusion&karhunen-loeve(ADKL),guided filter(GF),random walk(RW),principal component analysis(PCA),and convolutional neural network(CNN)methods.

An infrared and visible image fusion method based upon multi-scale and top-hat transforms

The high-frequency components in the traditional multi-scale transform method are approximately sparse, which can represent different information of the details. But in the low-frequency component, the coefficients around the zero value are very few, so we cannot sparsely represent low-frequency image information. The low-frequency component contains the main energy of the image and depicts the profile of the image. Direct fusion of the low-frequency component will not be conducive to obtain highly accurate fusion result. Therefore, this paper presents an infrared and visible image fusion method combining the multi-scale and top-hat transforms. On one hand, the new top-hat-transform can effectively extract the salient features of the low-frequency component. On the other hand, the multi-scale transform can extract highfrequency detailed information in multiple scales and from diverse directions. The combination of the two methods is conducive to the acquisition of more characteristics and more accurate fusion results. Among them, for the low-frequency component, a new type of top-hat transform is used to extract low-frequency features, and then different fusion rules are applied to fuse the low-frequency features and low-frequency background; for high-frequency components, the product of characteristics method is used to integrate the detailed information in high-frequency. Experimental results show that the proposed algorithm can obtain more detailed information and clearer infrared target fusion results than the traditional multiscale transform methods. Compared with the state-of-the-art fusion methods based on sparse representation, the proposed algorithm is simple and efficacious, and the time consumption is significantly reduced.

Sub-Regional Infrared-Visible Image Fusion Using Multi-Scale Transformation

Infrared-visible image fusion plays an important role in multi-source data fusion,which has the advantage of integrating useful information from multi-source sensors.However,there are still challenges in target enhancement and visual improvement.To deal with these problems,a sub-regional infrared-visible image fusion method(SRF)is proposed.First,morphology and threshold segmentation is applied to extract targets interested in infrared images.Second,the infrared back-ground is reconstructed based on extracted targets and the visible image.Finally,target and back-ground regions are fused using a multi-scale transform.Experimental results are obtained using public data for comparison and evaluation,which demonstrate that the proposed SRF has poten-tial benefits over other methods.

Design and implementation of GM- APD array readout circuit for infrared imaging

Based on an avalanche photodiode（ APD） detecting array working in Geiger mode（ GM-APD）, a high-performance infrared sensor readout integrated circuit（ ROIC） used for infrared 3D（ three-dimensional） imaging is proposed. The system mainly consists of three functional modules, including active quenching circuit（ AQC）, time-to-digital converter（ TDC） circuit and other timing controller circuit. Each AQC and TDC circuit together constitutes the pixel circuit. Under the cooperation with other modules, the current signal generated by the GM-APD sensor is detected by the AQC, and the photon time-of-flight（ TOF） is measured and converted to a digital signal output to achieve a better noise suppression and a higher detection sensitivity by the TDC. The ROIC circuit is fabricated by the CSMC 0. 5 μm standard CMOS technology. The array size is 8 × 8, and the center distance of two adjacent cells is 100μm. The measurement results of the chip showthat the performance of the circuit is good, and the chip can achieve 1 ns time resolution with a 250 MHz reference clock, and the circuit can be used in the array structure of the infrared detection system or focal plane array（ FPA）.

Multiscale feature learning and attention mechanism for infrared and visible image fusion

Current fusion methods for infrared and visible images tend to extract features at a single scale,which results in insufficient detail and incomplete feature preservation.To address these issues,we propose an infrared and visible image fusion network based on a multiscale feature learning and attention mechanism(MsAFusion).A multiscale dilation convolution framework is employed to capture image features across various scales and broaden the perceptual scope.Furthermore,an attention network is introduced to enhance the focus on salient targets in infrared images and detailed textures in visible images.To compensate for information loss during convolution,jump connections are utilized during the image reconstruction phase.The fusion process utilizes a combined loss function consisting of pixel loss and gradient loss for unsupervised fusion of infrared and visible images.Extensive experiments on the dataset of electricity facilities demonstrate that our proposed method outperforms nine state-of-theart methods in terms of visual perception and four objective evaluation metrics.

Pseudo Color Fusion of Infrared and Visible Images Based on the Rattlesnake Vision Imaging System

Image fusion is a key technology in the field of digital image processing.In the present study,an effect-based pseudo color fusion model of infrared and visible images based on the rattlesnake vision imaging system(the rattlesnake bimodal cell fusion mechanism and the visual receptive field model)is proposed.The innovation point of the proposed model lies in the following three features:first,the introduction of a simple mathematical model of the visual receptive field reduce computational complexity;second,the enhanced image is obtained by extracting the common information and unique information of source images,which improves fusion image quality;and third,the Waxman typical fusion structure is improved for the pseudo color image fusion model.The performance of the image fusion model is verified through comparative experiments.In the subjective visual evaluation,we find that the color of the fusion image obtained through the proposed model is natural and can highlight the target and scene details.In the objective quantitative evaluation,we observe that the best values on the four indicators,namely standard deviation,average gradient,entropy,and spatial frequency,accounts for 90%,100%,90%,and 100%,respectively,indicating that the fusion image exhibits superior contrast,image clarity,information content,and overall activity.Experimental results reveal that the performance of the proposed model is superior to that of other models and thus verified the validity and reliability of the model.

Multi-sensors Image Fusion via NSCT and GoogLeNet

In order to improve the detail preservation and target information integrity of different sensor fusion images,an image fusion method of different sensors based on non-subsampling contourlet transform(NSCT)and GoogLeNet neural network model is proposed. First,the different sensors images,i. e.,infrared and visible images,are transformed by NSCT to obtain a low frequency sub-band and a series of high frequency sub-bands respectively.Then,the high frequency sub-bands are fused with the max regional energy selection strategy,the low frequency subbands are input into GoogLeNet neural network model to extract feature maps,and the fusion weight matrices are adaptively calculated from the feature maps. Next,the fused low frequency sub-band is obtained with weighted summation. Finally,the fused image is obtained by inverse NSCT. The experimental results demonstrate that the proposed method improves the image visual effect and achieves better performance in both edge retention and mutual information.

A Thin Cloud Removal Method from Remote Sensing Image for Water Body Identification

In this paper,a thin cloud removal method was put forward based on the linear relationships between the thin cloud reflectance in the channels from 0.4 μm to 1.0 μm and 1.38 μm.Channels of 0.66 μm,0.86 μm and 1.38 μm were chosen to extract the water body information under the thin cloud.Two study cases were selected to validate the thin cloud removal method.One case was applied with the Earth Observation System Moderate Resolution Imaging Spectroradiometer(EOS/MODIS) data,and the other with the Medium Resolution Spectral Imager(MERSI) and Visible and Infrared Radiometer(VIRR) data from Fengyun-3A(FY-3A).The test results showed that thin cloud removal method did not change the reflectivity of the ground surface under the clear sky.To the area contaminated by the thin cloud,the reflectance decreased to be closer to the reference reflectance under the clear sky after the thin cloud removal.The spatial distribution of the water body area could not be extracted before the thin cloud removal,while water information could be easily identified by using proper near infrared channel threshold after removing the thin cloud.The thin cloud removal method could improve the image quality and water body extraction precision effectively.

Fusion of the low-light-level visible and infrared images for night-vision context enhancement

For better night-vision applications using the low-light-level visible and infrared imaging, a fusion framework for night-vision context enhancement（FNCE） method is proposed. An adaptive brightness stretching method is first proposed for enhancing the visible image. Then, a hybrid multi-scale decomposition with edge-preserving filtering is proposed to decompose the source images. Finally, the fused result is obtained via a combination of the decomposed images in three different rules. Experimental results demonstrate that the FNCE method has better performance on the details（edges）, the contrast, the sharpness, and the human visual perception. Therefore,better results for the night-vision context enhancement can be achieved.

Fabrication of multi-wavelength visible and infrared filter for solar atmosphere tomographic imaging

To simultaneously obtain high-resolution multi-wavelength （from visible to near infrared） tomographic images of the solar atmosphere, a high-performance multi-wavelength optical filter has to be used in solar imaging telescopes. In this Letter, the fabrication of the multi-wavelength filter for solar tomographic imaging is described in detail. For this filter, Ta2O5 and SiO2 are used as high- and low-index materials, respectively, and the multilayer structure is optimized by commercial Optilayer software at a 7.5° angle of incidence. Experimentally, this multi-wavelength optical filter is prepared by a plasma ion-assisted deposition technique with optimized deposition parameters. High transmittance at 393.3, 396.8, 430.5, 525, 532.4, 656.8, 705.8, 854.2, 1083, and 1565.3 nm, as well as high reflectance at 500 and 589 nm are achieved. Excellent environmental durability, demonstrated via temperature and humidity tests, is also established.

Integrative Multi-Spectral Sensor Device for Far-Infrared and Visible Light Fusion

Infrared and visible light image fusion technology is a hot spot in the research of multi-sensor fusion technology in recent years. Existing infrared and visible light fusion technologies need to register before fusion because of using two cameras. However, the application effect of the registration technology has yet to be improved. Hence, a novel integrative multi-spectral sensor device is proposed for infrared and visible light fusion, and by using the beam splitter prism, the coaxial light incident from the same lens is projected to the infrared charge coupled device （CCD） and visible light CCD, respectively. In this paper, the imaging mechanism of the proposed sensor device is studied with the process of the signals acquisition and fusion. The simulation experiment, which involves the entire process of the optic system, signal acquisition, and signal fusion, is constructed based on imaging effect model. Additionally, the quality evaluation index is adopted to analyze the simulation result. The experimental results demonstrate that the proposed sensor device is effective and feasible.

Fusion of visible and thermal images for facial expression recognition

Most present research into facial expression recognition focuses on the visible spectrum, which is sen- sitive to illumination change. In this paper, we focus on in- tegrating thermal infrared data with visible spectrum images for spontaneous facial expression recognition. First, the ac- tive appearance model AAM parameters and three defined head motion features are extracted from visible spectrum im- ages, and several thermal statistical features are extracted from infrared （IR） images. Second, feature selection is per- formed using the F-test statistic. Third, Bayesian networks BNs and support vector machines SVMs are proposed for both decision-level and feature-level fusion. Experiments on the natural visible and infrared facial expression （NVIE） spontaneous database show the effectiveness of the proposed methods, and demonstrate thermal 1R images＇ supplementary role for visible facial expression recognition.

面向双模态夜视图像的混合尺度融合算法

针对传统红外与可见光图像融合算法存在的细节模糊、对比度降低、背景信息缺失等不足,提出了一种基于混合尺度的红外与可见光融合方法。通过潜在低秩表示变换将源图像分解低秩子带和显著子带;利用非下采样轮廓波变换将低秩子带继续分解为低频分量与高频分量;针对显著子带采用基于卷积稀疏表示的方法进行融合;并结合全局均值、区域均值与能量的优势融合低频分量;利用权重决策图融合高频分量。基于自建库及公开库的实验结果表明,与其他5种图像融合算法相比,所提算法在充分继承源图像有效信息的同时,融合图像整体对比度更均衡,有效提升了融合图像的清晰度,包含更丰富的图像细节信息,在主客观评价上均取得了更好的效果。

基于目标增强与鼠群优化的红外与可见光图像融合算法

针对传统红外与可见光图像融合结果存在目标模糊、信息丢失问题,提出一种基于目标增强与鼠群优化的红外与可见光图像融合方法,记为TERSFuse。为了减少融合结果中原始图像细节信息丢失,分别构建了红外对比度增强模块和基于亮度感知的可见光图像增强模块;利用拉普拉斯金字塔变换对红外和可见光增强图像进行多尺度分解,从而得到对应的高、低频图像;为了使融合结果充分保留原始图像信息,分别采用“最大绝对值”规则对红外和可见光高频图像进行融合以及通过计算权重系数对低频图像进行融合;设计了基于鼠群优化的图像重构模块以实现高频图像和低频图像重构权重的自适应分配,进而提高融合图像的视觉效果。为了验证所提算法优势,与7种经典融合算法进行比较,实验结果表明所提算法不仅具有良好的视觉效果,而且融合图像能够保留原始图像丰富的边缘纹理和对比度信息。

基于自编码器的红外与可见光图像融合算法

针对目前红外与可见光图像融合过程中,图像特征提取不充分、中间层信息丢失以及融合图像细节不够清晰的问题,提出了一种基于自编码器的端到端图像融合网络结构。该网络由编码器、融合网络和解码器3部分组成。将高效通道注意力机制和混合注意力机制引入到编码器和融合网络中,利用卷积残差网络(convolutional residual network,CRN)基本块来提取并融合红外图像和可见光图像的基本特征,然后将融合后的特征图输入到解码器进行解码,重建出融合图像。选取目前具有典型代表性的5种方法在主客观方面进行对比。在客观方面,较第2名平均梯度、空间频率和视觉保真度分别提升了21%、10.2%、7.2%。在主观方面,融合后的图像目标清晰、细节突出、轮廓明显,符合人类视觉感受。

利用Transformer的多模态目标跟踪算法

目前目标跟踪方法大多通过融合不同模态信息进行定位决策,存在信息提取不充分、融合方法简单、弱光场景无法准确跟踪目标的问题。为此,提出一种基于Transformer的多模态目标跟踪算法(Trans-RGBT):利用伪孪生网络对可见光图像和红外图像分别进行特征提取,并在特征层面充分融合;将首帧目标信息调制到待跟踪帧的特征向量中,得到一个专用跟踪器;应用Transformer的方法对视野中的目标进行编解码,通过空间位置预测分支预测目标在视野中的空间位置,并结合历史信息滤除干扰目标,得到目标的准确位置;使用矩形框回归网络预测目标的外接矩形框,从而实现目标准确跟踪。在最新的大规模数据集VTUAV、RGBT234上进行了实验,与孪生网络(Siambased)、滤波(filter-based)算法相比,Trans-RGBT精度更高、鲁棒性更好、速度接近实时,达22 FPS。

基于特征相似性的红外与可见光图像融合方法

单一图像无法全面描述目标的信息,实际应用价值低,针对当前红外与可见光图像融合方法存在的一些不足,如:融合质量差等,为了获得更加理想的红外与可见光图像融合效果,提出了基于特征相似性的红外与可见光图像融合方法。首先分析当前红外与可见光图像融合的研究进展,指出各种方法的局限性,然后采用红外图像和可见光图像,并对它们进行图像去噪、增强处理,采用卷积神经网络提取红外与可见光图像的特征,最后根据特征相似性进行红外与可见光图像融合,并对红外与可见光图像融合效果进行了测试,结果表明,本方法提升了红外与可见光图像融合质量,融合效果要明显优于其他红外与可见光图像融合方法。