Infrared and Visible Image Fusion Based on Res2Net-Transformer Automatic Encoding and Decoding

A novel image fusion network framework with an autonomous encoder and decoder is suggested to increase thevisual impression of fused images by improving the quality of infrared and visible light picture fusion. The networkcomprises an encoder module, fusion layer, decoder module, and edge improvementmodule. The encoder moduleutilizes an enhanced Inception module for shallow feature extraction, then combines Res2Net and Transformerto achieve deep-level co-extraction of local and global features from the original picture. An edge enhancementmodule (EEM) is created to extract significant edge features. A modal maximum difference fusion strategy isintroduced to enhance the adaptive representation of information in various regions of the source image, therebyenhancing the contrast of the fused image. The encoder and the EEM module extract features, which are thencombined in the fusion layer to create a fused picture using the decoder. Three datasets were chosen to test thealgorithmproposed in this paper. The results of the experiments demonstrate that the network effectively preservesbackground and detail information in both infrared and visible images, yielding superior outcomes in subjectiveand objective evaluations.

CAEFusion: A New Convolutional Autoencoder-Based Infrared and Visible Light Image Fusion Algorithm

To address the issues of incomplete information,blurred details,loss of details,and insufficient contrast in infrared and visible image fusion,an image fusion algorithm based on a convolutional autoencoder is proposed.The region attention module is meant to extract the background feature map based on the distinct properties of the background feature map and the detail feature map.A multi-scale convolution attention module is suggested to enhance the communication of feature information.At the same time,the feature transformation module is introduced to learn more robust feature representations,aiming to preserve the integrity of image information.This study uses three available datasets from TNO,FLIR,and NIR to perform thorough quantitative and qualitative trials with five additional algorithms.The methods are assessed based on four indicators:information entropy(EN),standard deviation(SD),spatial frequency(SF),and average gradient(AG).Object detection experiments were done on the M3FD dataset to further verify the algorithm’s performance in comparison with five other algorithms.The algorithm’s accuracy was evaluated using the mean average precision at a threshold of 0.5(mAP@0.5)index.Comprehensive experimental findings show that CAEFusion performs well in subjective visual and objective evaluation criteria and has promising potential in downstream object detection tasks.

Intelligent Fusion of Infrared and Visible Image Data Based on Convolutional Sparse Representation and Improved Pulse-Coupled Neural Network

Multi-source information can be obtained through the fusion of infrared images and visible light images,which have the characteristics of complementary information.However,the existing acquisition methods of fusion images have disadvantages such as blurred edges,low contrast,and loss of details.Based on convolution sparse representation and improved pulse-coupled neural network this paper proposes an image fusion algorithm that decompose the source images into high-frequency and low-frequency subbands by non-subsampled Shearlet Transform(NSST).Furthermore,the low-frequency subbands were fused by convolutional sparse representation(CSR),and the high-frequency subbands were fused by an improved pulse coupled neural network(IPCNN)algorithm,which can effectively solve the problem of difficulty in setting parameters of the traditional PCNN algorithm,improving the performance of sparse representation with details injection.The result reveals that the proposed method in this paper has more advantages than the existing mainstream fusion algorithms in terms of visual effects and objective indicators.

Fusion of Infrared and Visible Images Using Fuzzy Based Siamese Convolutional Network

Traditional techniques based on image fusion are arduous in integrating complementary or heterogeneous infrared(IR)/visible(VS)images.Dissimilarities in various kind of features in these images are vital to preserve in the single fused image.Hence,simultaneous preservation of both the aspects at the same time is a challenging task.However,most of the existing methods utilize the manual extraction of features;and manual complicated designing of fusion rules resulted in a blurry artifact in the fused image.Therefore,this study has proposed a hybrid algorithm for the integration of multi-features among two heterogeneous images.Firstly,fuzzification of two IR/VS images has been done by feeding it to the fuzzy sets to remove the uncertainty present in the background and object of interest of the image.Secondly,images have been learned by two parallel branches of the siamese convolutional neural network(CNN)to extract prominent features from the images as well as high-frequency information to produce focus maps containing source image information.Finally,the obtained focused maps which contained the detailed integrated information are directly mapped with the source image via pixelwise strategy to result in fused image.Different parameters have been used to evaluate the performance of the proposed image fusion by achieving 1.008 for mutual information(MI),0.841 for entropy(EG),0.655 for edge information(EI),0.652 for human perception(HP),and 0.980 for image structural similarity(ISS).Experimental results have shown that the proposed technique has attained the best qualitative and quantitative results using 78 publically available images in comparison to the existing discrete cosine transform(DCT),anisotropic diffusion&karhunen-loeve(ADKL),guided filter(GF),random walk(RW),principal component analysis(PCA),and convolutional neural network(CNN)methods.

An infrared and visible image fusion method based upon multi-scale and top-hat transforms

The high-frequency components in the traditional multi-scale transform method are approximately sparse, which can represent different information of the details. But in the low-frequency component, the coefficients around the zero value are very few, so we cannot sparsely represent low-frequency image information. The low-frequency component contains the main energy of the image and depicts the profile of the image. Direct fusion of the low-frequency component will not be conducive to obtain highly accurate fusion result. Therefore, this paper presents an infrared and visible image fusion method combining the multi-scale and top-hat transforms. On one hand, the new top-hat-transform can effectively extract the salient features of the low-frequency component. On the other hand, the multi-scale transform can extract highfrequency detailed information in multiple scales and from diverse directions. The combination of the two methods is conducive to the acquisition of more characteristics and more accurate fusion results. Among them, for the low-frequency component, a new type of top-hat transform is used to extract low-frequency features, and then different fusion rules are applied to fuse the low-frequency features and low-frequency background; for high-frequency components, the product of characteristics method is used to integrate the detailed information in high-frequency. Experimental results show that the proposed algorithm can obtain more detailed information and clearer infrared target fusion results than the traditional multiscale transform methods. Compared with the state-of-the-art fusion methods based on sparse representation, the proposed algorithm is simple and efficacious, and the time consumption is significantly reduced.

Sub-Regional Infrared-Visible Image Fusion Using Multi-Scale Transformation

Infrared-visible image fusion plays an important role in multi-source data fusion,which has the advantage of integrating useful information from multi-source sensors.However,there are still challenges in target enhancement and visual improvement.To deal with these problems,a sub-regional infrared-visible image fusion method(SRF)is proposed.First,morphology and threshold segmentation is applied to extract targets interested in infrared images.Second,the infrared back-ground is reconstructed based on extracted targets and the visible image.Finally,target and back-ground regions are fused using a multi-scale transform.Experimental results are obtained using public data for comparison and evaluation,which demonstrate that the proposed SRF has poten-tial benefits over other methods.

NormFuse: Infrared and Visible Image Fusion With Pixel-Adaptive Normalization

Dear Editor.This letter presents a normalization mechanism to effectively fuse infrared and visible images in an encoder-decoder network.Source images are decomposed into source-invariant structure and sourcespecific detail features.Then,the information of detail features is sufficiently incorporated into the structure features using this normalization mechanism in the decoder,which generates high-contrast fused images with highlighted targets and abundant texture information.Qualitative and quantitative experiments on two challenging datasets demonstrate the superiority of our method over current stateof-the-art methods.

Advancing automated pupillometry:a practical deep learning model utilizing infrared pupil images

AIM:To establish pupil diameter measurement algorithms based on infrared images that can be used in real-world clinical settings.METHODS:A total of 188 patients from outpatient clinic at He Eye Specialist Shenyang Hospital from Spetember to December 2022 were included,and 13470 infrared pupil images were collected for the study.All infrared images for pupil segmentation were labeled using the Labelme software.The computation of pupil diameter is divided into four steps:image pre-processing,pupil identification and localization,pupil segmentation,and diameter calculation.Two major models are used in the computation process:the modified YoloV3 and Deeplabv 3+models,which must be trained beforehand.RESULTS:The test dataset included 1348 infrared pupil images.On the test dataset,the modified YoloV3 model had a detection rate of 99.98% and an average precision(AP)of 0.80 for pupils.The DeeplabV3+model achieved a background intersection over union(IOU)of 99.23%,a pupil IOU of 93.81%,and a mean IOU of 96.52%.The pupil diameters in the test dataset ranged from 20 to 56 pixels,with a mean of 36.06±6.85 pixels.The absolute error in pupil diameters between predicted and actual values ranged from 0 to 7 pixels,with a mean absolute error(MAE)of 1.06±0.96 pixels.CONCLUSION:This study successfully demonstrates a robust infrared image-based pupil diameter measurement algorithm,proven to be highly accurate and reliable for clinical application.

Meibomian glands segmentation in infrared images with limited annotation

●AIM:To investigate a pioneering framework for the segmentation of meibomian glands(MGs),using limited annotations to reduce the workload on ophthalmologists and enhance the efficiency of clinical diagnosis.●METHODS:Totally 203 infrared meibomian images from 138 patients with dry eye disease,accompanied by corresponding annotations,were gathered for the study.A rectified scribble-supervised gland segmentation(RSSGS)model,incorporating temporal ensemble prediction,uncertainty estimation,and a transformation equivariance constraint,was introduced to address constraints imposed by limited supervision information inherent in scribble annotations.The viability and efficacy of the proposed model were assessed based on accuracy,intersection over union(IoU),and dice coefficient.●RESULTS:Using manual labels as the gold standard,RSSGS demonstrated outcomes with an accuracy of 93.54%,a dice coefficient of 78.02%,and an IoU of 64.18%.Notably,these performance metrics exceed the current weakly supervised state-of-the-art methods by 0.76%,2.06%,and 2.69%,respectively.Furthermore,despite achieving a substantial 80%reduction in annotation costs,it only lags behind fully annotated methods by 0.72%,1.51%,and 2.04%.●CONCLUSION:An innovative automatic segmentation model is developed for MGs in infrared eyelid images,using scribble annotation for training.This model maintains an exceptionally high level of segmentation accuracy while substantially reducing training costs.It holds substantial utility for calculating clinical parameters,thereby greatly enhancing the diagnostic efficiency of ophthalmologists in evaluating meibomian gland dysfunction.

Novel imaging system for positioning of the indocyanine green (ICG) target;visible projection of the near-infrared fluorescence image

Background: Even though NIR fluorescence imaging has many advantages in SLN mapping and cancer detection, NIR fluorescence imaging shows a serious drawback that NIR cannot be detected by the naked eye without any detectors. This limitation further disturbs accurate SLN detection and adequate tumor resection resulting in the presence of cancerous cells near the boundaries of surgically removed tissues. Materials and methods: To overcome the drawback of the conventional NIR imaging method, we suggest a novel NIR imaging system which can make the NIR fluorescence image visible to the naked eye as NIR fluorescence image detected by a video camera is processed by a computer and then projected back onto the NIR fluorescence excitation position with a projector using conspicuous color light. Image processing techniques were used for projection onto the exact position of the NIR fluorescence image. Also, we implemented a phantom experiment to evaluate the performance of the developed NIR fluorescence projection system by use of the ICG. Results: The developed NIR fluorescence projection system was applied in normal mouse model to confirm the usefulness of the system in the clinical field. A BALB/c nude mouse was prepared to be applied in normal mouse model and 0.25 mg/ml stock solution of the ICG was injected through a tail vein of the mouse. From the application in normal mouse model, we could confirm that the injected ICG stayed in the liver of the mouse and verify that the projection system projected the ICG fluorescence image at the exact location of the ICG by performing laparotomy of the mouse. Conclusions: From the application in normal mouse model, we could verify that the ICG fluorescence image was precisely projected back on the site where ICG fluorescence generated. It can be demonstrated that the NIR fluorescence projection system can make it possible to visualize the invisible NIR fluorescence image and to realize that SLN mapping and cancer detection in clinical surgery.

Automated Registration for Infrared Image Based on Wavelet Analysis

To develop a quick, accurate and antinoise automated image registration technique for infrared images, the wavelet analysis technique was used to extract the feature points in two images followed by the compensation for input image with angle difference between them. A hi erarchical feature matching algorithm was adopted to get the final transform parameters between the two images. The simulation results for two infrared images show that the method can effectively, quickly and accurately register images and be antinoise to some extent.

Infrared polarization image fusion based on combination of NSST and improved PCA

In view of the problem that current mainstream fusion method of infrared polarization image—Multiscale Geometry Analysis method only focuses on a certain characteristic to image representation.And spatial domain fusion method,Principal Component Analysis(PCA)method has the shortcoming of losing small target,this paper presents a new fusion method of infrared polarization images based on combination of Nonsubsampled Shearlet Transformation(NSST)and improved PCA.This method can make full use of the effectiveness to image details expressed by NSST and the characteristics that PCA can highlight the main features of images.The combination of the two methods can integrate the complementary features of themselves to retain features of targets and image details fully.Firstly,intensity and polarization images are decomposed into low frequency and high frequency components with different directions by NSST.Secondly,the low frequency components are fused with improved PCA,while the high frequency components are fused by joint decision making rule with local energy and local variance.Finally,the fused image is reconstructed with the inverse NSST to obtain the final fused image of infrared polarization.The experiment results show that the method proposed has higher advantages than other methods in terms of detail preservation and visual effect.

High Performance of Imaging Extraction for Infrared Satellite Cloud Image

The isotherm is an important feature of infrared satellite cloud images (ISCI), which can directly reveal substantial information of cloud systems. The isotherm extraction of ISCI can remove the redundant information and therefore helps to compress the information of ISCI. In this paper, an isotherm extraction method is presented. The main aggregate of clouds can be segmented based on mathematical morphology. T algorithm and IP algorithm are then applied to extract the isotherms from the main aggregate of clouds. A concrete example for the extraction of isotherm based on IBM SP2 is described. The result shows that this is a high efficient algorithm. It can be used in feature extractions of infrared images for weather forecasts.

Deep learning-assisted common temperature measurement based on visible light imaging

Real-time,contact-free temperature monitoring of low to medium range(30℃-150℃)has been extensively used in industry and agriculture,which is usually realized by costly infrared temperature detection methods.This paper proposes an alternative approach of extracting temperature information in real time from the visible light images of the monitoring target using a convolutional neural network(CNN).A mean-square error of<1.119℃was reached in the temperature measurements of low to medium range using the CNN and the visible light images.Imaging angle and imaging distance do not affect the temperature detection using visible optical images by the CNN.Moreover,the CNN has a certain illuminance generalization ability capable of detection temperature information from the images which were collected under different illuminance and were not used for training.Compared to the conventional machine learning algorithms mentioned in the recent literatures,this real-time,contact-free temperature measurement approach that does not require any further image processing operations facilitates temperature monitoring applications in the industrial and civil fields.

Optical design of a novel near-infrared phase contrast imaging(NI-PCI)diagnostic on the HL-2A tokamak

The optical design of near-infrared phase contrast imaging(NI-PCI)diagnosis on HL-2A is introduced in this paper.This scheme benefits from the great progress of near-infrared laser technology and is a broadening of traditional phase contrast technology.This diagnostic can work as a keen tool to measure plasma wavenumber spectra by inferring string-integrated plasma density fluctuations.Design of both the front optical path which is the path before the laser transmitting into the tokamak plasma and the rear optics which is the path after the laser passing through the plasma is detailed.The 1550 nm laser is chosen as the probe beam and highprecision optical components are designed to fit the laser beam,in which a phase plate with a 194-nm-deep silver groove is the key.Compared with the conventional 10.6μm laser-based PCI system on HL-2A,NI-PCI significantly overcomes the unwanted phase scintillation effect and promotes the measurement capability of high-wavenumber turbulence with an increased maximal measurable wavenumber from 15 cm^(-1)to 32.6 cm^(-1).

Matching Performance among Visible and near Infrared Coating,Low Infrared Emitting Coating and Microwave Absorbing Coating

The matching performance among the visible and near infrared coating.the low infrared emitting coating and the microwave absorbing coating was investigated.Experimental results show that the resulting malerial is characteristic of wideband effect ranging from the visible,near infrared and 3-5μm,8-14μm infrared protion of the spectrum,as well as the radar region from 8 to 18GHz when these three materials form αlayerstructure material system.The microwave absorbing ability of material is hardly changed.The resonance peak moves towards lower frequency as the thickness of the visible,near infrared coating and the low infrared emitting coating increases.This problem can be resolved by controlling the thickness of the matrial.On the other hand, the infrared emissivity εof the material system increases as the thickness of the visible,near infrared coating increases.This can be resolved by increasing infrared transparency of the visible and near infrared topcoating or controlling its thickness.The experimental resulting material system has spectral reflection characteristics in visible and near infrared regions that are similar to those of the natural background.

Infrared image segmentation method based on 2D histogram shape modification and optimal objective function

In the methods of image thresholding segmentation, such methods based on two-dimensional （2D） histogram and optimal objective functions are important. However, when they are used for infrared image segmentation, they are weak in suppressing background noises and worse in segmenting targets with non-uniform gray level. The concept of 2D histogram shape modification is proposed, which is realized by target information prior restraint after enhancing target information using plateau histogram equalization. The formula of 2D minimum Renyi entropy is deduced for image segmentation, then the shape-modified 2D histogram is combined wfth four optimal objective functions （i.e., maximum between-class variance, maximum entropy, maximum correlation and minimum Renyi entropy） respectively for the appli- cation of infrared image segmentation. Simultaneously, F-measure is introduced to evaluate the segmentation effects objectively. The experimental results show that F-measure is an effective evaluation index for image segmentation since its value is fully consistent with the subjective evaluation, and after 2D histogram shape modification, the methods of optimal objective functions can overcome their original forms＇ deficiency and their segmentation effects are more or less improvements, where the best one is the maximum entropy method based on 2D histogram shape modification.

Utilization of Thermal Infrared Image for Inversion of Winter Wheat Yield and Biomass

The present paper utilizes thermal infrared image for inversion of winter wheat yield and biomass with different technology of irrigation(drip irrigation,sprinkler irrigation,flood irrigation).It is the first time that thermal infrared image is used for predicting the winter wheat yield and biomass.The temperature of crop and background was measured by thermal infrared image.It is necessary to get the crop background separation index(CBSIL,CBSIH),which can be used for distinguishing the crop value from the image.CBSIL and CBSIH(the temperature when the leaves are wet adequately;the temperature when the stomata of leaf is closed completely) are the threshold values.The temperature of crop ranged from CBSIL to CBSIH.Then the ICWSI was calculated based on relevant theoretical method.The value of stomata leaf has strong negative correlation with ICWSI proving the reliable value of ICWSI.In order to construct the high accuracy simulation model,the samples were divided into two parts.One was used for constructing the simulation model,the other for checking the accuracy of the model.Such result of the model was concluded as:(1) As for the simulation model of soil moisture,the correlation coefficient(R2) is larger than 0.887 6,the average of relative error(Er) ranges from 13.33% to 16.88%;(2) As for the simulation model of winter wheat yield,drip irrigation(0.887 6,16.89%,-0.12),sprinkler irrigation(0.970 0,14.85%,-0.12),flood irrigation(0.969 0,18.87%,0.18),with the values of R2,Er and CRM listed in the parentheses followed by the individual term.(3) As for winter wheat biomass,drip irrigation(0.980 0,13.70%,0.13),sprinkler irrigation(0.95,13.15%,-0.14),flood irrigation(0.970 0,14.48%,-0.13),and the values in the parentheses are demonstrated the same as above.Both the CRM and Er are shown to be very low values,which points to the accuracy and reliability of the model investigated.The accuracy of model is high and reliable.The results indicated that thermal infrared image can be used potentially for inversion of winter wheat yield and biomass.

Image Processing Technology of Flaws within Infrared Transmitting Glasses

We investigated image processing algorithms of the original infrared glass flaw image. Using the Laplacian edge enhancement following LSD （Line Segment Detector） algorithm, we can get a good flaw image very consistent with the original one. This study is very helpful to further enhance the infrared glass flaw inspection technique.

Fault diagnosis of electric transformers based on infrared image processing and semi-supervised learning

It is crucial to maintain the safe and stable operation of distribution transformers,which constitute a key part of power systems.In the event of transformer failure,the fault type must be diagnosed in a timely and accurate manner.To this end,a transformer fault diagnosis method based on infrared image processing and semi-supervised learning is proposed herein.First,we perform feature extraction on the collected infrared-image data to extract temperature,texture,and shape features as the model reference vectors.Then,a generative adversarial network(GAN)is constructed to generate synthetic samples for the minority subset of labelled samples.The proposed method can learn information from unlabeled sample data,unlike conventional supervised learning methods.Subsequently,a semi-supervised graph model is trained on the entire dataset,i.e.,both labeled and unlabeled data.Finally,we test the proposed model on an actual dataset collected from a Chinese electricity provider.The experimental results show that the use of feature extraction,sample generation,and semi-supervised learning model can improve the accuracy of transformer fault classification.This verifies the effectiveness of the proposed method.