Towards complex scenes: A deep learning-based camouflaged people detection method for snapshot multispectral images

Camouflaged people are extremely expert in actively concealing themselves by effectively utilizing cover and the surrounding environment. Despite advancements in optical detection capabilities through imaging systems, including spectral, polarization, and infrared technologies, there is still a lack of effective real-time method for accurately detecting small-size and high-efficient camouflaged people in complex real-world scenes. Here, this study proposes a snapshot multispectral image-based camouflaged detection model, multispectral YOLO(MS-YOLO), which utilizes the SPD-Conv and Sim AM modules to effectively represent targets and suppress background interference by exploiting the spatial-spectral target information. Besides, the study constructs the first real-shot multispectral camouflaged people dataset(MSCPD), which encompasses diverse scenes, target scales, and attitudes. To minimize information redundancy, MS-YOLO selects an optimal subset of 12 bands with strong feature representation and minimal inter-band correlation as input. Through experiments on the MSCPD, MS-YOLO achieves a mean Average Precision of 94.31% and real-time detection at 65 frames per second, which confirms the effectiveness and efficiency of our method in detecting camouflaged people in various typical desert and forest scenes. Our approach offers valuable support to improve the perception capabilities of unmanned aerial vehicles in detecting enemy forces and rescuing personnel in battlefield.

One-year results for myopia control of orthokeratology with different back optic zone diameters: a randomized trial using a novel multispectral-based topographer

AIM:To present the 1-year results of a prospective cohort study investigating the efficacy,potential mechanism,and safety of orthokeratology(ortho-k)with different back optic zone diameters(BOZD)for myopia control in children.METHODS:This randomized clinical study was performed between Dec.2020 and Dec.2021.Participants were randomly assigned to three groups wearing ortho-k:5 mm BOZD(5-MM group),5.5 mm BOZD(5.5-MM group),and 6 mm BOZD(6-MM group).The 1-year data were recorded,including axial length,relative peripheral refraction(RPR,measured by multispectral refractive topography,MRT),and visual quality.The contrast sensitivity(CS)was evaluated by CSV-1000 instrument with spatial frequencies of 3,6,12,and 18 cycles/degree(c/d);the corneal higher-order aberrations(HOAs)were measured by iTrace aberration analyzer.The one-way ANOVA was performed to assess the differences between the three groups.The correlation between the change in AL and RPR was calculated by Pearson’s correlation coefficient.RESULTS:The 1-year results of 20,21,and 21 subjects in the 5-MM,5.5-MM,and 6-MM groups,respectively,were presented.There were no statistical differences in baseline age,sex,or ocular parameters between the three groups(all P>0.05).At the 1-year visit,the 5-MM group had lower axial elongation than the 6-MM group(0.07±0.09 vs 0.18±0.11 mm,P=0.001).The 5-MM group had more myopic total RPR(TRPR,P=0.014),with RPR in the 15°–30°(RPR 15–30,P=0.015),30°–45°(RPR 30–45,P=0.011),temporal(RPR-T,P=0.008),and nasal area(RPR-N,P<0.001)than the 6-MM group.RPR 15–30 in the 5.5-MM group was more myopic than that in the 6-MM group(P=0.002),and RPR-N in the 5-MM group was more myopic than that in the 5.5-MM group(P<0.001).There were positive correlations between the axial elongation and the change in TRPR(r=0.756,P<0.001),RPR 15–30(r=0.364,P=0.004),RPR 30–45(r=0.306,P=0.016),and RPR-N(r=0.253,P=0.047).The CS decreased at 3 c/d(P<0.001),and the corneal HOAs increased in the 5-MM group(P=0.030).CONCLUSION:Ortho-k with 5 mm BOZD can control myopia progression more effectively.The mechanism may be associated with greater myopic shifts in RPR.

Lightweight Cross-Modal Multispectral Pedestrian Detection Based on Spatial Reweighted Attention Mechanism

Multispectral pedestrian detection technology leverages infrared images to provide reliable information for visible light images, demonstrating significant advantages in low-light conditions and background occlusion scenarios. However, while continuously improving cross-modal feature extraction and fusion, ensuring the model’s detection speed is also a challenging issue. We have devised a deep learning network model for cross-modal pedestrian detection based on Resnet50, aiming to focus on more reliable features and enhance the model’s detection efficiency. This model employs a spatial attention mechanism to reweight the input visible light and infrared image data, enhancing the model’s focus on different spatial positions and sharing the weighted feature data across different modalities, thereby reducing the interference of multi-modal features. Subsequently, lightweight modules with depthwise separable convolution are incorporated to reduce the model’s parameter count and computational load through channel-wise and point-wise convolutions. The network model algorithm proposed in this paper was experimentally validated on the publicly available KAIST dataset and compared with other existing methods. The experimental results demonstrate that our approach achieves favorable performance in various complex environments, affirming the effectiveness of the multispectral pedestrian detection technology proposed in this paper.

End-to-end computational design for an EUV solar corona multispectral imager with stray light suppression

An extreme ultraviolet solar corona multispectral imager can allow direct observation of high temperature coronal plasma,which is related to solar flares,coronal mass ejections and other significant coronal activities.This manuscript proposes a novel end-to-end computational design method for an extreme ultraviolet(EUV)solar corona multispectral imager operating at wavelengths near 100 nm,including a stray light suppression design and computational image recovery.To suppress the strong stray light from the solar disk,an outer opto-mechanical structure is designed to protect the imaging component of the system.Considering the low reflectivity(less than 70%)and strong-scattering(roughness)of existing extreme ultraviolet optical elements,the imaging component comprises only a primary mirror and a curved grating.A Lyot aperture is used to further suppress any residual stray light.Finally,a deep learning computational imaging method is used to correct the individual multi-wavelength images from the original recorded multi-slit data.In results and data,this can achieve a far-field angular resolution below 7",and spectral resolution below 0.05 nm.The field of view is±3 R☉along the multi-slit moving direction,where R☉represents the radius of the solar disk.The ratio of the corona's stray light intensity to the solar center's irradiation intensity is less than 10-6 at the circle of 1.3 R☉.

Multiple omics datasets reveal significant physical and physiological dormancy in alfalfa hard seeds identified by multispectral imaging analysis

Physical dormancy(PY) commonly present in the seeds of higher plants is believed to be responsible for the germination failure by impermeable seed coat in hard seeds of legume species, instead of physiological dormancy(PD). In this study, a non-destructive approach involving multispectral imaging was used to successfully identify hard seeds from non-hard seeds in Medicago sativa, with accuracy as high as96.8%–99.0%. We further adopted multiple-omics strategies to investigate the differences of physiology,metabolomics, methylomics, and transcriptomics in alfalfa hard seeds, with non-hard seeds as control.The hard seeds showed dramatically increased antioxidants and 125 metabolites of significant differences in non-targeted metabolomics analysis, which are enriched in the biosynthesis pathways of flavonoids, lipids and hormones, especially with significantly higher ABA, a hormone known to induce dormancy. In our transcriptomics results, the enrichment pathway of “response to abscisic acid” of differential expressed genes(DEG) supported the key role of ABA in metabolomics results. The methylome analysis identified 54,899, 46,216 and 54,452 differential methylation regions for contexts of CpG, CHG and CHH, and 344 DEGs might be regulated by hypermethylation and hypomethylation of promoter and exon regions, including four ABA-and JA-responsive genes. Among 8% hard seeds in seed lots,24.5% still did not germinate after scarifying seed coat, and were named as non-PY hard seeds.Compared to hard seeds, significantly higher contents of ABA/IAA and ABA/JA were identified in nonPY hard seeds, which indicated the potential presence of PD. In summary, the significantly changed metabolites, gene expressions, and methylations all suggested involvement of ABA responses in hard seeds, and germination failure of alfalfa hard seeds was caused by combinational dormancy(PY + PD),rather than PY alone.

A double-layer model for improving the estimation of wheat canopy nitrogen content from unmanned aerial vehicle multispectral imagery

The accurate and rapid estimation of canopy nitrogen content(CNC)in crops is the key to optimizing in-season nitrogen fertilizer application in precision agriculture.However,the determination of CNC from field sampling data for leaf area index(LAI),canopy photosynthetic pigments(CPP;including chlorophyll a,chlorophyll b and carotenoids)and leaf nitrogen concentration(LNC)can be time-consuming and costly.Here we evaluated the use of high-precision unmanned aerial vehicle(UAV)multispectral imagery for estimating the LAI,CPP and CNC of winter wheat over the whole growth period.A total of 23 spectral features(SFs;five original spectrum bands,17 vegetation indices and the gray scale of the RGB image)and eight texture features(TFs;contrast,entropy,variance,mean,homogeneity,dissimilarity,second moment,and correlation)were selected as inputs for the models.Six machine learning methods,i.e.,multiple stepwise regression(MSR),support vector regression(SVR),gradient boosting decision tree(GBDT),Gaussian process regression(GPR),back propagation neural network(BPNN)and radial basis function neural network(RBFNN),were compared for the retrieval of winter wheat LAI,CPP and CNC values,and a double-layer model was proposed for estimating CNC based on LAI and CPP.The results showed that the inversion of winter wheat LAI,CPP and CNC by the combination of SFs+TFs greatly improved the estimation accuracy compared with that by using only the SFs.The RBFNN and BPNN models outperformed the other machine learning models in estimating winter wheat LAI,CPP and CNC.The proposed double-layer models(R^(2)=0.67-0.89,RMSE=13.63-23.71 mg g^(-1),MAE=10.75-17.59 mg g^(-1))performed better than the direct inversion models(R^(2)=0.61-0.80,RMSE=18.01-25.12 mg g^(-1),MAE=12.96-18.88 mg g^(-1))in estimating winter wheat CNC.The best winter wheat CNC accuracy was obtained by the double-layer RBFNN model with SFs+TFs as inputs(R^(2)=0.89,RMSE=13.63 mg g^(-1),MAE=10.75 mg g^(-1)).The results of this study can provide guidance for the accurate and rapid determination of winter wheat canopy nitrogen content in the field.

Shallow water bathymetry based on a back propagation neural network and ensemble learning using multispectral satellite imagery

The back propagation(BP)neural network method is widely used in bathymetry based on multispectral satellite imagery.However,the classical BP neural network method faces a potential problem because it easily falls into a local minimum,leading to model training failure.This study confirmed that the local minimum problem of the BP neural network method exists in the bathymetry field and cannot be ignored.Furthermore,to solve the local minimum problem of the BP neural network method,a bathymetry method based on a BP neural network and ensemble learning(BPEL)is proposed.First,the remote sensing imagery and training sample were used as input datasets,and the BP method was used as the base learner to produce multiple water depth inversion results.Then,a new ensemble strategy,namely the minimum outlying degree method,was proposed and used to integrate the water depth inversion results.Finally,an ensemble bathymetric map was acquired.Anda Reef,northeastern Jiuzhang Atoll,and Pingtan coastal zone were selected as test cases to validate the proposed method.Compared with the BP neural network method,the root-mean-square error and the average relative error of the BPEL method can reduce by 0.65–2.84 m and 16%–46%in the three test cases at most.The results showed that the proposed BPEL method could solve the local minimum problem of the BP neural network method and obtain highly robust and accurate bathymetric maps.

Adaptive Window Based 3-D Feature Selection for Multispectral Image Classification Using Firefly Algorithm

Feature extraction is the most critical step in classification of multispectral image.The classification accuracy is mainly influenced by the feature sets that are selected to classify the image.In the past,handcrafted feature sets are used which are not adaptive for different image domains.To overcome this,an evolu-tionary learning method is developed to automatically learn the spatial-spectral features for classification.A modified Firefly Algorithm(FA)which achieves maximum classification accuracy with reduced size of feature set is proposed to gain the interest of feature selection for this purpose.For extracting the most effi-cient features from the data set,we have used 3-D discrete wavelet transform which decompose the multispectral image in all three dimensions.For selecting spatial and spectral features we have studied three different approaches namely overlapping window(OW-3DFS),non-overlapping window(NW-3DFS)adaptive window cube(AW-3DFS)and Pixel based technique.Fivefold Multiclass Support Vector Machine(MSVM)is used for classification purpose.Experiments con-ducted on Madurai LISS IV multispectral image exploited that the adaptive win-dow approach is used to increase the classification accuracy.

Novel application of multispectral refraction topography in the observation of myopic control effect by orthokeratology lens in adolescents

BACKGROUND Myopia,as one of the common ocular diseases,often occurs in adolescence.In addition to the harm from itself,it may also lead to serious complications.Thus,prevention and control of myopia are attracting more and more attention.Previous research revealed that single-focal glasses and orthokeratology lenses(OK lenses)played an important part in slowing down myopia and preventing high myopia.AIM To compare the clinical effects of OK lenses and frame glasses against the increase of diopter in adolescent myopia and further explore the mechanism of the OK lens.METHODS Changes in diopter and axial length were collected among 70 adolescent myopia patients(124 eyes)wearing OK lenses for 1 year(group A)and 59 adolescent myopia patients(113 eyes)wearing frame glasses(group B).Refractive states of their retina were inspected through multispectral refraction topography.The obtained hyperopic defocus was analyzed for the mechanism of OK lenses on slowing down the increase of myopic diopter by delaying the increase of ocular axis length and reducing the near hyperopia defocus.RESULTS Teenagers in groups A and B were divided into low myopia(0 D--3.00 D)and moderate myopia(-3.25 D--6.00 D),without statistical differences among gender and age.After 1-year treatment,the increase of diopter and axis length and changes of retinal hyperopic defocus amount of group A were significantly less than those of group B.According to the multiple linear analysis,the retinal defocus in the upper,lower,nasal,and temporal directions had almost the same effect on the total defocus.The amount of peripheral retinal defocus(15°-53°)in group A was significantly lower than that in group B.CONCLUSION Multispectral refraction topography is progressive and instructive in clinical prevention and control of myopia.

Optical design of common-aperture multispectral and polarization optical imaging system with wide field of view

Multispectral and polarization cameras that can simultaneously acquire the spatial,spectral,and polarization characteristics of an object have considerable potential applications in target detection,biomedical imaging,and remote sensing.In this work,we develop a common-aperture optical system that can capture multispectral and polarization information.An off-axis three-mirror optical system is mounted on the front end of the proposed system and used as a common-aperture telescope in the visible light(400 nm-750 nm)and long-wave infrared(LWIR,8μm-12μm)waveband.The system can maintain a wide field of view(4.5°)and it can demonstrate an enhanced identification ability.The off-axis three-mirror system gets rid of central obscuration while further yielding stable system resolution and energy.Light that has passed through the front-end common-aperture reflection system is divided into the visible light and LWIR waveband by a beamsplitter.The two wavebands then converge on two detectors through two groups of lenses.Our simulation results indicate that the proposed system can obtain clear images in each waveband to meet the diverse imaging requirements.

D-SS Frame:deep spectral-spatial feature extraction and fusion for classification of panchromatic and multispectral images

Facing the very high-resolution( VHR) image classification problem,a feature extraction and fusion framework is presented for VHR panchromatic and multispectral image classification based on deep learning techniques. The proposed approach combines spectral and spatial information based on the fusion of features extracted from panchromatic( PAN) and multispectral( MS) images using sparse autoencoder and its deep version. There are three steps in the proposed method,the first one is to extract spatial information of PAN image,and the second one is to describe spectral information of MS image. Finally,in the third step,the features obtained from PAN and MS images are concatenated directly as a simple fusion feature. The classification is performed using the support vector machine( SVM) and the experiments carried out on two datasets with very high spatial resolution. MS and PAN images from WorldView-2 satellite indicate that the classifier provides an efficient solution and demonstrate that the fusion of the features extracted by deep learning techniques from PAN and MS images performs better than that when these techniques are used separately. In addition,this framework shows that deep learning models can extract and fuse spatial and spectral information greatly,and have huge potential to achieve higher accuracy for classification of multispectral and panchromatic images.

MRF-Based Multispectral Image Fusion Using an Adaptive Approach Based on Edge-Guided Interpolation

In interpretation of remote sensing images, it is possible that some images which are supplied by different sensors become incomprehensible. For better visual perception of these images, it is essential to operate series of pre-processing and elementary corrections and then operate a series of main processing steps for more precise analysis on the images. There are several approaches for processing which are depended on the type of remote sensing images. The discussed approach in this article, i.e. image fusion, is the use of natural colors of an optical image for adding color to a grayscale satellite image which gives us the ability for better observation of the HR image of OLI sensor of Landsat-8. This process with emphasis on details of fusion technique has previously been performed;however, we are going to apply the concept of the interpolation process. In fact, we see many important software tools such as ENVI and ERDAS as the most famous remote sensing image processing tools have only classical interpolation techniques (such as bi-linear (BL) and bi-cubic/cubic convolution (CC)). Therefore, ENVI- and ERDAS-based researches in image fusion area and even other fusion researches often don’t use new and better interpolators and are mainly concentrated on the fusion algorithm’s details for achieving a better quality, so we only focus on the interpolation impact on fusion quality in Landsat-8 multispectral images. The important feature of this approach is to use a statistical, adaptive, and edge-guided interpolation method for improving the color quality in the images in practice. Numerical simulations show selecting the suitable interpolation techniques in MRF-based images creates better quality than the classical interpolators.

Characterization of a Multimodal and Multispectral Led Imager: Application to Organic Polymer’s Microspheres with Diameter Φ = 10.2 μm

Multispectral microscopy enables information enhancement in the study of specimens because of the large spectral band used in this technique. A low cost multimode multispectral microscope using a camera and a set of quasi-monochromatic Light Emitting Diodes (LEDs) ranging from ultraviolet to near-infrared wavelengths as illumination sources was constructed. But the use of a large spectral band provided by non-monochromatic sources induces variation of focal plan of the imager due to chromatic aberration which rises up the diffraction effects and blurs the images causing shadow around them. It results in discrepancies between standard spectra and extracted spectra with microscope. So we need to calibrate that instrument to be a standard one. We proceed with two types of images comparison to choose the reference wavelength for image acquisition where diffraction effect is more reduced. At each wavelength chosen as a reference, one image is well contrasted. First, we compare the thirteen well contrasted images to identify that presenting more reduced shadow. In second time, we determine the mean of the shadow size over the images from each set. The correction of the discrepancies required measurements on filters using a standard spectrometer and the microscope in transmission mode and reflection mode. To evaluate the capacity of our device to transmit information in frequency domain, its modulation transfer function is evaluated. Multivariate analysis is used to test its capacity to recognize properties of well-known sample. The wavelength 700 nm was chosen to be the reference for the image acquisition, because at this wavelength the images are well contrasted. The measurement made on the filters suggested correction coefficients in transmission mode and reflection mode. The experimental instrument recognized the microsphere’s properties and led to the extraction of the standard transmittance and reflectance spectra. Therefore, this microscope is used as a conventional instrument.

Seismic hazard prediction using multispectral amplification maps in a complex topographic area: A case study of Qiaozhuang town, Sichuan Province, Southwest China

Earthquakes can cause widely distributed slope failures and damage in mountainous areas.The accurate prediction of ground motions in mountainous areas is essential for managing the seismic risk of urban cities near mountains but is restricted primarily by complex seismic site amplification effects in areas of uneven terrain.This study selected Qiaozhuang town located in the Qingchuan–Pingwu fault zone,Southwest China,as a case study.A simulator for mapped seismic responses using a hybrid model(Si Se RHMap)was applied to compute the multispectral seismic topographic amplification maps at the three slope units surrounding Qiaozhuang town(Weigan hill,Mt.Dong,and Mt.Shizi).Post-earthquake damage survey maps,1 D seismic site response spectral ratios,and H/V spectral ratios of earthquake data were used to validate the computed seismic site amplification factors and resonance frequencies.The results suggest that strong topographic amplification effects usually occur at distinct slope locations,such as hilltops,convex slope positions,upslope,and narrow ridges.The computed topographic amplification factors in the study area reached up to 2.4 at upslope or hilltops,and the resonance frequencies were between 3 and 10 Hz.Topographic effects can be as important as stratigraphic effects when assessing seismic amplification effects in the study area.We conclude that both topographic and stratigraphic effects should be considered in the comprehensive seismic hazard assessment of the study area or other similar mountain towns.

Lithological mapping with multispectral data–setup and application of a spectral database for rocks in the Balakot area, Northern Pakistan

In the frame of landslide susceptibility assessment, a spectral library was created to support the identification of materials confined to a particular region using remote sensing images. This library, called Pakistan spectral library(pklib) version 0.1, contains the analysis data of sixty rock samples taken in the Balakot region in Northern Pakistan.The spectral library is implemented as SQLite database. Structure and naming are inspired by the convention system of the ASTER Spectral Library. Usability, application and benefit of the pklib were evaluated and depicted taking two approaches, the multivariate and the spectral based. The spectral information were used to create indices. The indices were applied to Landsat and ASTER data tosupportthespatial delineation of outcropping rock sequences instratigraphic formations. The application of the indices introduced in this paper helps to identify spots where specific lithological characteristics occur. Especially in areas with sparse or missing detailed geological mapping, the spectral discrimination via remote sensing data can speed up the survey. The library can be used not only to support the improvement of factor maps for landslide susceptibility analysis, but also to provide a geoscientific basisto further analyze the lithological spotin numerous regions in the Hindu Kush.

MULTISPECTRAL UNMIXING OF FLUORESCENCE MOLECULAR TOMOGRAPHY DATA

Even though multispectral imaging is considered very significant in biological imaging,it is only commonly used in microscopy in a 2D approach.Here,we present a Fluorescence Molecular Tomography system capable of recording simultaneously tomographic data at several spectral windows,enabling multispectral tomography.3D reconstructed data from several spectral windows is used to construct a linear unmixing algorithm for multispectral deconvolution of overlapping fluorescence signals.The method is applied on tomographic 3D fluorescence concentration maps in tissue-mimicking phantoms,yielding absolute quantification of the concentration of each individual fluorophore.Results are compared to the case when unmixing is performed in the raw 2D data instead of the reconstructed 3D concentration map,showing greater accuracy when unmixing algorithms are applied in the reconstructed data.Both the reflection and transmission geometries are considered.

Multispectral imaging of acute wound tissue oxygenation

This paper investigates the appropriate range of values for the transcutaneous blood oxygen saturation(StO2)of granulating tissues and the surrounding tissue that can ensure timely wound recovery.This work has used a multispectral imaging system to collect wound images at wave-lengths ranging between 520 nm and 600 nm with a resolution of 10 nm.As part of this research,a pilot study was conducted on three injured individuals with superfcial wounds of different wound ages at different skin locations.The S_(t)O_(2)value predicted for the examined wounds using the Extended Modified Lambert-Beer model revealed a mean S_(t)O_(2)of 61±10.3%compared to 41.6±6.2%at the surrounding tissues,and 50.1±1.53%for control sites.These preliminary results contribute to the existing knowledge on the possible range and variation of wound bed S_(t)O_(2)that are to be used as indicators of the functioning of the vasomotion system and wound health.This study has concluded that a high S_(t)O_(2)of approximately 60%and a large fuctuation in this value should precede a good progression in wound healing.

Robust Core Tensor Dictionary Learning with Modified Gaussian Mixture Model for Multispectral Image Restoration

The multispectral remote sensing image(MS-RSI)is degraded existing multi-spectral camera due to various hardware limitations.In this paper,we propose a novel core tensor dictionary learning approach with the robust modified Gaussian mixture model for MS-RSI restoration.First,the multispectral patch is modeled by three-order tensor and high-order singular value decomposition is applied to the tensor.Then the task of MS-RSI restoration is formulated as a minimum sparse core tensor estimation problem.To improve the accuracy of core tensor coding,the core tensor estimation based on the robust modified Gaussian mixture model is introduced into the proposed model by exploiting the sparse distribution prior in image.When applied to MS-RSI restoration,our experimental results have shown that the proposed algorithm can better reconstruct the sharpness of the image textures and can outperform several existing state-of-the-art multispectral image restoration methods in both subjective image quality and visual perception.

An image compression method for space multispectral time delay and integration charge coupled device camera

Multispectral time delay and integration charge coupled device(TDICCD) image compression requires a lowcomplexity encoder because it is usually completed on board where the energy and memory are limited.The Consultative Committee for Space Data Systems(CCSDS) has proposed an image data compression(CCSDS-IDC) algorithm which is so far most widely implemented in hardware.However,it cannot reduce spectral redundancy in multispectral images.In this paper,we propose a low-complexity improved CCSDS-IDC(ICCSDS-IDC)-based distributed source coding(DSC) scheme for multispectral TDICCD image consisting of a few bands.Our scheme is based on an ICCSDS-IDC approach that uses a bit plane extractor to parse the differences in the original image and its wavelet transformed coefficient.The output of bit plane extractor will be encoded by a first order entropy coder.Low-density parity-check-based Slepian-Wolf(SW) coder is adopted to implement the DSC strategy.Experimental results on space multispectral TDICCD images show that the proposed scheme significantly outperforms the CCSDS-IDC-based coder in each band.

ASRNet: Adversarial Segmentation and Registration Networks for Multispectral Fundus Images

Multispectral imaging (MSI) technique is often used to capture imagesof the fundus by illuminating it with different wavelengths of light. However,these images are taken at different points in time such that eyeball movementscan cause misalignment between consecutive images. The multispectral imagesequence reveals important information in the form of retinal and choroidal bloodvessel maps, which can help ophthalmologists to analyze the morphology of theseblood vessels in detail. This in turn can lead to a high diagnostic accuracy of several diseases. In this paper, we propose a novel semi-supervised end-to-end deeplearning framework called “Adversarial Segmentation and Registration Nets”(ASRNet) for the simultaneous estimation of the blood vessel segmentation andthe registration of multispectral images via an adversarial learning process. ASRNet consists of two subnetworks: (i) A segmentation module S that fulfills theblood vessel segmentation task, and (ii) A registration module R that estimatesthe spatial correspondence of an image pair. Based on the segmention-drivenregistration network, we train the segmentation network using a semi-supervisedadversarial learning strategy. Our experimental results show that the proposedASRNet can achieve state-of-the-art accuracy in segmentation and registrationtasks performed with real MSI datasets.