Towards complex scenes: A deep learning-based camouflaged people detection method for snapshot multispectral images

Camouflaged people are extremely expert in actively concealing themselves by effectively utilizing cover and the surrounding environment. Despite advancements in optical detection capabilities through imaging systems, including spectral, polarization, and infrared technologies, there is still a lack of effective real-time method for accurately detecting small-size and high-efficient camouflaged people in complex real-world scenes. Here, this study proposes a snapshot multispectral image-based camouflaged detection model, multispectral YOLO(MS-YOLO), which utilizes the SPD-Conv and Sim AM modules to effectively represent targets and suppress background interference by exploiting the spatial-spectral target information. Besides, the study constructs the first real-shot multispectral camouflaged people dataset(MSCPD), which encompasses diverse scenes, target scales, and attitudes. To minimize information redundancy, MS-YOLO selects an optimal subset of 12 bands with strong feature representation and minimal inter-band correlation as input. Through experiments on the MSCPD, MS-YOLO achieves a mean Average Precision of 94.31% and real-time detection at 65 frames per second, which confirms the effectiveness and efficiency of our method in detecting camouflaged people in various typical desert and forest scenes. Our approach offers valuable support to improve the perception capabilities of unmanned aerial vehicles in detecting enemy forces and rescuing personnel in battlefield.

End-to-end computational design for an EUV solar corona multispectral imager with stray light suppression

An extreme ultraviolet solar corona multispectral imager can allow direct observation of high temperature coronal plasma,which is related to solar flares,coronal mass ejections and other significant coronal activities.This manuscript proposes a novel end-to-end computational design method for an extreme ultraviolet(EUV)solar corona multispectral imager operating at wavelengths near 100 nm,including a stray light suppression design and computational image recovery.To suppress the strong stray light from the solar disk,an outer opto-mechanical structure is designed to protect the imaging component of the system.Considering the low reflectivity(less than 70%)and strong-scattering(roughness)of existing extreme ultraviolet optical elements,the imaging component comprises only a primary mirror and a curved grating.A Lyot aperture is used to further suppress any residual stray light.Finally,a deep learning computational imaging method is used to correct the individual multi-wavelength images from the original recorded multi-slit data.In results and data,this can achieve a far-field angular resolution below 7",and spectral resolution below 0.05 nm.The field of view is±3 R_(☉)along the multi-slit moving direction,where R☉represents the radius of the solar disk.The ratio of the corona's stray light intensity to the solar center's irradiation intensity is less than 10-6 at the circle of 1.3 R_(☉).

Adaptive Window Based 3-D Feature Selection for Multispectral Image Classification Using Firefly Algorithm

Feature extraction is the most critical step in classification of multispectral image.The classification accuracy is mainly influenced by the feature sets that are selected to classify the image.In the past,handcrafted feature sets are used which are not adaptive for different image domains.To overcome this,an evolu-tionary learning method is developed to automatically learn the spatial-spectral features for classification.A modified Firefly Algorithm(FA)which achieves maximum classification accuracy with reduced size of feature set is proposed to gain the interest of feature selection for this purpose.For extracting the most effi-cient features from the data set,we have used 3-D discrete wavelet transform which decompose the multispectral image in all three dimensions.For selecting spatial and spectral features we have studied three different approaches namely overlapping window(OW-3DFS),non-overlapping window(NW-3DFS)adaptive window cube(AW-3DFS)and Pixel based technique.Fivefold Multiclass Support Vector Machine(MSVM)is used for classification purpose.Experiments con-ducted on Madurai LISS IV multispectral image exploited that the adaptive win-dow approach is used to increase the classification accuracy.

Fusion of multispectral image and panchromatic image based on NSCT and NMF

A novel fusion method of multispectral image and panchromatic image based on nonsubsampled contourlet transform（NSCT） and non-negative matrix factorization（NMF） is presented,the aim of which is to preserve both spectral and spatial information simultaneously in fused image.NMF is a matrix factorization method,which can extract the local feature by choosing suitable dimension of the feature subspace.Firstly the multispectral image was represented in intensity hue saturation（IHS） system.Then the I component and panchromatic image were decomposed by NSCT.Next we used NMF to learn the feature of both multispectral and panchromatic images＇ low-frequency subbands,and the selection principle of the other coefficients was absolute maximum criterion.Finally the new coefficients were reconstructed to get the fused image.Experiments are carried out and the results are compared with some other methods,which show that the new method performs better in improving the spatial resolution and preserving the feature information than the other existing relative methods.

D-SS Frame:deep spectral-spatial feature extraction and fusion for classification of panchromatic and multispectral images

Facing the very high-resolution( VHR) image classification problem,a feature extraction and fusion framework is presented for VHR panchromatic and multispectral image classification based on deep learning techniques. The proposed approach combines spectral and spatial information based on the fusion of features extracted from panchromatic( PAN) and multispectral( MS) images using sparse autoencoder and its deep version. There are three steps in the proposed method,the first one is to extract spatial information of PAN image,and the second one is to describe spectral information of MS image. Finally,in the third step,the features obtained from PAN and MS images are concatenated directly as a simple fusion feature. The classification is performed using the support vector machine( SVM) and the experiments carried out on two datasets with very high spatial resolution. MS and PAN images from WorldView-2 satellite indicate that the classifier provides an efficient solution and demonstrate that the fusion of the features extracted by deep learning techniques from PAN and MS images performs better than that when these techniques are used separately. In addition,this framework shows that deep learning models can extract and fuse spatial and spectral information greatly,and have huge potential to achieve higher accuracy for classification of multispectral and panchromatic images.

Analysis of variograms with various sample sizes from a multispectral image

Variogram plays a crucial role in remote sensing application and geostatistics.It is very important to estimate variogram reliably from sufficient data.In this study,the analysis of variograms computed on various sample sizes of remotely sensed data was conducted.A 100×100-pixel subset was chosen randomly from an aerial multispectral image which contains three wavebands,Green,Red and near-infrared(NIR).Green,Red,NIR and Normalized Difference Vegetation Index(NDVI)datasets were imported into R software for spatial analysis.Variograms of these four full image datasets and sub-samples with simple random sampling method were investigated.In this case,half size of the subset image data was enough to reliably estimate the variograms for NIR and Red wavebands.To map the variation on NDVI within the weed field,ground sampling interval should be smaller than 12 m.The information will be particularly important for Kriging and also give a good guide of field sampling on the weed field in the future study.

Multi-temporal NDVI analysis using UAV images of tree crowns in a northern Mexican pine-oak forest

The use of unmanned aerial vehicles(UAV)for forest monitoring has grown significantly in recent years,providing information with high spatial resolution and temporal versatility.UAV with multispectral sensors allow the use of indexes such as the normalized difference vegetation index(NDVI),which determines the vigor,physiological stress and photo synthetic activity of vegetation.This study aimed to analyze the spectral responses and variations of NDVI in tree crowns,as well as their correlation with climatic factors over the course of one year.The study area encompassed a 1.6-ha site in Durango,Mexico,where Pinus cembroides,Pinus engelmannii,and Quercus grisea coexist.Multispectral images were acquired with UAV and information on meteorological variables was obtained from NASA/POWER database.An ANOVA explored possible differences in NDVI among the three species.Pearson correlation was performed to identify the linear relationship between NDVI and meteorological variables.Significant differences in NDVI values were found at the genus level(Pinus and Quercus),possibly related to the physiological features of the species and their phenology.Quercus grisea had the lowest NDVI values throughout the year which may be attributed to its sensitivity to relative humidity and temperatures.Although the use of UAV with a multispectral sensor for NDVI monitoring allowed genera differentiation,in more complex forest analyses hyperspectral and LiDAR sensors should be integrated,as well other vegetation indexes be considered.

Image Fusion Based on NSCT and Sparse Representation for Remote Sensing Data

The practice of integrating images from two or more sensors collected from the same area or object is known as image fusion.The goal is to extract more spatial and spectral information from the resulting fused image than from the component images.The images must be fused to improve the spatial and spectral quality of both panchromatic and multispectral images.This study provides a novel picture fusion technique that employs L0 smoothening Filter,Non-subsampled Contour let Transform(NSCT)and Sparse Representation(SR)followed by the Max absolute rule(MAR).The fusion approach is as follows:first,the multispectral and panchromatic images are divided into lower and higher frequency components using the L0 smoothing filter.Then comes the fusion process,which uses an approach that combines NSCT and SR to fuse low frequency components.Similarly,the Max-absolute fusion rule is used to merge high frequency components.Finally,the final image is obtained through the disintegration of fused low and high frequency data.In terms of correlation coefficient,Entropy,spatial frequency,and fusion mutual information,our method outperforms other methods in terms of image quality enhancement and visual evaluation.

Hyperspectral Image Sharpening Based on Deep Convolutional Neural Network and Spatial-Spectral Spread Transform Models

In order to improve the spatial resolution of hyperspectral(HS)image and minimize the spectral distortion,an HS and multispectral(MS)image fusion approach based on convolutional neural network(CNN)is proposed.The proposed approach incorporates the linear spectral mixture model and spatial-spectral spread transform model into the learning phase of network,aiming to fully exploit the spatial-spectral information of HS and MS images,and improve the spectral fidelity of fusion images.Experiments on two real remote sensing data under different resolutions demonstrate that compared with some state-of-the-art HS and MS image fusion methods,the proposed approach achieves superior spectral fidelities and lower fusion errors.

ASRNet: Adversarial Segmentation and Registration Networks for Multispectral Fundus Images

Multispectral imaging (MSI) technique is often used to capture imagesof the fundus by illuminating it with different wavelengths of light. However,these images are taken at different points in time such that eyeball movementscan cause misalignment between consecutive images. The multispectral imagesequence reveals important information in the form of retinal and choroidal bloodvessel maps, which can help ophthalmologists to analyze the morphology of theseblood vessels in detail. This in turn can lead to a high diagnostic accuracy of several diseases. In this paper, we propose a novel semi-supervised end-to-end deeplearning framework called “Adversarial Segmentation and Registration Nets”(ASRNet) for the simultaneous estimation of the blood vessel segmentation andthe registration of multispectral images via an adversarial learning process. ASRNet consists of two subnetworks: (i) A segmentation module S that fulfills theblood vessel segmentation task, and (ii) A registration module R that estimatesthe spatial correspondence of an image pair. Based on the segmention-drivenregistration network, we train the segmentation network using a semi-supervisedadversarial learning strategy. Our experimental results show that the proposedASRNet can achieve state-of-the-art accuracy in segmentation and registrationtasks performed with real MSI datasets.

Characterization of a Multimodal and Multispectral Led Imager: Application to Organic Polymer’s Microspheres with Diameter Φ = 10.2 μm

Multispectral microscopy enables information enhancement in the study of specimens because of the large spectral band used in this technique. A low cost multimode multispectral microscope using a camera and a set of quasi-monochromatic Light Emitting Diodes (LEDs) ranging from ultraviolet to near-infrared wavelengths as illumination sources was constructed. But the use of a large spectral band provided by non-monochromatic sources induces variation of focal plan of the imager due to chromatic aberration which rises up the diffraction effects and blurs the images causing shadow around them. It results in discrepancies between standard spectra and extracted spectra with microscope. So we need to calibrate that instrument to be a standard one. We proceed with two types of images comparison to choose the reference wavelength for image acquisition where diffraction effect is more reduced. At each wavelength chosen as a reference, one image is well contrasted. First, we compare the thirteen well contrasted images to identify that presenting more reduced shadow. In second time, we determine the mean of the shadow size over the images from each set. The correction of the discrepancies required measurements on filters using a standard spectrometer and the microscope in transmission mode and reflection mode. To evaluate the capacity of our device to transmit information in frequency domain, its modulation transfer function is evaluated. Multivariate analysis is used to test its capacity to recognize properties of well-known sample. The wavelength 700 nm was chosen to be the reference for the image acquisition, because at this wavelength the images are well contrasted. The measurement made on the filters suggested correction coefficients in transmission mode and reflection mode. The experimental instrument recognized the microsphere’s properties and led to the extraction of the standard transmittance and reflectance spectra. Therefore, this microscope is used as a conventional instrument.

Multiple omics datasets reveal significant physical and physiological dormancy in alfalfa hard seeds identified by multispectral imaging analysis

Physical dormancy(PY) commonly present in the seeds of higher plants is believed to be responsible for the germination failure by impermeable seed coat in hard seeds of legume species, instead of physiological dormancy(PD). In this study, a non-destructive approach involving multispectral imaging was used to successfully identify hard seeds from non-hard seeds in Medicago sativa, with accuracy as high as96.8%–99.0%. We further adopted multiple-omics strategies to investigate the differences of physiology,metabolomics, methylomics, and transcriptomics in alfalfa hard seeds, with non-hard seeds as control.The hard seeds showed dramatically increased antioxidants and 125 metabolites of significant differences in non-targeted metabolomics analysis, which are enriched in the biosynthesis pathways of flavonoids, lipids and hormones, especially with significantly higher ABA, a hormone known to induce dormancy. In our transcriptomics results, the enrichment pathway of “response to abscisic acid” of differential expressed genes(DEG) supported the key role of ABA in metabolomics results. The methylome analysis identified 54,899, 46,216 and 54,452 differential methylation regions for contexts of CpG, CHG and CHH, and 344 DEGs might be regulated by hypermethylation and hypomethylation of promoter and exon regions, including four ABA-and JA-responsive genes. Among 8% hard seeds in seed lots,24.5% still did not germinate after scarifying seed coat, and were named as non-PY hard seeds.Compared to hard seeds, significantly higher contents of ABA/IAA and ABA/JA were identified in nonPY hard seeds, which indicated the potential presence of PD. In summary, the significantly changed metabolites, gene expressions, and methylations all suggested involvement of ABA responses in hard seeds, and germination failure of alfalfa hard seeds was caused by combinational dormancy(PY + PD),rather than PY alone.

An image compression method for space multispectral time delay and integration charge coupled device camera

Multispectral time delay and integration charge coupled device （TDICCD） image compression requires a low- complexity encoder because it is usually completed on board where the energy and memory are limited. The Consultative Committee for Space Data Systems （CCSDS） has proposed an image data compression （CCSDS-IDC） algorithm which is so far most widely implemented in hardware. However, it cannot reduce spectral redundancy in mukispectral images. In this paper, we propose a low-complexity improved CCSDS-IDC （ICCSDS-IDC）-based distributed source coding （DSC） scheme for multispectral TDICCD image consisting of a few bands. Our scheme is based on an ICCSDS-IDC approach that uses a bit plane extractor to parse the differences in the original image and its wavelet transformed coefficient. The output of bit plane extractor will be encoded by a first order entropy coder. Low-density parity-check-based Slepian-Wolf （SW） coder is adopted to implement the DSC strategy. Experimental results on space multispectral TDICCD images show that the proposed scheme significantly outperforms the CCSDS-IDC-based coder in each band.

PAPS: Progressive Attention-Based Pan-sharpening

Pan-sharpening aims to seek high-resolution multispectral(HRMS) images from paired multispectral images of low resolution(LRMS) and panchromatic(PAN) images, the key to which is how to maximally integrate spatial and spectral information from PAN and LRMS images. Following the principle of gradual advance, this paper designs a novel network that contains two main logical functions, i.e., detail enhancement and progressive fusion, to solve the problem. More specifically, the detail enhancement module attempts to produce enhanced MS results with the same spatial sizes as corresponding PAN images, which are of higher quality than directly up-sampling LRMS images.Having a better MS base(enhanced MS) and its PAN, we progressively extract information from the PAN and enhanced MS images, expecting to capture pivotal and complementary information of the two modalities for the purpose of constructing the desired HRMS. Extensive experiments together with ablation studies on widely-used datasets are provided to verify the efficacy of our design, and demonstrate its superiority over other state-of-the-art methods both quantitatively and qualitatively. Our code has been released at https://github.com/JiaYN1/PAPS.

Spatial-Resolution Independent Object Detection Framework for Aerial Imagery

Earth surveillance through aerial images allows more accurate identification and characterization of objects present on the surface from space and airborne platforms.The progression of deep learning and computer vision methods and the availability of heterogeneous multispectral remote sensing data make the field more fertile for research.With the evolution of optical sensors,aerial images are becoming more precise and larger,which leads to a new kind of problem for object detection algorithms.This paper proposes the“Sliding Region-based Convolutional Neural Network(SRCNN),”which is an extension of the Faster Region-based Convolutional Neural Network(RCNN)object detection framework to make it independent of the image’s spatial resolution and size.The sliding box strategy is used in the proposed model to segment the image while detecting.The proposed framework outperforms the state-of-the-art Faster RCNN model while processing images with significantly different spatial resolution values.The SRCNN is also capable of detecting objects in images of any size.

Fusion of Landsat 8 OLI and PlanetScope Images for Urban Forest Management in Baton Rouge, Louisiana

In recent years image fusion method has been used widely in different studies to improve spatial resolution of multispectral images. This study aims to fuse high resolution satellite imagery with low multispectral imagery in order to assist policymakers in the effective planning and management of urban forest ecosystem in Baton Rouge. To accomplish these objectives, Landsat 8 and PlanetScope satellite images were acquired from United States Geological Survey (USGS) Earth Explorer and Planet websites with pixel resolution of 30m and 3m respectively. The reference images (observed Landsat 8 and PlanetScope imagery) were acquired on 06/08/2020 and 11/19/2020. The image processing was performed in ArcMap and used 6-5-4 band combination for Landsat 8 to visually inspect healthy vegetation and the green spaces. The near-infrared (NIR) panchromatic band for PlanetScope was merged with Landsat 8 image using the Create Pan-Sharpened raster tool in ArcMap and applied the Intensity-Hue-Saturation (IHS) method. In addition, location of urban forestry parks in the study area was picked using the handheld GPS and recorded in an excel sheet. This sheet was converted into Excel (.csv) file and imported into ESRI ArcMap to identify the spatial distribution of the green spaces in East Baton Rouge parish. Results show fused images have better contrast and improve visualization of spatial features than non-fused images. For example, roads, trees, buildings appear sharper, easily discernible, and less pixelated compared to the Landsat 8 image in the fused image. The paper concludes by outlining policy recommendations in the form of sequential measurement of urban forest over time to help track changes and allows for better informed policy and decision making with respect to urban forest management.

Image Analysis in Microbiology: A Review

This review is focused on using computer image analysis as a means of objective and quantitative characterizing optical images of the macroscopic (e.g. microbial colonies) and the microscopic (e.g. single cell) objects in the microbiological research. This is the way of making many visual inspection assays more objective and less time and labor consuming. Also, it can provide new visually inaccessible information on relation between some optical parameters and various biological features of the microbial cul-tures. Of special interest is application of image analysis in fluorescence microscopy as it opens new ways of using fluorescence based methodology for single microbial cell studies. Examples of using image analysis in the studies of both the macroscopic and the microscopic microbiological objects obtained by various imaging techniques are presented and discussed.

MULTISPECTRAL UNMIXING OF FLUORESCENCE MOLECULAR TOMOGRAPHY DATA

Even though multispectral imaging is considered very significant in biological imaging,it is only commonly used in microscopy in a 2D approach.Here,we present a Fluorescence Molecular Tomography system capable of recording simultaneously tomographic data at several spectral windows,enabling multispectral tomography.3D reconstructed data from several spectral windows is used to construct a linear unmixing algorithm for multispectral deconvolution of overlapping fluorescence signals.The method is applied on tomographic 3D fluorescence concentration maps in tissue-mimicking phantoms,yielding absolute quantification of the concentration of each individual fluorophore.Results are compared to the case when unmixing is performed in the raw 2D data instead of the reconstructed 3D concentration map,showing greater accuracy when unmixing algorithms are applied in the reconstructed data.Both the reflection and transmission geometries are considered.

Multispectral imaging of acute wound tissue oxygenation

This paper investigates the appropriate range of values for the transcutaneous blood oxygen saturation(StO2)of granulating tissues and the surrounding tissue that can ensure timely wound recovery.This work has used a multispectral imaging system to collect wound images at wave-lengths ranging between 520 nm and 600 nm with a resolution of 10 nm.As part of this research,a pilot study was conducted on three injured individuals with superfcial wounds of different wound ages at different skin locations.The S_(t)O_(2)value predicted for the examined wounds using the Extended Modified Lambert-Beer model revealed a mean S_(t)O_(2)of 61±10.3%compared to 41.6±6.2%at the surrounding tissues,and 50.1±1.53%for control sites.These preliminary results contribute to the existing knowledge on the possible range and variation of wound bed S_(t)O_(2)that are to be used as indicators of the functioning of the vasomotion system and wound health.This study has concluded that a high S_(t)O_(2)of approximately 60%and a large fuctuation in this value should precede a good progression in wound healing.

Multi-foci metalens for spectra and polarization ellipticity recognition and reconstruction

Multispectral and polarized focusing and imaging are key functions that are vitally important for a broad range of optical applications.Conventional techniques generally require multiple shots to unveil desired optical information and are implemented via bulky multi-pass systems or mechanically moving parts that are difficult to integrate into compact and integrated optical systems.Here,a design of ultra-compact transversely dispersive metalens capable of both spectrum and polarization ellipticity recognition and reconstruction in just a single shot is demonstrated with both coherent and incoherent light.Our design is well suited for integrated and high-speed optical information analysis and can significantly reduce the size and weight of conventional devices while simplifying the process of collecting optical information,thereby promising for various applications,including machine vision,minimized spectrometers,material characterization,remote sensing,and other areas which require comprehensive optical analysis.