Towards complex scenes: A deep learning-based camouflaged people detection method for snapshot multispectral images

Camouflaged people are extremely expert in actively concealing themselves by effectively utilizing cover and the surrounding environment. Despite advancements in optical detection capabilities through imaging systems, including spectral, polarization, and infrared technologies, there is still a lack of effective real-time method for accurately detecting small-size and high-efficient camouflaged people in complex real-world scenes. Here, this study proposes a snapshot multispectral image-based camouflaged detection model, multispectral YOLO(MS-YOLO), which utilizes the SPD-Conv and Sim AM modules to effectively represent targets and suppress background interference by exploiting the spatial-spectral target information. Besides, the study constructs the first real-shot multispectral camouflaged people dataset(MSCPD), which encompasses diverse scenes, target scales, and attitudes. To minimize information redundancy, MS-YOLO selects an optimal subset of 12 bands with strong feature representation and minimal inter-band correlation as input. Through experiments on the MSCPD, MS-YOLO achieves a mean Average Precision of 94.31% and real-time detection at 65 frames per second, which confirms the effectiveness and efficiency of our method in detecting camouflaged people in various typical desert and forest scenes. Our approach offers valuable support to improve the perception capabilities of unmanned aerial vehicles in detecting enemy forces and rescuing personnel in battlefield.

D-SS Frame:deep spectral-spatial feature extraction and fusion for classification of panchromatic and multispectral images

Facing the very high-resolution( VHR) image classification problem,a feature extraction and fusion framework is presented for VHR panchromatic and multispectral image classification based on deep learning techniques. The proposed approach combines spectral and spatial information based on the fusion of features extracted from panchromatic( PAN) and multispectral( MS) images using sparse autoencoder and its deep version. There are three steps in the proposed method,the first one is to extract spatial information of PAN image,and the second one is to describe spectral information of MS image. Finally,in the third step,the features obtained from PAN and MS images are concatenated directly as a simple fusion feature. The classification is performed using the support vector machine( SVM) and the experiments carried out on two datasets with very high spatial resolution. MS and PAN images from WorldView-2 satellite indicate that the classifier provides an efficient solution and demonstrate that the fusion of the features extracted by deep learning techniques from PAN and MS images performs better than that when these techniques are used separately. In addition,this framework shows that deep learning models can extract and fuse spatial and spectral information greatly,and have huge potential to achieve higher accuracy for classification of multispectral and panchromatic images.

Automated Classification of Segmented Cancerous Cells in Multispectral Images

Automatic reading procedures in colon cells biopsies allow a faster and precise reading of microscopic biopsies. These procedures implement automatic image segmentation in order to classify cell types as cancerous or noncancerous. The authors have developed a new approach aiming to detect colon cancer cells derived from the ＂Snake＂ method but using a progressive division of the dimensions of the image to achieve rapid segmentation. The aim of the present paper was to classify different cancerous cell types based on nine morphological parameters and on probabilistic neural network. Three types of cells were used to assess the efficiency of our classifications models, including BH （Benign Hyperplasia）, IN （Intraepithelial Neoplasia） that is a precursor state for cancer, and Ca （Carcinoma） that corresponds to abnormal tissue proliferation （cancer）. Results showed that among the nine parameters used to classify cells, only three morphologic parameters （area, Xor convex and solidity） were found to be effective in distinguishing the three types of cells. In addition, classification of unknown cells was possible using this method.

End-to-end computational design for an EUV solar corona multispectral imager with stray light suppression

An extreme ultraviolet solar corona multispectral imager can allow direct observation of high temperature coronal plasma,which is related to solar flares,coronal mass ejections and other significant coronal activities.This manuscript proposes a novel end-to-end computational design method for an extreme ultraviolet(EUV)solar corona multispectral imager operating at wavelengths near 100 nm,including a stray light suppression design and computational image recovery.To suppress the strong stray light from the solar disk,an outer opto-mechanical structure is designed to protect the imaging component of the system.Considering the low reflectivity(less than 70%)and strong-scattering(roughness)of existing extreme ultraviolet optical elements,the imaging component comprises only a primary mirror and a curved grating.A Lyot aperture is used to further suppress any residual stray light.Finally,a deep learning computational imaging method is used to correct the individual multi-wavelength images from the original recorded multi-slit data.In results and data,this can achieve a far-field angular resolution below 7",and spectral resolution below 0.05 nm.The field of view is±3 R_(☉)along the multi-slit moving direction,where R☉represents the radius of the solar disk.The ratio of the corona's stray light intensity to the solar center's irradiation intensity is less than 10-6 at the circle of 1.3 R_(☉).

Multi-temporal NDVI analysis using UAV images of tree crowns in a northern Mexican pine-oak forest

The use of unmanned aerial vehicles(UAV)for forest monitoring has grown significantly in recent years,providing information with high spatial resolution and temporal versatility.UAV with multispectral sensors allow the use of indexes such as the normalized difference vegetation index(NDVI),which determines the vigor,physiological stress and photo synthetic activity of vegetation.This study aimed to analyze the spectral responses and variations of NDVI in tree crowns,as well as their correlation with climatic factors over the course of one year.The study area encompassed a 1.6-ha site in Durango,Mexico,where Pinus cembroides,Pinus engelmannii,and Quercus grisea coexist.Multispectral images were acquired with UAV and information on meteorological variables was obtained from NASA/POWER database.An ANOVA explored possible differences in NDVI among the three species.Pearson correlation was performed to identify the linear relationship between NDVI and meteorological variables.Significant differences in NDVI values were found at the genus level(Pinus and Quercus),possibly related to the physiological features of the species and their phenology.Quercus grisea had the lowest NDVI values throughout the year which may be attributed to its sensitivity to relative humidity and temperatures.Although the use of UAV with a multispectral sensor for NDVI monitoring allowed genera differentiation,in more complex forest analyses hyperspectral and LiDAR sensors should be integrated,as well other vegetation indexes be considered.

ASRNet: Adversarial Segmentation and Registration Networks for Multispectral Fundus Images

Multispectral imaging (MSI) technique is often used to capture imagesof the fundus by illuminating it with different wavelengths of light. However,these images are taken at different points in time such that eyeball movementscan cause misalignment between consecutive images. The multispectral imagesequence reveals important information in the form of retinal and choroidal bloodvessel maps, which can help ophthalmologists to analyze the morphology of theseblood vessels in detail. This in turn can lead to a high diagnostic accuracy of several diseases. In this paper, we propose a novel semi-supervised end-to-end deeplearning framework called “Adversarial Segmentation and Registration Nets”(ASRNet) for the simultaneous estimation of the blood vessel segmentation andthe registration of multispectral images via an adversarial learning process. ASRNet consists of two subnetworks: (i) A segmentation module S that fulfills theblood vessel segmentation task, and (ii) A registration module R that estimatesthe spatial correspondence of an image pair. Based on the segmention-drivenregistration network, we train the segmentation network using a semi-supervisedadversarial learning strategy. Our experimental results show that the proposedASRNet can achieve state-of-the-art accuracy in segmentation and registrationtasks performed with real MSI datasets.

Multiple omics datasets reveal significant physical and physiological dormancy in alfalfa hard seeds identified by multispectral imaging analysis

Physical dormancy(PY) commonly present in the seeds of higher plants is believed to be responsible for the germination failure by impermeable seed coat in hard seeds of legume species, instead of physiological dormancy(PD). In this study, a non-destructive approach involving multispectral imaging was used to successfully identify hard seeds from non-hard seeds in Medicago sativa, with accuracy as high as96.8%–99.0%. We further adopted multiple-omics strategies to investigate the differences of physiology,metabolomics, methylomics, and transcriptomics in alfalfa hard seeds, with non-hard seeds as control.The hard seeds showed dramatically increased antioxidants and 125 metabolites of significant differences in non-targeted metabolomics analysis, which are enriched in the biosynthesis pathways of flavonoids, lipids and hormones, especially with significantly higher ABA, a hormone known to induce dormancy. In our transcriptomics results, the enrichment pathway of “response to abscisic acid” of differential expressed genes(DEG) supported the key role of ABA in metabolomics results. The methylome analysis identified 54,899, 46,216 and 54,452 differential methylation regions for contexts of CpG, CHG and CHH, and 344 DEGs might be regulated by hypermethylation and hypomethylation of promoter and exon regions, including four ABA-and JA-responsive genes. Among 8% hard seeds in seed lots,24.5% still did not germinate after scarifying seed coat, and were named as non-PY hard seeds.Compared to hard seeds, significantly higher contents of ABA/IAA and ABA/JA were identified in nonPY hard seeds, which indicated the potential presence of PD. In summary, the significantly changed metabolites, gene expressions, and methylations all suggested involvement of ABA responses in hard seeds, and germination failure of alfalfa hard seeds was caused by combinational dormancy(PY + PD),rather than PY alone.

Adaptive Window Based 3-D Feature Selection for Multispectral Image Classification Using Firefly Algorithm

Feature extraction is the most critical step in classification of multispectral image.The classification accuracy is mainly influenced by the feature sets that are selected to classify the image.In the past,handcrafted feature sets are used which are not adaptive for different image domains.To overcome this,an evolu-tionary learning method is developed to automatically learn the spatial-spectral features for classification.A modified Firefly Algorithm(FA)which achieves maximum classification accuracy with reduced size of feature set is proposed to gain the interest of feature selection for this purpose.For extracting the most effi-cient features from the data set,we have used 3-D discrete wavelet transform which decompose the multispectral image in all three dimensions.For selecting spatial and spectral features we have studied three different approaches namely overlapping window(OW-3DFS),non-overlapping window(NW-3DFS)adaptive window cube(AW-3DFS)and Pixel based technique.Fivefold Multiclass Support Vector Machine(MSVM)is used for classification purpose.Experiments con-ducted on Madurai LISS IV multispectral image exploited that the adaptive win-dow approach is used to increase the classification accuracy.

PAPS: Progressive Attention-Based Pan-sharpening

Pan-sharpening aims to seek high-resolution multispectral(HRMS) images from paired multispectral images of low resolution(LRMS) and panchromatic(PAN) images, the key to which is how to maximally integrate spatial and spectral information from PAN and LRMS images. Following the principle of gradual advance, this paper designs a novel network that contains two main logical functions, i.e., detail enhancement and progressive fusion, to solve the problem. More specifically, the detail enhancement module attempts to produce enhanced MS results with the same spatial sizes as corresponding PAN images, which are of higher quality than directly up-sampling LRMS images.Having a better MS base(enhanced MS) and its PAN, we progressively extract information from the PAN and enhanced MS images, expecting to capture pivotal and complementary information of the two modalities for the purpose of constructing the desired HRMS. Extensive experiments together with ablation studies on widely-used datasets are provided to verify the efficacy of our design, and demonstrate its superiority over other state-of-the-art methods both quantitatively and qualitatively. Our code has been released at https://github.com/JiaYN1/PAPS.

Fusion of Landsat 8 OLI and PlanetScope Images for Urban Forest Management in Baton Rouge, Louisiana

In recent years image fusion method has been used widely in different studies to improve spatial resolution of multispectral images. This study aims to fuse high resolution satellite imagery with low multispectral imagery in order to assist policymakers in the effective planning and management of urban forest ecosystem in Baton Rouge. To accomplish these objectives, Landsat 8 and PlanetScope satellite images were acquired from United States Geological Survey (USGS) Earth Explorer and Planet websites with pixel resolution of 30m and 3m respectively. The reference images (observed Landsat 8 and PlanetScope imagery) were acquired on 06/08/2020 and 11/19/2020. The image processing was performed in ArcMap and used 6-5-4 band combination for Landsat 8 to visually inspect healthy vegetation and the green spaces. The near-infrared (NIR) panchromatic band for PlanetScope was merged with Landsat 8 image using the Create Pan-Sharpened raster tool in ArcMap and applied the Intensity-Hue-Saturation (IHS) method. In addition, location of urban forestry parks in the study area was picked using the handheld GPS and recorded in an excel sheet. This sheet was converted into Excel (.csv) file and imported into ESRI ArcMap to identify the spatial distribution of the green spaces in East Baton Rouge parish. Results show fused images have better contrast and improve visualization of spatial features than non-fused images. For example, roads, trees, buildings appear sharper, easily discernible, and less pixelated compared to the Landsat 8 image in the fused image. The paper concludes by outlining policy recommendations in the form of sequential measurement of urban forest over time to help track changes and allows for better informed policy and decision making with respect to urban forest management.

Analysis of rice paper's morphological features based on multispectral imaging technology

Computer forensics and identification for traditional Chinese painting arts have caught the attention of various fields. Rice paper＇s feature extraction and analysis methods are of high significance for the rice paper is an important carrier of traditional Chinese painting arts. In this paper, rice paper＇s morphological feature analysis is done using multi spectral imaging technology. The multispectral imaging system is utilized to acquire rice paper＇s spectral images in different wave- length channels, and then those spectral images are measured using texture parameter statistics to acquire sensitive bands for rice paper＇s feature. The mathematical morphology and grayscale statistical principle are utilized to establish a rice paper＇s morphological feature analytical model which is used to acquire rice paper＇ s one-dimensional vector. For the purpose of eval- uating these feature vectors＇ accuracy, they are entered into the support vector machine（SVM） classifier for detection and classification. The results show that the rice paper＇s feature is out loud in the spectral band 550 nm, and the average classifi- cation accuracy of feature vectors output from the analytical model is 96 %. The results indicate that the rice paper＇s feature analytical model can extract most of rice paper＇s features with accuracy and efficiency.

Fusion of multispectral image and panchromatic image based on NSCT and NMF

A novel fusion method of multispectral image and panchromatic image based on nonsubsampled contourlet transform（NSCT） and non-negative matrix factorization（NMF） is presented,the aim of which is to preserve both spectral and spatial information simultaneously in fused image.NMF is a matrix factorization method,which can extract the local feature by choosing suitable dimension of the feature subspace.Firstly the multispectral image was represented in intensity hue saturation（IHS） system.Then the I component and panchromatic image were decomposed by NSCT.Next we used NMF to learn the feature of both multispectral and panchromatic images＇ low-frequency subbands,and the selection principle of the other coefficients was absolute maximum criterion.Finally the new coefficients were reconstructed to get the fused image.Experiments are carried out and the results are compared with some other methods,which show that the new method performs better in improving the spatial resolution and preserving the feature information than the other existing relative methods.

Characterization of a Multimodal and Multispectral Led Imager: Application to Organic Polymer’s Microspheres with Diameter Φ = 10.2 μm

Multispectral microscopy enables information enhancement in the study of specimens because of the large spectral band used in this technique. A low cost multimode multispectral microscope using a camera and a set of quasi-monochromatic Light Emitting Diodes (LEDs) ranging from ultraviolet to near-infrared wavelengths as illumination sources was constructed. But the use of a large spectral band provided by non-monochromatic sources induces variation of focal plan of the imager due to chromatic aberration which rises up the diffraction effects and blurs the images causing shadow around them. It results in discrepancies between standard spectra and extracted spectra with microscope. So we need to calibrate that instrument to be a standard one. We proceed with two types of images comparison to choose the reference wavelength for image acquisition where diffraction effect is more reduced. At each wavelength chosen as a reference, one image is well contrasted. First, we compare the thirteen well contrasted images to identify that presenting more reduced shadow. In second time, we determine the mean of the shadow size over the images from each set. The correction of the discrepancies required measurements on filters using a standard spectrometer and the microscope in transmission mode and reflection mode. To evaluate the capacity of our device to transmit information in frequency domain, its modulation transfer function is evaluated. Multivariate analysis is used to test its capacity to recognize properties of well-known sample. The wavelength 700 nm was chosen to be the reference for the image acquisition, because at this wavelength the images are well contrasted. The measurement made on the filters suggested correction coefficients in transmission mode and reflection mode. The experimental instrument recognized the microsphere’s properties and led to the extraction of the standard transmittance and reflectance spectra. Therefore, this microscope is used as a conventional instrument.

MULTISPECTRAL UNMIXING OF FLUORESCENCE MOLECULAR TOMOGRAPHY DATA

Even though multispectral imaging is considered very significant in biological imaging,it is only commonly used in microscopy in a 2D approach.Here,we present a Fluorescence Molecular Tomography system capable of recording simultaneously tomographic data at several spectral windows,enabling multispectral tomography.3D reconstructed data from several spectral windows is used to construct a linear unmixing algorithm for multispectral deconvolution of overlapping fluorescence signals.The method is applied on tomographic 3D fluorescence concentration maps in tissue-mimicking phantoms,yielding absolute quantification of the concentration of each individual fluorophore.Results are compared to the case when unmixing is performed in the raw 2D data instead of the reconstructed 3D concentration map,showing greater accuracy when unmixing algorithms are applied in the reconstructed data.Both the reflection and transmission geometries are considered.

Multispectral imaging of acute wound tissue oxygenation

This paper investigates the appropriate range of values for the transcutaneous blood oxygen saturation(StO2)of granulating tissues and the surrounding tissue that can ensure timely wound recovery.This work has used a multispectral imaging system to collect wound images at wave-lengths ranging between 520 nm and 600 nm with a resolution of 10 nm.As part of this research,a pilot study was conducted on three injured individuals with superfcial wounds of different wound ages at different skin locations.The S_(t)O_(2)value predicted for the examined wounds using the Extended Modified Lambert-Beer model revealed a mean S_(t)O_(2)of 61±10.3%compared to 41.6±6.2%at the surrounding tissues,and 50.1±1.53%for control sites.These preliminary results contribute to the existing knowledge on the possible range and variation of wound bed S_(t)O_(2)that are to be used as indicators of the functioning of the vasomotion system and wound health.This study has concluded that a high S_(t)O_(2)of approximately 60%and a large fuctuation in this value should precede a good progression in wound healing.

Feature spectrum extraction of human fingernails based on LCTF multispectral imaging

A multispectral imaging system consisting of liquid crystal tunable filter(LCTF)and charge coupled device(CCD)camera was used to collect the images of fingernail samples at intervals of 10 nm during the spectral range of 450-1 000 nm,and a multispectral image of human fingernails containing 56 bands was obtained.The accurate reflectivity information of fingernails was obtained through referring whiteboard comparative measurement method.Principal component analysis(PCA)and band index method were used to reduce the dimension of the sample images respectively and two feature spaces were obtained.Spectral angle mapping(SAM)was used to classify human fingernails in these two feature spaces.The classification accuracy were above 92.5%and 82.9%respectively.Therefore,the feature space obtained by the PCA can be used as the characteristic spectrum of human fingernails,which provides a reliable basis for the analysis of multispectral spectrum of fingernails and human health assessment in the future.

A rate control approach to distributed source coding for interferential multispectral image compression

Distributed source coding （DSC） is applied to interferential multispectral image compression owing to strong correlation among the image frames. Many DSC systems in the literature use feedback channel （FC） to control rate at the decoder, which limits the application of DSC. Upon an analysis of the image data, a rate control approach is proposed to avoid FC. Low-complexity motion compensation is applied first to estimate side information at the encoder. Using a polynomial fitting method, a new mathematical model is then derived to estimate rate based on the correlation between the source and side information. The experimental results show that our estimated rate is a good approximation to the actual rate required by FC while incurring a little bit-rate overhead. Our compression scheme performs comparable with the FC based DSC system and outperforms JPEG2000 significantly.

An image compression method for space multispectral time delay and integration charge coupled device camera

Multispectral time delay and integration charge coupled device （TDICCD） image compression requires a low- complexity encoder because it is usually completed on board where the energy and memory are limited. The Consultative Committee for Space Data Systems （CCSDS） has proposed an image data compression （CCSDS-IDC） algorithm which is so far most widely implemented in hardware. However, it cannot reduce spectral redundancy in mukispectral images. In this paper, we propose a low-complexity improved CCSDS-IDC （ICCSDS-IDC）-based distributed source coding （DSC） scheme for multispectral TDICCD image consisting of a few bands. Our scheme is based on an ICCSDS-IDC approach that uses a bit plane extractor to parse the differences in the original image and its wavelet transformed coefficient. The output of bit plane extractor will be encoded by a first order entropy coder. Low-density parity-check-based Slepian-Wolf （SW） coder is adopted to implement the DSC strategy. Experimental results on space multispectral TDICCD images show that the proposed scheme significantly outperforms the CCSDS-IDC-based coder in each band.

Multi-foci metalens for spectra and polarization ellipticity recognition and reconstruction

Multispectral and polarized focusing and imaging are key functions that are vitally important for a broad range of optical applications.Conventional techniques generally require multiple shots to unveil desired optical information and are implemented via bulky multi-pass systems or mechanically moving parts that are difficult to integrate into compact and integrated optical systems.Here,a design of ultra-compact transversely dispersive metalens capable of both spectrum and polarization ellipticity recognition and reconstruction in just a single shot is demonstrated with both coherent and incoherent light.Our design is well suited for integrated and high-speed optical information analysis and can significantly reduce the size and weight of conventional devices while simplifying the process of collecting optical information,thereby promising for various applications,including machine vision,minimized spectrometers,material characterization,remote sensing,and other areas which require comprehensive optical analysis.

Aerial multi-spectral AI-based detection system for unexploded ordnance

Unexploded ordnance(UXO)poses a threat to soldiers operating in mission areas,but current UXO detection systems do not necessarily provide the required safety and efficiency to protect soldiers from this hazard.Recent technological advancements in artificial intelligence(AI)and small unmanned aerial systems(sUAS)present an opportunity to explore a novel concept for UXO detection.The new UXO detection system proposed in this study takes advantage of employing an AI-trained multi-spectral(MS)sensor on sUAS.This paper explores feasibility of AI-based UXO detection using sUAS equipped with a single(visible)spectrum(SS)or MS digital electro-optical(EO)sensor.Specifically,it describes the design of the Deep Learning Convolutional Neural Network for UXO detection,the development of an AI-based algorithm for reliable UXO detection,and also provides a comparison of performance of the proposed system based on SS and MS sensor imagery.