基于HF-Net光谱特征重定位的三维光谱成像技术研究

Exploration into 3D Spectral Imaging Realized through HF-Net-based Relocation of Spectral Feature

本文研究采用新型卷积神经网络HF-Net的深度学习技术将光谱数据在三维重建稠密点云上进行重定位的可行性。首先基于HF-Net多任务学习的光谱重定位方法,是一种异构的定位方法,使用MobileNet和NetVLAD层提取光谱图像的全局描述子,在三维重建彩色点云的数据集进行全局检索,得到此光谱照片对应的三维点云的大致位置。使用光谱图像的局部描述子和关键点得分,进行局部特征匹配,找到光谱图像中的光谱信息,对应三维模型点云中的匹配点,从而完成光谱信息和三维模型的映射。结果表明,通过HF-Net实现光谱特征点匹配后,可以将光谱信息完整地映射到三维重建模型上。本文提出的方法,可以实现立体物证的三维空间信息和光谱特征的精细定位,是人工智能在物证全维度影像数据融合技术中的新应用。

3D-reconstructed relocation of spectral feature is a key technology in multi-spectral imaging fusion and 3D reconstruction.Here,the feasibility was to explore about relocating spectral data onto 3D-reconstructed dense point cloud through a new neural network HF-Net,one deep-learning-based AI technology and also a hierarchical localization approach based on a monolithic CNN that is able to simultaneously predict local features and global descriptors for spectral localization.Such an HFNet was adopted to carry out the spectral relocation of heterogeneous localization.Specifically,the MobileNet and NetVLAD layers were taken to extract the global descriptors among the dataset of three-dimensional color point cloud from the spectral image so as to find the approximate position of spectral image in the three-dimensional point cloud.With the conjectural locations obtained through the prior global retrieval within those candidate places,the SuperPoint was utilized to get the local descriptors and key point scores of the spectral image so that the matching spectral points were found in the three-dimensional point cloud,therewith having mapped out the spectral information and three-dimensional reconstruction pattern.By leveraging the learned descriptors,this assay achieved remarkable localization robustness across large variations of appearance,demonstrating more robust and efficient than SSD algorithm.Due to limits with GPU(graphics processing unit)memory,the extracted spectral features were down-sampled from image to the largest resolution of 6016×4512 pixels.With HF-Net being trained through multitask distillation in TensorFlow 1.12,a spectral image had been able to relocate into 3D reconstruction pattern in 12s under having run on the device of NVidia TESLA V100 with 32G memory and CPU of Intel(R)Xeon(R)Silver 4114 with 12G memory.The approach proposed here can realize the fine positioning of 3D spatial information and spectral features of 3D physical evidence.At present,there seems no such an exploration in forensic science home and abroad,revealing that the exploration tried here is a new application of artificial intelligence to the full-dimensional imaging data fusion technology of physical evidence.Such an HF-Net-based relocation is accurate,scalable,efficient,and a monolithic deep neural network choice for descriptor extraction,capable of achieving higher exactness on large-scale multi-spectral imaging fusion and 3D reconstruction.