Neuromorphic vision sensors: Principle, progress and perspectives

Conventional frame-based image sensors suffer greatly from high energy consumption and latency.Mimicking neurobiological structures and functionalities of the retina provides a promising way to build a neuromorphic vision sensor with highly efficient image processing.In this review article,we will start with a brief introduction to explain the working mechanism and the challenges of conventional frame-based image sensors,and introduce the structure and functions of biological retina.In the main section,we will overview recent developments in neuromorphic vision sensors,including the silicon retina based on conventional Si CMOS digital technologies,and the neuromorphic vision sensors with the implementation of emerging devices.Finally,we will provide a brief outline of the prospects and outlook for the development of this field.

Progress of Materials and Devices for Neuromorphic Vision Sensors

The latest developments in bio-inspired neuromorphic vision sensors can be summarized in 3 keywords:smaller,faster,and smarter.(1)Smaller:Devices are becoming more compact by integrating previously separated components such as sensors,memory,and processing units.As a prime example,the transition from traditional sensory vision computing to in-sensor vision computing has shown clear benefits,such as simpler circuitry,lower power consumption,and less data redundancy.(2)Swifter:Owing to the nature of physics,smaller and more integrated devices can detect,process,and react to input more quickly.In addition,the methods for sensing and processing optical information using various materials(such as oxide semiconductors)are evolving.(3)Smarter:Owing to these two main research directions,we can expect advanced applications such as adaptive vision sensors,collision sensors,and nociceptive sensors.This review mainly focuses on the recent progress,working mechanisms,image pre-processing techniques,and advanced features of two types of neuromorphic vision sensors based on near-sensor and in-sensor vision computing methodologies.

Low-dimensional optoelectronic synaptic devices for neuromorphic vision sensors

Neuromorphic systems represent a promising avenue for the development of the next generation of artificial intelligence hardware.Machine vision,one of the cores in artificial intelligence,requires system-level support with low power consumption,low latency,and parallel computing.Neuromorphic vision sensors provide an efficient solution for machine vision by simulating the structure and function of the biological retina.Optoelectronic synapses,which use light as the main means to achieve the dual functions of photosensitivity and synapse,are the basic units of the neuromorphic vision sensor.Therefore,it is necessary to develop various optoelectronic synaptic devices to expand the application scenarios of neuromorphic vision systems.This review compares the structure and function for both biological and artificial retina systems,and introduces various optoelectronic synaptic devices based on low-dimensional materials and working mechanisms.In addition,advanced applications of optoelectronic synapses as neuromorphic vision sensors are comprehensively summarized.Finally,the challenges and prospects in this field are briefly discussed.

Vision Sensing-Based Online Correction System for Robotic Weld Grinding

The service cycle and dynamic performance of structural parts are afected by the weld grinding accuracy and surface consistency. Because of reasons such as assembly errors and thermal deformation, the actual track of the robot does not coincide with the theoretical track when the weld is ground ofine, resulting in poor workpiece surface quality. Considering these problems, in this study, a vision sensing-based online correction system for robotic weld grinding was developed. The system mainly included three subsystems: weld feature extraction, grinding, and robot real-time control. The grinding equipment was frst set as a substation for the robot using the WorkVisual software. The input/output (I/O) ports for communication between the robot and the grinding equipment were confgured via the I/O mapping function to enable the robot to control the grinding equipment (start, stop, and speed control). Subsequently, the Ethernet KRL software package was used to write the data interaction structure to realize realtime communication between the robot and the laser vision system. To correct the measurement error caused by the bending deformation of the workpiece, we established a surface profle model of the base material in the weld area using a polynomial ftting algorithm to compensate for the measurement data. The corrected extracted weld width and height errors were reduced by 2.01% and 9.3%, respectively. Online weld seam extraction and correction experiments verifed the efectiveness of the system’s correction function, and the system could control the grinding trajectory error within 0.2 mm. The reliability of the system was verifed through actual weld grinding experiments. The roughness, Ra, could reach 0.504 µm and the average residual height was within 0.21 mm. In this study, we developed a vision sensing-based online correction system for robotic weld grinding with a good correction efect and high robustness.

Three-dimensional catadioptric vision sensor using omnidirectional dot matrix projection

In order to solve the problem of low measurement accuracy caused by uneven imaging resolutions,we develop a three-dimensional catadioptric vision sensor using 20 to 100 lasers arranged in a circular array called omnidirectional dot maxtric projection(ODMP).Based on the imaging characteristic of the sensor,the ODMP can image the area with a high image resolution.The proposed sensor with ODMP can minimize the loss of the detail information by adjusting the projection density.In evaluating the performance of the sensor,real experiments show the designed sensor has high efficiency and high precision for the measurement of the inner surfaces of pipelines.

基于线结构光扫描的烟支表面缺陷检测方法研究

在烟支表面缺陷检测时,为解决传统二维图像方法无法获得深度信息的问题,提出一种基于线结构光传感器旋转扫描的检测方法。烟支在两同向辊轮的旋转带动下实现自转,通过线结构光视觉传感器获取物体轴向轮廓线,使用随机采样一致性拟合轮廓线,通过计算原始点到拟合轮廓的距离判断缺陷点。选取某品牌烟支在检测过程中得到的原始数据,其中多组样本的Kappa值均达到0.97,重复性达到99.3%。研究为烟支缺陷检测提供参考。

基于多维投影时空事件帧的动态视觉传感手势识别

基于视觉的手势识别是虚拟现实、游戏仿真等领域常用的人机交互手段。在实际应用中,手势动作快速变化将导致传统RGB相机或深度相机成像模糊,给手势识别带来巨大挑战。针对上述问题,利用动态视觉传感器捕捉高速手势运动信息,提出一种基于多维投影时空事件帧(spatiotemporal event frame,STEF)的动态视觉数据手势识别方法。将时空信息嵌入到数据投影面融合形成多维投影时空事件帧,克服现有动态视觉信息事件帧表达方法时域信息丢失的局限性,提升动态视觉传感数据的特征表达能力。在此基础上,采用先进的脉冲神经网络对时空事件帧进行分类实现手势识别。在公开数据集上的识别精度达到96.67%,性能优于同类方法,表明该方法可显著提升动态视觉传感数据手势识别准确率。

基于视觉传感器采集光流特征的精神疾病识别方法

目前,精神疾病的筛查主要依靠临床医生的访谈和问卷进行评估,具有高成本、主观性等问题。提出了一种基于视觉传感器采集光流特征的精神疾病识别方法,可以实时监测和预警,以较低的成本和非侵入性的方式来进行精神疾病的辅助筛查。该算法通过视觉传感器全面采集患者的面部数据来提取光流特征,并输入TSMOSNet进行训练。TSMOSNet以TSM为基础网络进行改进,替换普通卷积为光流提取头,针对视觉传感器采集而来的光流特征图进行密集采样,并添加了时序注意力模块、DML蒸馏和VideoMix数据增强等方法,增强了视觉特征,以提高精神疾病识别的准确率。实验结果表明,在H7-BDSN数据集上相比于其他方法,本研究提出的识别算法在精神疾病的筛查任务中取得了最好的效果,准确率为85%,F1分值为0.84。

激光雷达和视觉传感器组合系统的在线校准方法

针对激光雷达和视觉传感器在车辆行驶过程中因振动发生漂移的问题,提出一种组合系统的在线校准方法。基于激光点云和图像的边缘匹配原理进行数据实时检测,检测到融合的传感器数据不准确时对其采取紧急措施,以提高车辆行驶安全性。针对未检测出错误的校准参数,采用梯度下降法优化成本函数,获取新的校准参数,实现传感器数据校正。通过数据集KITTI进行多次参数偏差实验,测试表明,算法可以检测较大的错误校准且能校正较小范围内的平移和旋转误差。

基于视觉-光电传感器的光伏板清洁机器人研究

我国南方湿地及丘陵地区太阳能资源丰富,但路面条件差,光伏板安装位置高,导致灰尘和鸟粪等污染物难以清理,影响光伏电站的经济性和安全性.为解决地面及空间复杂的光伏电站的清洁难题,设计了一种基于视觉-光电传感器的光伏板清洁机器人,分析倾斜光伏板上机器人的附着条件和所需吸力.根据视觉传感器和光电传感器提供的信息,控制机器人自适应路径规划并执行清洁策略,实现光伏板表面灰尘的全面清洁和鸟粪的定位清洁.试验结果表明,光伏板清洁机器人的清洁效率为50 m^(2)/h,鸟粪的平均定位误差为1.38 mm,除尘率为91.16%,有利于促进太阳能光伏发电的提质增效.

Second-order divided difference filter for vision-based relative navigation

A second-order divided difference filter (SDDF) is derived for integrating line of sight measurement from vision sensor with acceleration and angular rate measurements of the follower to estimate the precise relative position,velocity and attitude of two unmanned aerial vehicles (UAVs).The second-order divided difference filter which makes use of multidimensional interpolation formulations to approximate the nonlinear transformations could achieve more accurate estimation and faster convergence from inaccurate initial conditions than standard extended Kalman filter.The filter formulation is based on relative motion equations.The global attitude parameterization is given by quarternion,while a generalized three-dimensional attitude representation is used to define the local attitude error.Simulation results are shown to compare the performance of the second-order divided difference filter with a standard extended Kalman filter approach.

基于激光视觉传感的布局规划方法研究

对于激光跟踪测量任务来说,测量场规划结果十分关键,设计基于激光视觉传感的布局规划方法,以满足实际应用需求。基于激光视觉传感技术设计激光视觉传感器,由相机、滤光片、激光器构成,用于采集测量场图像。对于采集图像,使用自适应中值滤波器对其实施去噪处理。设计应用规划引擎与CAM2的布局规划算法,在多次迭代中,使用粒子群算法求解,通过该算法实现测量场的布局规划。测试结果表明,设计方法两个站位的布局规划结果是互补的,说明该布局规划能够完成测量任务,规划结果有效。

港口拆锁机激光与视觉融合案例分析

针对3D照相、单点激光传感器等单一技术方案在实现集装箱的位置定位过程中的不足之处,提出一种激光与视觉融合识别方案,给出了方案整体思路、平面视觉的模板匹配、点激光制导机械手臂运动等技术要点。实际应用表明,该方案在保留传统光电传感器稳定性优势的前提下,融合使用机器视觉技术,是一种可行的识别方案。

基于机器视觉和PLC的谷糙分级控制系统设计

针对传统谷糙分离机应用人工调节分级板存在定位精度低、成本高、工作效率受限等弊端,提出一种基于机器视觉和PLC的谷糙分级控制系统。搭建嵌入式视觉检测平台,对谷糙检测的图像处理算法进行多级式递进融合设计,对谷糙的特征分级点进行提取;基于位置偏移的方法联动电机驱动程序,实现PLC控制系统的精确定位;结合传感器技术,根据谷糙的重力、表面粗糙度、流量、设备振动频率及倾斜角等特性,在上位机组态监控下进行设备校正,进而实现谷糙自动化、智能化分级。实验结果表明:该谷糙分级控制系统的平均定位精度为96%,回砻谷含糙率低于5%,具有良好的工作性能和稳定性,能够取代传统人工机械操作,降低人工成本,提高工作效率,并增加大米生产线的生产效益。

Calibration of line structured light vision system based on camera’s projective center

Based on the characteristics of line structured light sensor, a speedy method for the calibration was established. With the coplanar reference target, the spacial pose between camera and optical plane can be calibrated by using of the camera’s projective center and the light’s information in the camera’s image surface. Without striction to the movement of the coplanar reference target and assistant adjustment equipment, this calibration method can be implemented. This method has been used and decreased the cost of calibration equipment, simplified the calibration procedure, improved calibration efficiency. Using experiment, the sensor can attain relative accuracy about 0.5%, which indicates the rationality and effectivity of this method.

基于新型多模态触觉传感器的机器人交互物体分类

多模态传感融合对于机器人探索外界环境十分重要,而现有的触觉传感器只能收集一种触觉模态信息,其收集到的多模态信息存在弱配对问题,为了解决此问题,研究了一种基于触觉和视觉融合的多模态触觉传感器。该传感器可以利用一个感知层同时收集2种异构触觉模态信息,弥补了传统触觉传感器的缺陷,同时可以利用收集到的多模态信息对不同物体的几何形状进行分类,在研究过程中,通过按压实验收集了圆形、正方形、长方形和三角形4种形状物体的触觉信息,再利用K最近邻(KNN)算法进行几何形状分类,实验结果证明了该传感器在区分不同物体的形状上具有良好的效果。

受电弓滑板磨耗检测中多视觉传感器标定方法研究

受电弓磨耗缺陷作为受电弓最主要故障缺陷,但由于碳滑板横向尺度大、边缘精度低等特点,单个视觉传感器不能覆盖到所有受电弓轮廓,因此提出一种利用多视觉传感器进行全局标定的标定方法。首先,使用二维靶标对单个视觉传感器进行标定得到摄像机内部参数,然后对激光平面进行标定和求解,最后把三维靶标置于相邻两个摄像机的交叉视角范围内,并求解出两个相邻相机的外部参数。综合摄像机内外部参数和光平面方程即可建立统一坐标系下的数学测量模型,当得到线结构光测量模型后将二维图像的特征点转换为空间点云信息即可实现对滑板磨耗的测量。该系统通过测量三维靶标的尺寸并和真实距离进行对比来验证测量模型的准确性。对比结果表明该方法标定精度高、实现简单、结构稳定,能够有效用来对受电弓碳滑板磨耗的测量。

同步定位与建图技术研究进展

同步定位与建图(SLAM)技术是自主移动机器人的主要技术支撑,成为当今的研究热点。介绍了SLAM技术的发展历程及主要传感器,梳理了基于视觉、激光雷达以及多传感器融合的SLAM技术,并对常见的SLAM算法进行归纳总结,对比分析各实现方案的优缺点。最后探讨了SLAM的技术难题和发展趋势。

A MAP Approach for Vision-based Self-localization of Mobile Robot

一当场，自我本地化系统为在有深入的 3D 里程碑的 3D 环境起作用的活动机器人被开发。机器人通过合并从 odometry 和单向性的照相机收集的信息的一个地图评估者递归地估计它的姿势。我们为这二个传感器造非线性的模型并且坚持说机器人运动和不精密的传感器大小的无常操作应该全部被嵌入并且追踪我们的系统。我们在一个概率的几何学观点和使用 unscented 变换描述无常框架宣传无常，它经历给定的非线性的功能。就我们的机器人的处理力量而言，图象特征在相应投射特征的附近被提取。另外，数据协会被统计距离评估。最后，一系列系统的实验被进行证明我们的系统的可靠、精确的性能。