Neuromorphic vision sensors: Principle, progress and perspectives

Conventional frame-based image sensors suffer greatly from high energy consumption and latency.Mimicking neurobiological structures and functionalities of the retina provides a promising way to build a neuromorphic vision sensor with highly efficient image processing.In this review article,we will start with a brief introduction to explain the working mechanism and the challenges of conventional frame-based image sensors,and introduce the structure and functions of biological retina.In the main section,we will overview recent developments in neuromorphic vision sensors,including the silicon retina based on conventional Si CMOS digital technologies,and the neuromorphic vision sensors with the implementation of emerging devices.Finally,we will provide a brief outline of the prospects and outlook for the development of this field.

Progress of Materials and Devices for Neuromorphic Vision Sensors

The latest developments in bio-inspired neuromorphic vision sensors can be summarized in 3 keywords:smaller,faster,and smarter.(1)Smaller:Devices are becoming more compact by integrating previously separated components such as sensors,memory,and processing units.As a prime example,the transition from traditional sensory vision computing to in-sensor vision computing has shown clear benefits,such as simpler circuitry,lower power consumption,and less data redundancy.(2)Swifter:Owing to the nature of physics,smaller and more integrated devices can detect,process,and react to input more quickly.In addition,the methods for sensing and processing optical information using various materials(such as oxide semiconductors)are evolving.(3)Smarter:Owing to these two main research directions,we can expect advanced applications such as adaptive vision sensors,collision sensors,and nociceptive sensors.This review mainly focuses on the recent progress,working mechanisms,image pre-processing techniques,and advanced features of two types of neuromorphic vision sensors based on near-sensor and in-sensor vision computing methodologies.

Low-dimensional optoelectronic synaptic devices for neuromorphic vision sensors

Neuromorphic systems represent a promising avenue for the development of the next generation of artificial intelligence hardware.Machine vision,one of the cores in artificial intelligence,requires system-level support with low power consumption,low latency,and parallel computing.Neuromorphic vision sensors provide an efficient solution for machine vision by simulating the structure and function of the biological retina.Optoelectronic synapses,which use light as the main means to achieve the dual functions of photosensitivity and synapse,are the basic units of the neuromorphic vision sensor.Therefore,it is necessary to develop various optoelectronic synaptic devices to expand the application scenarios of neuromorphic vision systems.This review compares the structure and function for both biological and artificial retina systems,and introduces various optoelectronic synaptic devices based on low-dimensional materials and working mechanisms.In addition,advanced applications of optoelectronic synapses as neuromorphic vision sensors are comprehensively summarized.Finally,the challenges and prospects in this field are briefly discussed.

CMOS vision sensor with fully digital image process integrated into low power 1/8-inch chip

A digital still camera image processing system on a chip, different from the video camera system, is pre- sented for mobile phone to reduce the power consumption and size. A new color interpolation algorithm is proposed to enhance the image quality. The system can also process fixed patten noise （FPN） reduction, color correction, gamma correction, RGB/YUV space transfer, etc. The chip is controlled by sensor regis- ters by inter-integrated circuit （I2C） interface. The voltage for both the front-end analog and the pad cir- cuits is 2.8 V, and the volatge for the image signal processing is 1.8 V. The chip running under the external 13.5-MHz clock has a video data rate of 30 frames/s and the measured power dissipation is about 75 roW.

Vision Sensing-Based Online Correction System for Robotic Weld Grinding

The service cycle and dynamic performance of structural parts are afected by the weld grinding accuracy and surface consistency. Because of reasons such as assembly errors and thermal deformation, the actual track of the robot does not coincide with the theoretical track when the weld is ground ofine, resulting in poor workpiece surface quality. Considering these problems, in this study, a vision sensing-based online correction system for robotic weld grinding was developed. The system mainly included three subsystems: weld feature extraction, grinding, and robot real-time control. The grinding equipment was frst set as a substation for the robot using the WorkVisual software. The input/output (I/O) ports for communication between the robot and the grinding equipment were confgured via the I/O mapping function to enable the robot to control the grinding equipment (start, stop, and speed control). Subsequently, the Ethernet KRL software package was used to write the data interaction structure to realize realtime communication between the robot and the laser vision system. To correct the measurement error caused by the bending deformation of the workpiece, we established a surface profle model of the base material in the weld area using a polynomial ftting algorithm to compensate for the measurement data. The corrected extracted weld width and height errors were reduced by 2.01% and 9.3%, respectively. Online weld seam extraction and correction experiments verifed the efectiveness of the system’s correction function, and the system could control the grinding trajectory error within 0.2 mm. The reliability of the system was verifed through actual weld grinding experiments. The roughness, Ra, could reach 0.504 µm and the average residual height was within 0.21 mm. In this study, we developed a vision sensing-based online correction system for robotic weld grinding with a good correction efect and high robustness.

Three-dimensional catadioptric vision sensor using omnidirectional dot matrix projection

In order to solve the problem of low measurement accuracy caused by uneven imaging resolutions,we develop a three-dimensional catadioptric vision sensor using 20 to 100 lasers arranged in a circular array called omnidirectional dot maxtric projection（ODMP）.Based on the imaging characteristic of the sensor,the ODMP can image the area with a high image resolution.The proposed sensor with ODMP can minimize the loss of the detail information by adjusting the projection density.In evaluating the performance of the sensor,real experiments show the designed sensor has high efficiency and high precision for the measurement of the inner surfaces of pipelines.

基于线结构光扫描的烟支表面缺陷检测方法研究

在烟支表面缺陷检测时,为解决传统二维图像方法无法获得深度信息的问题,提出一种基于线结构光传感器旋转扫描的检测方法。烟支在两同向辊轮的旋转带动下实现自转,通过线结构光视觉传感器获取物体轴向轮廓线,使用随机采样一致性拟合轮廓线,通过计算原始点到拟合轮廓的距离判断缺陷点。选取某品牌烟支在检测过程中得到的原始数据,其中多组样本的Kappa值均达到0.97,重复性达到99.3%。研究为烟支缺陷检测提供参考。

基于多维投影时空事件帧的动态视觉传感手势识别

基于视觉的手势识别是虚拟现实、游戏仿真等领域常用的人机交互手段。在实际应用中,手势动作快速变化将导致传统RGB相机或深度相机成像模糊,给手势识别带来巨大挑战。针对上述问题,利用动态视觉传感器捕捉高速手势运动信息,提出一种基于多维投影时空事件帧(spatiotemporal event frame,STEF)的动态视觉数据手势识别方法。将时空信息嵌入到数据投影面融合形成多维投影时空事件帧,克服现有动态视觉信息事件帧表达方法时域信息丢失的局限性,提升动态视觉传感数据的特征表达能力。在此基础上,采用先进的脉冲神经网络对时空事件帧进行分类实现手势识别。在公开数据集上的识别精度达到96.67%,性能优于同类方法,表明该方法可显著提升动态视觉传感数据手势识别准确率。

基于视觉传感器采集光流特征的精神疾病识别方法

目前,精神疾病的筛查主要依靠临床医生的访谈和问卷进行评估,具有高成本、主观性等问题。提出了一种基于视觉传感器采集光流特征的精神疾病识别方法,可以实时监测和预警,以较低的成本和非侵入性的方式来进行精神疾病的辅助筛查。该算法通过视觉传感器全面采集患者的面部数据来提取光流特征,并输入TSMOSNet进行训练。TSMOSNet以TSM为基础网络进行改进,替换普通卷积为光流提取头,针对视觉传感器采集而来的光流特征图进行密集采样,并添加了时序注意力模块、DML蒸馏和VideoMix数据增强等方法,增强了视觉特征,以提高精神疾病识别的准确率。实验结果表明,在H7-BDSN数据集上相比于其他方法,本研究提出的识别算法在精神疾病的筛查任务中取得了最好的效果,准确率为85%,F1分值为0.84。

基于视觉和惯性传感器的大型邮轮室内旅客身份识别方法

邮轮内部结构及场景复杂,基于监控用单目摄像头的旅客身份识别方法缺乏深度信息,无法准确识别旅客位置、航向及航向变化信息,难以在复杂场景下准确识别旅客身份。针对上述问题,提出了基于监控用单目摄像头与手持惯性传感器的大型邮轮室内旅客身份识别方法。根据YOLOv5视觉目标检测算法,提取监控视频帧中旅客的像素坐标与边界框;利用2D-3D坐标转换公式,将相机图像中旅客的像素坐标转换为物理世界中旅客与相机的相对坐标;再基于改进神经网络模型,估计旅客在相机相对坐标系下的航向角。利用旅客智能手机中惯性传感器,采集旅客运动数据,检测旅客加速度的变化,判别旅客行走状态;融合磁场强度,推算旅客在大地坐标系下的真实航向角;融合提取的视觉和惯性传感器数据,对旅客的有限特征及其关系进行编码,包括瞬时时刻行走状态、步长、相对航向角和相对距离,解决传感器信号的误差累积问题;基于构建的2幅多关联图,提出特征之间的相似度计算公式,再利用视觉与惯性传感器特征图匹配(vision and inertial sensors graph matching,VIGM)算法求解最大相似度矩阵,实现对2幅图中的同一旅客的识别。经长江“黄金3号”邮轮大厅、棋牌室、多功能厅和走廊4个场景实验,可以发现:VIGM算法在1~3 s窗口内平均匹配准确率达83.9%,与使用高成本深度相机的ViTag身份匹配算法相比,平均匹配准确率仅相差4.5%。实验结果表明:所提基于摄像头与惯性传感器的旅客身份识别方法及算法实现成本低,但识别效果与使用高成本深度相机的方法相当。

激光雷达和视觉传感器组合系统的在线校准方法

针对激光雷达和视觉传感器在车辆行驶过程中因振动发生漂移的问题,提出一种组合系统的在线校准方法。基于激光点云和图像的边缘匹配原理进行数据实时检测,检测到融合的传感器数据不准确时对其采取紧急措施,以提高车辆行驶安全性。针对未检测出错误的校准参数,采用梯度下降法优化成本函数,获取新的校准参数,实现传感器数据校正。通过数据集KITTI进行多次参数偏差实验,测试表明,算法可以检测较大的错误校准且能校正较小范围内的平移和旋转误差。

基于视觉-光电传感器的光伏板清洁机器人研究

我国南方湿地及丘陵地区太阳能资源丰富,但路面条件差,光伏板安装位置高,导致灰尘和鸟粪等污染物难以清理,影响光伏电站的经济性和安全性.为解决地面及空间复杂的光伏电站的清洁难题,设计了一种基于视觉-光电传感器的光伏板清洁机器人,分析倾斜光伏板上机器人的附着条件和所需吸力.根据视觉传感器和光电传感器提供的信息,控制机器人自适应路径规划并执行清洁策略,实现光伏板表面灰尘的全面清洁和鸟粪的定位清洁.试验结果表明,光伏板清洁机器人的清洁效率为50 m^(2)/h,鸟粪的平均定位误差为1.38 mm,除尘率为91.16%,有利于促进太阳能光伏发电的提质增效.

Simultaneous observation of keyhole and weld pool in plasma arc welding with a single cost-effective sensor

The dynamic behaviors of the keyhole and weld pool are coupled together in plasma arc welding, and the geometric variations of both the keyhole and the weld pool determine the weld quality. It is of great significance to simultaneously sense and monitor the keyhole and the weld pool behaviors by using a single low-cost vision sensor in plasma arc welding process. In this study, the keyhole and weld pool were observed and measured under different levels of welding current by using the near infrared sensing technology and the charge coupled device （CCD） sensing system. The shapes and relative position of weld pool and keyhole under different conditions were compared and analyzed. The observation results lay solid foundation for controlling weld quality and understanding the underlying process mechanisms.

Second-order divided difference filter for vision-based relative navigation

A second-order divided difference filter (SDDF) is derived for integrating line of sight measurement from vision sensor with acceleration and angular rate measurements of the follower to estimate the precise relative position,velocity and attitude of two unmanned aerial vehicles (UAVs).The second-order divided difference filter which makes use of multidimensional interpolation formulations to approximate the nonlinear transformations could achieve more accurate estimation and faster convergence from inaccurate initial conditions than standard extended Kalman filter.The filter formulation is based on relative motion equations.The global attitude parameterization is given by quarternion,while a generalized three-dimensional attitude representation is used to define the local attitude error.Simulation results are shown to compare the performance of the second-order divided difference filter with a standard extended Kalman filter approach.

基于虚拟传感器的结构位移视觉测量误差抑制方法

温度、热浪和振动等环境因素给结构位移监测造成不可忽视的误差。虽然利用静止标志物降低视觉测量误差的原理可行,但是其适用范围却因背景标志物不易获得和控制而受到很大限制。利用视觉测量系统的现有条件,在近场布置主动光源作为误差检测虚拟传感器,摆脱对天然标志物的依赖。针对焦外成像产生的弥散圆等伴生问题,采用图像滤波、伽马变换、图像质量评估、图像二值化、边缘检测和目标检测等算法,形成一套面向弥散圆光斑的自适应图像增强算法;利用虚拟传感器的虚假位移,实现结构位移测量误差补偿,开展了仿真实验和室外结构位移视觉测量实验。研究结果表明:该方法可以显著降低结构位移的初始误差,残留误差满足测量精度要求。

基于激光视觉传感的布局规划方法研究

对于激光跟踪测量任务来说,测量场规划结果十分关键,设计基于激光视觉传感的布局规划方法,以满足实际应用需求。基于激光视觉传感技术设计激光视觉传感器,由相机、滤光片、激光器构成,用于采集测量场图像。对于采集图像,使用自适应中值滤波器对其实施去噪处理。设计应用规划引擎与CAM2的布局规划算法,在多次迭代中,使用粒子群算法求解,通过该算法实现测量场的布局规划。测试结果表明,设计方法两个站位的布局规划结果是互补的,说明该布局规划能够完成测量任务,规划结果有效。

港口拆锁机激光与视觉融合案例分析

针对3D照相、单点激光传感器等单一技术方案在实现集装箱的位置定位过程中的不足之处,提出一种激光与视觉融合识别方案,给出了方案整体思路、平面视觉的模板匹配、点激光制导机械手臂运动等技术要点。实际应用表明,该方案在保留传统光电传感器稳定性优势的前提下,融合使用机器视觉技术,是一种可行的识别方案。

基于机器视觉和PLC的谷糙分级控制系统设计

针对传统谷糙分离机应用人工调节分级板存在定位精度低、成本高、工作效率受限等弊端,提出一种基于机器视觉和PLC的谷糙分级控制系统。搭建嵌入式视觉检测平台,对谷糙检测的图像处理算法进行多级式递进融合设计,对谷糙的特征分级点进行提取;基于位置偏移的方法联动电机驱动程序,实现PLC控制系统的精确定位;结合传感器技术,根据谷糙的重力、表面粗糙度、流量、设备振动频率及倾斜角等特性,在上位机组态监控下进行设备校正,进而实现谷糙自动化、智能化分级。实验结果表明:该谷糙分级控制系统的平均定位精度为96%,回砻谷含糙率低于5%,具有良好的工作性能和稳定性,能够取代传统人工机械操作,降低人工成本,提高工作效率,并增加大米生产线的生产效益。

Calibration of line structured light vision system based on camera’s projective center

Based on the characteristics of line structured light sensor, a speedy method for the calibration was established. With the coplanar reference target, the spacial pose between camera and optical plane can be calibrated by using of the camera’s projective center and the light’s information in the camera’s image surface. Without striction to the movement of the coplanar reference target and assistant adjustment equipment, this calibration method can be implemented. This method has been used and decreased the cost of calibration equipment, simplified the calibration procedure, improved calibration efficiency. Using experiment, the sensor can attain relative accuracy about 0.5%, which indicates the rationality and effectivity of this method.

基于新型多模态触觉传感器的机器人交互物体分类

多模态传感融合对于机器人探索外界环境十分重要,而现有的触觉传感器只能收集一种触觉模态信息,其收集到的多模态信息存在弱配对问题,为了解决此问题,研究了一种基于触觉和视觉融合的多模态触觉传感器。该传感器可以利用一个感知层同时收集2种异构触觉模态信息,弥补了传统触觉传感器的缺陷,同时可以利用收集到的多模态信息对不同物体的几何形状进行分类,在研究过程中,通过按压实验收集了圆形、正方形、长方形和三角形4种形状物体的触觉信息,再利用K最近邻(KNN)算法进行几何形状分类,实验结果证明了该传感器在区分不同物体的形状上具有良好的效果。