基于视觉和惯性传感器的大型邮轮室内旅客身份识别方法

A Method for Indoor Passenger Identity Recognition on Large Cruise Ships Based on Vision and Inertial Sensors

邮轮内部结构及场景复杂,基于监控用单目摄像头的旅客身份识别方法缺乏深度信息,无法准确识别旅客位置、航向及航向变化信息,难以在复杂场景下准确识别旅客身份。针对上述问题,提出了基于监控用单目摄像头与手持惯性传感器的大型邮轮室内旅客身份识别方法。根据YOLOv5视觉目标检测算法,提取监控视频帧中旅客的像素坐标与边界框;利用2D-3D坐标转换公式,将相机图像中旅客的像素坐标转换为物理世界中旅客与相机的相对坐标;再基于改进神经网络模型,估计旅客在相机相对坐标系下的航向角。利用旅客智能手机中惯性传感器,采集旅客运动数据,检测旅客加速度的变化,判别旅客行走状态;融合磁场强度,推算旅客在大地坐标系下的真实航向角;融合提取的视觉和惯性传感器数据,对旅客的有限特征及其关系进行编码,包括瞬时时刻行走状态、步长、相对航向角和相对距离,解决传感器信号的误差累积问题;基于构建的2幅多关联图,提出特征之间的相似度计算公式,再利用视觉与惯性传感器特征图匹配(vision and inertial sensors graph matching,VIGM)算法求解最大相似度矩阵,实现对2幅图中的同一旅客的识别。经长江“黄金3号”邮轮大厅、棋牌室、多功能厅和走廊4个场景实验,可以发现:VIGM算法在1~3 s窗口内平均匹配准确率达83.9%,与使用高成本深度相机的ViTag身份匹配算法相比,平均匹配准确率仅相差4.5%。实验结果表明:所提基于摄像头与惯性传感器的旅客身份识别方法及算法实现成本低,但识别效果与使用高成本深度相机的方法相当。

The internal structure and scenes on cruise ships are complex and the surveillance camera offers limited depth information,which makes it difficult to identify the location,heading,changes in heading,and the identity of the passengers by the traditional passenger identity recognition method(PIRM)based on a single surveillance camera.To fill the gap,a novel method for indoor PIRM based on vision and inertial sensors is proposed.The YOLOv5 algorithm is used to extract the bounding box of each passenger and assign the pixel coordinate for each box;the pixel coordinate is further converted into the world coordinate system fixing on the camera according to the 2D-3D coordinate transformation formula;an improved neural network model then is used to estimate the true heading angle of passengers in the camera coordinate system.The inertial sensor data from passengers'smartphones are collected to detect the acceleration of the passengers and their walking states;the true heading angle of passengers in the world geodetic system is calculated by integrating magnetic field intensity;then,the extracted visual and inertial sensor data are fused,and limited features of passengers and their relationships are encoded,including walking state,step length,relative heading angle,relative distance,so as to solve the error accumulation problem of sensor signals.A similarity calculation formula between the features is proposed based on the two multi-correlation graphs,and the Vision and Inertial Sensors Graph Matching(VIGM)algorithm is employed to solve the maximum similarity matrix,which could identify the same passenger in both graphs.Lastly,to validate the proposed method,four scenes on the“Yangtze River Golden 3”cruise ship are employed(including the lobby,chess room,multi-function hall,and corridor),and it is found that:the average matching accuracy(AMA)of the proposed VIGM algorithm reaches 83.9% with the 1—3 s time window;the AMA of the proposed algorithm is only 4.5% lower than the ViTag algorithm using high-cost depth cameras.The results of experiments show that the proposed PIRM and VIGM algorithm have low implementation costs but equivalent performance compared to the method using high-cost depth cameras on large cruise ships.