End-to-End Multiview Gesture Recognition for Autonomous Car Parking System

The use of hand gestures can be the most intuitive human-machine interaction medium.The early approaches for hand gesture recognition used device-based methods.These methods use mechanical or optical sensors attached to a glove or markers,which hinder the natural human-machine communication.On the other hand,vision-based methods are less restrictive and allow for a more spontaneous communication without the need of an intermediary between human and machine.Therefore,vision gesture recognition has been a popular area of research for the past thirty years.Hand gesture recognition finds its application in many areas,particularly the automotive industry where advanced automotive human-machine interface(HMI)designers are using gesture recognition to improve driver and vehicle safety.However,technology advances go beyond active/passive safety and into convenience and comfort.In this context,one of America’s big three automakers has partnered with the Centre of Pattern Analysis and Machine Intelligence(CPAMI)at the University of Waterloo to investigate expanding their product segment through machine learning to provide an increased driver convenience and comfort with the particular application of hand gesture recognition for autonomous car parking.The present paper leverages the state-of-the-art deep learning and optimization techniques to develop a vision-based multiview dynamic hand gesture recognizer for a self-parking system.We propose a 3D-CNN gesture model architecture that we train on a publicly available hand gesture database.We apply transfer learning methods to fine-tune the pre-trained gesture model on custom-made data,which significantly improves the proposed system performance in a real world environment.We adapt the architecture of end-to-end solution to expand the state-of-the-art video classifier from a single image as input(fed by monocular camera)to a Multiview 360 feed,offered by a six cameras module.Finally,we optimize the proposed solution to work on a limited resource embedded platform(Nvidia Jetson TX2)that is used by automakers for vehicle-based features,without sacrificing the accuracy robustness and real time functionality of the system.

Home Automation-Based Health Assessment Along Gesture Recognition via Inertial Sensors

Hand gesture recognition (HGR) is used in a numerous applications,including medical health-care, industrial purpose and sports detection.We have developed a real-time hand gesture recognition system using inertialsensors for the smart home application. Developing such a model facilitatesthe medical health field (elders or disabled ones). Home automation has alsobeen proven to be a tremendous benefit for the elderly and disabled. Residentsare admitted to smart homes for comfort, luxury, improved quality of life,and protection against intrusion and burglars. This paper proposes a novelsystem that uses principal component analysis, linear discrimination analysisfeature extraction, and random forest as a classifier to improveHGRaccuracy.We have achieved an accuracy of 94% over the publicly benchmarked HGRdataset. The proposed system can be used to detect hand gestures in thehealthcare industry as well as in the industrial and educational sectors.

Deep Learning Based Hand Gesture Recognition and UAV Flight Controls

Dynamic hand gesture recognition is a desired alternative means for human-computer interactions.This paper presents a hand gesture recognition system that is designed for the control of flights of unmanned aerial vehicles(UAV).A data representation model that represents a dynamic gesture sequence by converting the 4-D spatiotemporal data to 2-D matrix and a 1-D array is introduced.To train the system to recognize designed gestures,skeleton data collected from a Leap Motion Controller are converted to two different data models.As many as 9124 samples of the training dataset,1938 samples of the testing dataset are created to train and test the proposed three deep learning neural networks,which are a 2-layer fully connected neural network,a 5-layer fully connected neural network and an 8-layer convolutional neural network.The static testing results show that the 2-layer fully connected neural network achieves an average accuracy of 96.7%on scaled datasets and 12.3%on non-scaled datasets.The 5-layer fully connected neural network achieves an average accuracy of 98.0%on scaled datasets and 89.1%on non-scaled datasets.The 8-layer convolutional neural network achieves an average accuracy of 89.6%on scaled datasets and 96.9%on non-scaled datasets.Testing on a drone-kit simulator and a real drone shows that this system is feasible for drone flight controls.

基于手势的无人机飞行控制实验教学设计

为解决新手在操纵无人机飞行的初始练习阶段容易造成丢失、坠落、碰撞等问题,利用AI技术设计一种基于手势的无人机飞行智能控制实验。首先通过机载或地面摄像头捕捉操纵者手势图像;其次使用加载于地面控制系统或机载设备的动态手势识别模型进一步处理图像,得到相关飞行控制指令;再次将飞行控制指令发送到无人机控制系统以达到操纵无人机飞行的目的。为了验证基于YOLOv5算法的动态手势识别模型性能,构建包含5个场景的2500余张图片的数据集进行验证。实验表明该方法不仅对手势识别具有较高的精确度,而且对新场景下的图像也具有更好的智能检测能力。实验设计涉及学科较为综合、功能扩展性好、实用性强,是控制系统教学、AI算法实践与嵌入式系统相互融合的创新实验项目。

Hand Gesture-based Artificial Neural Network Trained Hybrid Human–machine Interface System to Navigate a Powered Wheelchair

Individuals with cerebral palsy and muscular dystrophy often lack fine motor control of their fingers which makes it difficult to control traditional powered wheelchairs using a joystick.Studies have shown the use of surface electromyography to steer powered wheelchairs or automobiles either through simulations or gaming controllers.However,these studies significantly lack issues with real world scenarios such as user’s safety,real-time control,and efficiency of the controller mechanism.The purpose of this study was to design,evaluate,and implement a hybrid human–machine interface system for a powered wheelchair that can detect human intent based on artificial neural network trained hand gesture recognition and navigate a powered wheelchair without colliding with objects around the path.Scaled Conjugate Gradient(SCG),Bayesian Regularization(BR),and Levenberg Marquart(LM)supervised artificial neural networks were trained in offline testing on eight participants without disability followed by online testing using the classifier with highest accuracy.Bayesian Regularization architecture showed highest accuracy at 98.4%across all participants and hidden layers.All participants successfully completed the path in an average of 5 min and 50 s,touching an average of 22.1%of the obstacles.The proposed hybrid system can be implemented to assist people with neuromuscular disabilities in near future.

基于Kinect的手势识别算法研究及应用

手势识别技术是人机交互技术的重要研究内容。为了提高基于Kinect的手势识别性能,提出基于深度人手定位和hog特征的静态手势识别算法及基于改进HMMs的动态手势识别算法。静态手势识别算法首先通过Kinect的深度信息完成人手定位,而后在定位区域内提取基于梯度方向直方图的形状特征并利用级联Adaboost训练的手势模型,实现对静态手势的准确识别,在公开手势数据库中测试的实验结果表明提出的静态手势识别算法具有较高的识别率。动态手势识别算法首先通过Kinect获取手心轨迹并提取轨迹切线角度作为特征,利用改进的隐马尔科夫模型实现动态手势的判别,实验结果表明提出的动态手势识别算法相比于传统HMMs算法有效地排除了无效手势。此外利用提出的动静态手势识别算法有效地控制了模拟的数字电视。

智能轮椅嵌入式手势控制系统设计与实现

目前智能轮椅的手势控制系统多基于通用PC机,针对其携带不方便、功耗大、成本高等缺点,设计了一种嵌入式手势控制系统。该系统采用S3C2440为微处理器的ARM9核心板作为硬件平台,Linux操作系统作为软件平台,运用AdaBoost算法和Kalman滤波器实现了手势识别,并利用Qt/Embedded图形开发环境完成了嵌入式人机交互界面的设计。实验证明,该嵌入式手势控制系统能很好地应用在智能轮椅上,平均识别率为95.3%。

远程控制采摘机器人手势识别研究——基于势场蚁群算法

采摘机器人在作业时遇到通过自主导航无法越过的障碍物时,或者在危险的地带无法进行人工采摘作业时,需要借助远程方式进行实时控制,使其成功越过障碍物,并在高危环境中有效地展开采摘作业。为了优化采摘机器人远程控制系统,提出了一种基于手势识别的远程控制方案,并引入了势场蚁群算法,提高了机器人的控制的准确性和高效性。在远程控制方案中,将基于视觉的手势识别与远程控制机械手相结合,通过深度相机采集手势图像并提取手势特征,转换为机械手舵机的控制命令,并通过无线网络发送至采摘机器人控制单元,实现视觉手势对机器人的远程控制。对采摘机器人进行了测试,通过测试发现:基于蚁群算法的手势识别系统可以有效地追踪得到不同的动态手势,且可以准确地识别手势所代表的意义,成功实现了机器人远程控制的手势识别。该方法不仅可以远程实现机器人避障功能,还可以将其应用在山谷、沼泽等危险地带进行采摘作业,实现其非凡的使用价值。

基于视觉特征库的交互控制方法

针对现有人机交互控制方法在不同应用场景需要采用不同算法,各种样本库训练费时等情况,提出以凸包缺陷面积和来刻画手掌,不需要添加额外样本库,总结出了以物距、手势特征周长、手势特征面积、凸包缺陷三角形面积和等因素为基础的手势特征库,设计出了手势控制的整套算法,并以幻灯片控制为例,使用VS2008和OpenCV完成了交互控制的软件开发;实验结果表明该方法虚拟交互时间在1 ms内,能很好的识别出2种特定的动态手势,解析出多种控制命令,可用于交互系统控制。

OpenCV在体感遥控中的应用

利用Intelnet跨平台计算机视觉库OpenCV和微软Visual Studio 2010构建体感遥控器的软件开发平台。以OpenCV为基础,结合自定义算法和函数,开发利用人体手势识别来控制不同设备的体感遥控器。该设计采用基于codebook背景模型的运动检测方法对手势定位,以改进的CamShift算法和Kalman滤波器进行手势跟踪,进而对手势分割和识别。研究利用单目摄像头下对单手手势识别,以红外线和无线通信方式对目标区域和设备进行非接触控制。给出软件设计的基本思想,用OpenCV实现算法的具体过程和关键代码。该方案目标跟踪快、检测效率高、实时性好,为下一步在嵌入式系统中实施此方案打下一定基础。

基于ARM的嵌入式手姿态跟踪设备控制系统研究

人机交互过程中获得人的动作是一个关键,为了实现手部姿态的实时跟踪控制,分析了手形及手部的运动形式和特点,研究设计了一种基于嵌入式系统的手部姿态跟踪控制系统。该系统采用ARM9架构和Linux实时操作系统作为开发平台,设计开发了系统的应用程序,通过动态手势识别算法,实现了连续手姿态的识别。所研究实现的手部姿态跟踪控制系统能够较好地实现手姿态的跟踪,具有较高的实时性。

压阻式柔性应变传感纤维的手指姿态识别装置

针对传统应变传感器柔顺性受限、与织物结合性差的问题,设计了一套可穿戴织物手指姿态识别装置。该装置的传感器为弹性硅橡胶和多壁碳纳米管复合的压阻式芯-鞘结构传感纤维柔性应变传感器,采用同轴湿法纺丝工艺制备,具有很好的可编织性,拉伸应变可达300%,响应时间少于200 ms,可将其编织到织物手套的手指关节处,用于感知手指姿态;该装置的嵌入式控制系统采用基于STM32F103C8T6主控芯片的单片机,用于采集处理手指姿态传感信号;该装置的识别显示系统采用发光二极管、无源蜂鸣器和有机发光显示屏等功能器件,用于手指姿态的识别显示。实验结果表明,与现有手指姿态识别装置相比,该装置将柔性应变传感纤维、常规织物、单片机和功能器件集成在一起,具有舒适的可穿戴性,实现了呼吸灯、数字音乐和手语识别等移动便携式控制应用。

基于手势识别的机器人控制系统

研究了一种基于人体手势识别的机器人控制系统.首先,利用图像识别技术,通过YCr Cb皮肤颜色模型提取手掌并分析指尖和手心的相关信息;其次,利用帧差法对手掌运动趋势和简单的手势信息进行识别;最后,通过无线蓝牙串口将识别出来的手势信号发送给机器人,以达到手势控制机器人的目的.系统是在VS2010下利用Open CV计算机视觉库进行编译完成的,实现了通过简单的手势信息控制机器人的目的,从而摆脱了人机交互时必须依靠物理接触的限制.实验结果表明,该系统可以实现对机器人前进、后退、左转、右转、停止、加速的实时控制,对手势信息的识别率在90%以上.对进一步探索机器学习、自主识别等相关领域有着较高的参考价值.

Design and Myoelectric Control of an Anthropomorphic Prosthetic Hand

This paper presents an anthropomorphic prosthetic hand using flexure hinges, which is controlled by the surface electromyography （sEMG） signals from 2 electrodes only. The prosthetic hand has compact structure with 5 fingers and 4 Degree of Freedoms （DoFs） driven by 4 independent actuators. Helical springs are used as elastic joints and the joints of each finger are coupled by tendons. The myoelectric control system which can classify 8 prehensile hand gestures is built. Pattern recognition is employed where Mean Absolute Value （MAV）, Variance （VAR）, the fourth-order Autoregressive （AR） coefficient and Sample Entropy （SE） are chosen as the optimal feature set and Linear Discriminant Analysis （LDA） is utilized to reduce the dimension. A decision of hand gestures is generated by LDA classifier after the current projected feature set and the previous one are ＂pre-smoothed＂, and then the final decision is obtained when the current decision and previous decisions are ＂post-smoothed＂ from the decisions flow. The prosthetic hand can perform prehensile postures for activities of daily living and carry objects under the control of EMG signals.

面向嵌入式平台的轻量化神经网络手势识别方法

针对传统基于图像分割和特征提取的手势识别算法在复杂背景下识别准确率低、灵活性差的问题,基于目标检测神经网络的手势识别算法可以有效提高复杂环境下手势识别的准确性。受嵌入式处理器体积和功耗的限制,常用的目标检测神经网络在嵌入式上的识别速度较低,不能满足实时手势识别的要求。在SSD目标检测的基础上对其进行优化,使用MobileNetv3网络实现特征提取,目标检测方面则是使用SSD-lite结构,其使用深度可分离卷积替代普通卷积,实现了轻量化MobileNetv3-SSDLite手势识别算法的设计。针对手势识别的要求,制作了包含不同手势的数据集,利用它在服务器上完成了模型的训练。为了满足嵌入式的算力限制,通过模型的量化压缩将float64的网络参数量化为int8,并压缩网络结构,提高网络在嵌入式上的推理速度,实现基于嵌入式的手势识别。实验结果表明,基于嵌入式的MobileNetv3-SSDLite手势识别算法可以达到平均准确率99.61%,且识别速度达到每秒50帧以上,满足实时手势识别的要求。

基于HMM算法的Kinect工作台控制系统研究

手势识别是一种自然直接、简单丰富的人机交互方式,手势识别在日常生活中将会越来越多的被应用。本课题设计了一种基于Kinect设备的工作台控制系统。系统利用Kinect设备采集用户的手势信息,利用HMM算法实现训练数据与存储的手势数据进行比较,最后输出评价结果。得到的手势识别的结果经过ARM9处理器控制工作台的移动。本课题最终实现以较高的识别率控制了工作台的左移、右移、上移和下移。本课题创新地将手势识别应用于可改造的工作台,系统具有使用方便,成本低的特点,还可将工作台系统根据需求改造成物流分发小车或特殊人群用小车,可实现多种功能。

基于手势与五官状态识别的航空多媒体控制系统

为缓长途客机上旅客的疲劳、增加操作的趣味性,设计开发基于手势与五官状态识别的航空娱乐多媒体控制系统。交互系统由手势轨迹识别、手型识别、五官状态识别三部分组成,分别控制多媒体的菜单与游戏任务。系统硬件由普通计算机和摄像头组成,在MATLAB环境下完成软件程序。实验结果表明,系统完全脱离键盘与鼠标,利用手势、表情自由控制多媒体与游戏。手势及表情识别准确率超过98%,平均处理时间0.19秒/帧,保证系统具有较高准确性的同时可以达到实时控制的效果。