基于目标检测的图形用户界面控件识别方法

Graphical user interface widget extraction based on object detection

传统机器人流程自动化(Robot Process Automation,RPA)主要使用操作系统和应用程序提供的接口获取图形用户界面(Graphical User Interface,GUI)控件,受操作系统和应用程序接口限制,但一些系统,如Linux,不提供获取控件信息的接口.提出一种基于神经网络对图形界面控件进行识别的方法,利用目标检测模型提取图形用户界面控件特征,在不使用操作系统接口的前提下识别图形用户界面内控件类别和几何信息,减少RPA对于系统与程序接口的依赖.同时,针对桌面端图形用户界面数据集缺失的问题,提出一种针对RPA领域桌面端图形用户界面目标检测数据集的生成方法.在该图形用户界面数据集上使用各类目标检测模型进行测试,结果显示,常用目标检测模型在识别用户界面控件的类别和几何信息时均能获得92%以上的准确率.

Traditional RPA(Robot Process Automation)extracts GUI(Graphical User Interface)widget category and location information through the interface provided by operating system or application programs.However,the application scope of RPA is limited by operating system and application program interfaces.For example,some operation systems,such as Linux,have inadequate extraction interface access to widget category and location information.This paper proposes a method to extract GUI widget features based on neural network and extract the widget information by object detection.Object detection can fetch and understand the widget in desktop platform feature then further extract the widget category and location information without using any operation system or application program interface,which reduce the dependence of RPA on the operation system and application program interface.Meanwhile,aiming at the lack of the user interface database in desktop platform,an auto database generation method for target detection in RPA field is also proposed.The object detection models are tested on this user interface database in desktop platform.Experimental results show that the object detection models can obtain more than 92%mAP in extracting widget category and location information in desktop platform.