面向GUI测试的可触控控件检测方法

GUI-Oriented Testing Touchable Widget Detection Method

为了解决基于深度学习的GUI元素识别方法表现不佳以及无法判断是否可触控的问题,提高GUI测试的效率与覆盖率,提出有效的面向GUI测试的可触控控件训练与检测方法。首先定义可触控控件的检测类别,用于直接检测具备可触发属性的控件;考虑到UI页面存在堆叠元素,对数据集中不可见的控件进行过滤,并剔除视图层次结构与屏幕截图不同步的数据;通过分析安卓机制将UI页面中可触控控件进行了标记。最后基于YOLO v5s训练获得一个轻量级训练模型。结果表明,提出的训练及检测方法优于现有深度学习方法和经典方法,其F1达到了82%,在GUI测试中具有良好的使用价值。

In order to solve the problem of poor performance of GUI element recognition method based on deep learning and inability to judge whether it is touchable,and improve the efficiency and coverage of GUI testing,an effective touchable control training and detection method for GUI testing is proposed.First define the detection category of touchable controls to directly detect controls with triggerable properties;considering the existence of stacked elements on the UI page,filter the invisible controls in the dataset,and remove the data whose view hierarchy is out of sync with the screenshot;mark the touchable controls in the UI page by analyzing the Android mechanism.Finally,a lightweight training model is obtained based on YOLO v5s training.The results show that the proposed training and detection method outperforms the existing deep learning methods and classical methods,and its F1 reaches 82%,which has good use value in GUI testing.