基于文字识别与页面布局的APP控件识别算法

APP Control Recognition Algorithm Based on Text Recognition and Page Layout

在移动应用的开发周期中,使用图形用户界面(GUI)自动化测试技术可提高APP的开发测试效率。针对基于录制回放的GUI自动化测试方法存在不适用黑盒测试场景或识别正确率较低等不足,提出基于图像文字识别和应用页面布局的控件识别算法。根据移动应用的页面布局结构和手势操作坐标精确识别控件,采用控件树遍历的方式解决遍历过程中的页面布局重叠问题。分别在Android和iOS2个平台进行实验,结果表明,与基于坐标和源代码的控件识别算法相比,该控件识别算法具有更高的正确率。

In the development cycle of mobile applications,the use of Graphical User Interface(GUI) automated testing technology can improve the efficiency of the development and test of APP.Still,the GUI automated testing method based on recording playback is not applicable to black box test scenarios,and its recognition accuracy rate is not high.To deal with the advantages,this paper proposes a control recognition algorithm based on text recognition in images and application page layout.Based on the page layout structure of mobile applications and the accurate recognition control for the coordinates of gesture operation,the overlapping problem of page layout in the process of traversing is solved by traversing the control tree.Experiments are carried out on Android and iOS platforms respectively,and results show that the control recognition algorithm has higher accuracy than the control recognition algorithm based on coordinates and source codes.