An Empirical Study on Automated Test Generation Tools for Java:Effectiveness and Challenges

Automated test generation tools enable test automation and further alleviate the low efficiency caused by writing hand-crafted test cases.However,existing automated tools are not mature enough to be widely used by software testing groups.This paper conducts an empirical study on the state-of-the-art automated tools for Java,i.e.,EvoSuite,Randoop,JDoop,JTeXpert,T3,and Tardis.We design a test workflow to facilitate the process,which can automatically run tools for test generation,collect data,and evaluate various metrics.Furthermore,we conduct empirical analysis on these six tools and their related techniques from different aspects,i.e.,code coverage,mutation score,test suite size,readability,and real fault detection ability.We discuss about the benefits and drawbacks of hybrid techniques based on experimental results.Besides,we introduce our experience in setting up and executing these tools,and summarize their usability and user-friendliness.Finally,we give some insights into automated tools in terms of test suite readability improvement,meaningful assertion generation,test suite reduction for random testing tools,and symbolic execution integration.

Search-Based Software Test Data Generation for Path Coverage Based on a Feedback-Directed Mechanism

Automatically generating test cases by evolutionary algorithms to satisfy the path coverage criterion has attracted much research attention in software testing.In the context of generating test cases to cover many target paths,the efficiency of existing methods needs to be further improved when infeasible or difficult paths exist in the program under test.This is because a significant amount of the search budget(i.e.,time allocated for the search to run)is consumed when computing fitness evaluations of individuals on infeasible or difficult paths.In this work,we present a feedback-directed mechanism that temporarily removes groups of paths from the target paths when no improvement is observed for these paths in subsequent generations.To fulfill this task,our strategy first organizes paths into groups.Then,in each generation,the objective scores of each individual for all paths in each group are summed up.For each group,the lowest value of the summed up objective scores among all individuals is assigned as the best aggregated score for a group.A group is removed when no improvement is observed in its best aggregated score over the last two generations.The experimental results show that the proposed approach can significantly improve path coverage rates for programs under test with infeasible or difficult paths in case of a limited search budget.In particular,the feedback-directed mechanism reduces wasting the search budget on infeasible paths or on difficult target paths that require many fitness evaluations before getting an improvement.