Test Case Generation Evaluator for the Implementation of Test Case Generation Algorithms Based on Learning to Rank

In software testing,the quality of test cases is crucial,but manual generation is time-consuming.Various automatic test case generation methods exist,requiring careful selection based on program features.Current evaluation methods compare a limited set of metrics,which does not support a larger number of metrics or consider the relative importance of each metric to the final assessment.To address this,we propose an evaluation tool,the Test Case Generation Evaluator(TCGE),based on the learning to rank(L2R)algorithm.Unlike previous approaches,our method comprehensively evaluates algorithms by considering multiple metrics,resulting in a more reasoned assessment.The main principle of the TCGE is the formation of feature vectors that are of concern by the tester.Through training,the feature vectors are sorted to generate a list,with the order of the methods on the list determined according to their effectiveness on the tested assembly.We implement TCGE using three L2R algorithms:Listnet,LambdaMART,and RFLambdaMART.Evaluation employs a dataset with features of classical test case generation algorithms and three metrics—Normalized Discounted Cumulative Gain(NDCG),Mean Average Precision(MAP),and Mean Reciprocal Rank(MRR).Results demonstrate the TCGE’s superior effectiveness in evaluating test case generation algorithms compared to other methods.Among the three L2R algorithms,RFLambdaMART proves the most effective,achieving an accuracy above 96.5%,surpassing LambdaMART by 2%and Listnet by 1.5%.Consequently,the TCGE framework exhibits significant application value in the evaluation of test case generation algorithms.

SMINER:Detecting Unrestricted and Misimplemented Behaviors of Software Systems Based on Unit Test Cases

Despite the advances in automated vulnerability detection approaches,security vulnerabilities caused by design flaws in software systems are continuously appearing in real-world systems.Such security design flaws can bring unrestricted and misimplemented behaviors of a system and can lead to fatal vulnerabilities such as remote code execution or sensitive data leakage.Therefore,it is an essential task to discover unrestricted and misimplemented behaviors of a system.However,it is a daunting task for security experts to discover such vulnerabilities in advance because it is timeconsuming and error-prone to analyze the whole code in detail.Also,most of the existing vulnerability detection approaches still focus on detecting memory corruption bugs because these bugs are the dominant root cause of software vulnerabilities.This paper proposes SMINER,a novel approach that discovers vulnerabilities caused by unrestricted and misimplemented behaviors.SMINER first collects unit test cases for the target system from the official repository.Next,preprocess the collected code fragments.SMINER uses pre-processed data to show the security policies that can occur on the target system and creates a test case for security policy testing.To demonstrate the effectiveness of SMINER,this paper evaluates SMINER against Robot Operating System(ROS),a real-world system used for intelligent robots in Amazon and controlling satellites in National Aeronautics and Space Administration(NASA).From the evaluation,we discovered two real-world vulnerabilities in ROS.

Automated Test Case Generation from Requirements: A Systematic Literature Review

Software testing is an important and cost intensive activity in software development.The major contribution in cost is due to test case generations.Requirement-based testing is an approach in which test cases are derivative from requirements without considering the implementation’s internal structure.Requirement-based testing includes functional and nonfunctional requirements.The objective of this study is to explore the approaches that generate test cases from requirements.A systematic literature review based on two research questions and extensive quality assessment criteria includes studies.The study identies 30 primary studies from 410 studies spanned from 2000 to 2018.The review’s nding shows that 53%of journal papers,42%of conference papers,and 5%of book chapters’address requirementsbased testing.Most of the studies use UML,activity,and use case diagrams for test case generation from requirements.One of the signicant lessons learned is that most software testing errors are traced back to errors in natural language requirements.A substantial amount of work focuses on UML diagrams for test case generations,which cannot capture all the system’s developed attributes.Furthermore,there is a lack of UML-based models that can generate test cases from natural language requirements by rening them in context.Coverage criteria indicate how efciently the testing has been performed 12.37%of studies use requirements coverage,20%of studies cover path coverage,and 17%study basic coverage.

Class hierarchical test case generation algorithm based on expanded EMDPN model

A new model of event and message driven Petri network(EMDPN) based on the characteristic of class interaction for messages passing between two objects was extended. Using EMDPN interaction graph, a class hierarchical test-case generation algorithm with cooperated paths (copaths) was proposed, which can be used to solve the problems resulting from the class inheritance mechanism encountered in object-oriented software testing such as oracle, message transfer errors, and unreachable statement. Finally, the testing sufficiency was analyzed with the ordered sequence testing criterion(OSC). The results indicate that the test cases stemmed from newly proposed automatic algorithm of copaths generation satisfies synchronization message sequences testing criteria, therefore the proposed new algorithm of copaths generation has a good coverage rate.

An EFSM-Based Test Data Generation Approach in Model-Based Testing

Testing is an integral part of software development.Current fastpaced system developments have rendered traditional testing techniques obsolete.Therefore,automated testing techniques are needed to adapt to such system developments speed.Model-based testing(MBT)is a technique that uses system models to generate and execute test cases automatically.It was identified that the test data generation(TDG)in many existing model-based test case generation(MB-TCG)approaches were still manual.An automatic and effective TDG can further reduce testing cost while detecting more faults.This study proposes an automated TDG approach in MB-TCG using the extended finite state machine model(EFSM).The proposed approach integrates MBT with combinatorial testing.The information available in an EFSM model and the boundary value analysis strategy are used to automate the domain input classifications which were done manually by the existing approach.The results showed that the proposed approach was able to detect 6.62 percent more faults than the conventionalMB-TCG but at the same time generated 43 more tests.The proposed approach effectively detects faults,but a further treatment to the generated tests such as test case prioritization should be done to increase the effectiveness and efficiency of testing.

Multi-Objective Modified Particle Swarm Optimization for Test Suite Reduction(MOMPSO)

Software testing plays a pivotal role in entire software development lifecycle.It provides researchers with extensive opportunities to develop novel methods for the optimized and cost-effective test suite Although implementation of such a cost-effective test suite with regression testing is being under exploration still it contains lot of challenges and flaws while incorporating with any of the new regression testing algorithm due to irrelevant test cases in the test suite which are not required.These kinds of irrelevant test cases might create certain challenges such as code-coverage in the test suite,fault-tolerance,defects due to uncovered-statements and overall-performance at the time of execution.With this objective,the proposed a new Modified Particle Swarm optimization used for multi-objective test suite optimization.The experiment results involving six subject programs show that MOMPSO method can outer perform with respect to both reduction rate(90.78%to 100%)and failure detection rate(44.56%to 55.01%).Results proved MOMPSO outperformed the other stated algorithms.

Research of testing method based on UML statecharts

Unified modeling language （UML） is a powerful graphical modeling language with intuitional meaning. It provides various diagrams to depict system characteristics and complex environment from different viewpoints and different application layers. UML-based software development and modeling environments have been widely accepted in industry, including areas in which safety is an important issue such as spaceflight, defense, automobile, etc. To ensure and improve software quality becomes a main concern in the field. As one of the key techniques for software quality, software testing can effectively detect system faults. UML based software testing based is an important research direction in software engineering. The key to software testing is the generation of test cases. This dissertation studies an approach to generating test cases from UML statecharts.

Review of Software Model-Checking Techniques for Dealing with Error Detection in Program Codes

Debugging software code has been a challenge for software developers since the early days of computer programming. A simple need, because the world is run by software. So perhaps the biggest engineering challenge is finding ways to make software more reliable. This review provides an overview of techniques developed over time in the field of software model checking to solve the problem of detecting errors in program code. In addition, the challenges posed by this technology are discussed and ways to mitigate them in future research and applications are proposed. A comprehensive examination of the various model verification methods used to detect program code errors is intended to lay the foundation for future research in this area.

Intelligent test case generation based on branch and bound

Path-oriented test case generation is in essence a constraint satisfaction problem （CSP） solved by search strategies, among which backtracking algorithms are widely used. In this article, the backtracking algorithm branch and bound （BB） is introduced to generate path-oriented test cases automatically. A model based on state space search is proposed to construct the search tree dynamically. The BB is optimized from two perspectives. Variable permutation with a heuristic rule to break ties is adopted for the branching operation, and interval computation with analysis on the monotony of branching conditions is utilized for the bounding operation. Empirical experiments show that the proposed method performs well with linear complexity, and reaches 100% coverage on some benchmark programs with an advantage over some static and dynamic algorithms.

Search-Based Software Test Data Generation for Path Coverage Based on a Feedback-Directed Mechanism

Automatically generating test cases by evolutionary algorithms to satisfy the path coverage criterion has attracted much research attention in software testing.In the context of generating test cases to cover many target paths,the efficiency of existing methods needs to be further improved when infeasible or difficult paths exist in the program under test.This is because a significant amount of the search budget(i.e.,time allocated for the search to run)is consumed when computing fitness evaluations of individuals on infeasible or difficult paths.In this work,we present a feedback-directed mechanism that temporarily removes groups of paths from the target paths when no improvement is observed for these paths in subsequent generations.To fulfill this task,our strategy first organizes paths into groups.Then,in each generation,the objective scores of each individual for all paths in each group are summed up.For each group,the lowest value of the summed up objective scores among all individuals is assigned as the best aggregated score for a group.A group is removed when no improvement is observed in its best aggregated score over the last two generations.The experimental results show that the proposed approach can significantly improve path coverage rates for programs under test with infeasible or difficult paths in case of a limited search budget.In particular,the feedback-directed mechanism reduces wasting the search budget on infeasible paths or on difficult target paths that require many fitness evaluations before getting an improvement.

Worst-Input Mutation Approach to Web Services Vulnerability Testing Based on SOAP Messages

The growing popularity and application of Web services have led to increased attention regarding the vulnerability of software based on these services. Vulnerability testing examines the trustworthiness and reduces the security risks of software systems. This paper proposes a worst-input mutation approach for testing Web service vulnerability based on Simple Object Access Protocol （SOAP） messages. Based on characteristics of SOAP messages, the proposed approach uses the farthest neighbor concept to guide generation of the test suite. The corresponding automatic test case generation algorithm, namely, the Test Case generation based on the Farthest Neighbor （TCFN）, is also presented. The method involves partitioning the input domain into sub-domains according to the number and type of SOAP message parameters in the TCFN, selecting the candidate test case whose distance is the farthest from all executed test cases, and applying it to test the Web service. We also implement and describe a prototype Web service vulnerability testing tool. The tool was applied to the testing of Web services on the Internet. The experimental results show that the proposed approach can find more vulnerability faults than other related approaches.