C-CORE:Clustering by Code Representation to Prioritize Test Cases in Compiler Testing

Edge devices,due to their limited computational and storage resources,often require the use of compilers for program optimization.Therefore,ensuring the security and reliability of these compilers is of paramount importance in the emerging field of edge AI.One widely used testing method for this purpose is fuzz testing,which detects bugs by inputting random test cases into the target program.However,this process consumes significant time and resources.To improve the efficiency of compiler fuzz testing,it is common practice to utilize test case prioritization techniques.Some researchers use machine learning to predict the code coverage of test cases,aiming to maximize the test capability for the target compiler by increasing the overall predicted coverage of the test cases.Nevertheless,these methods can only forecast the code coverage of the compiler at a specific optimization level,potentially missing many optimization-related bugs.In this paper,we introduce C-CORE(short for Clustering by Code Representation),the first framework to prioritize test cases according to their code representations,which are derived directly from the source codes.This approach avoids being limited to specific compiler states and extends to a broader range of compiler bugs.Specifically,we first train a scaled pre-trained programming language model to capture as many common features as possible from the test cases generated by a fuzzer.Using this pre-trained model,we then train two downstream models:one for predicting the likelihood of triggering a bug and another for identifying code representations associated with bugs.Subsequently,we cluster the test cases according to their code representations and select the highest-scoring test case from each cluster as the high-quality test case.This reduction in redundant testing cases leads to time savings.Comprehensive evaluation results reveal that code representations are better at distinguishing test capabilities,and C-CORE significantly enhances testing efficiency.Across four datasets,C-CORE increases the average of the percentage of faults detected(APFD)value by 0.16 to 0.31 and reduces test time by over 50% in 46% of cases.When compared to the best results from approaches using predicted code coverage,C-CORE improves the APFD value by 1.1% to 12.3% and achieves an overall time-saving of 159.1%.

AN ALGORITHM FOR AUTOMATICALLY GENERATING BLACK-BOX TEST CASES

Selection of test cases plays a key role in improving testing efficiency. Black-box testing is an important way of testing, and its validity lies on the selection of test cases in some sense. A reasonable and effective method about the selection and generation of test cases is urgently needed. This letter first introduces some usualmethods on black-box test case generation,then proposes a new algorithm based on interface parameters and discusses its properties, finally shows the effectiveness of the algorithm.

Reusing Test Cases Based on the Function Point

Reusing test cases from existing test case library is quite common in the software testing field. Testing practice tells us that there is a strong relationship between the granularity of a function unit under testing and that of the test case. A function unit with small granularity usually results in the test cases with the same small granularity. Therefore a test case defined as the function point,i. e.,the smallest size function unit,was provided for the first time.Though test cases with smaller granularity usually have better reusability,the cost of accurately reusing and integrating such test cases is also higher. In order to balance the test case reusability and the cost of test case reuse,a novel test case reuse model based on the function point was proposed in this paper. In this model,a reusable test case for specification-based testing was defined and some reuse strategies and three formal reuse methods were given. Finally,the complete automatic software process was realized by a reusing generation tool. The new method has improved reuse accuracy,while greatly enhances the software productivity.

SMINER:Detecting Unrestricted and Misimplemented Behaviors of Software Systems Based on Unit Test Cases

Despite the advances in automated vulnerability detection approaches,security vulnerabilities caused by design flaws in software systems are continuously appearing in real-world systems.Such security design flaws can bring unrestricted and misimplemented behaviors of a system and can lead to fatal vulnerabilities such as remote code execution or sensitive data leakage.Therefore,it is an essential task to discover unrestricted and misimplemented behaviors of a system.However,it is a daunting task for security experts to discover such vulnerabilities in advance because it is timeconsuming and error-prone to analyze the whole code in detail.Also,most of the existing vulnerability detection approaches still focus on detecting memory corruption bugs because these bugs are the dominant root cause of software vulnerabilities.This paper proposes SMINER,a novel approach that discovers vulnerabilities caused by unrestricted and misimplemented behaviors.SMINER first collects unit test cases for the target system from the official repository.Next,preprocess the collected code fragments.SMINER uses pre-processed data to show the security policies that can occur on the target system and creates a test case for security policy testing.To demonstrate the effectiveness of SMINER,this paper evaluates SMINER against Robot Operating System(ROS),a real-world system used for intelligent robots in Amazon and controlling satellites in National Aeronautics and Space Administration(NASA).From the evaluation,we discovered two real-world vulnerabilities in ROS.

IPSETFUL： an iterative process of selecting test cases for effective fault localization by exploring concept lattice of program spectra

Fault localization is an important and challeng- ing task during software testing. Among techniques studied in this field, program spectrum based fault localization is a promising approach. To perform spectrum based fault local- ization, a set of test oracles should be provided, and the ef- fectiveness of fault localization depends highly on the quality of test oracles. Moreover, their effectiveness is usually af- fected when multiple simultaneous faults are present. Faced with multiple faults it is difficult for developers to determine when to stop the fault localization process. To address these issues, we propose an iterative fauk localization process, i.e., an iterative process of selecting test cases for effective fault localization （IPSETFUL）, to identify as many faults as pos- sible in the program until the stopping criterion is satisfied. It is performed based on a concept lattice of program spec- trum （CLPS） proposed in our previous work. Based on the labeling approach of CLPS, program statements are catego- rized as dangerous statements, safe statements, and sensitive statements. To identify the faults, developers need to check the dangerous statements. Meantime, developers need to se- lect a set of test cases covering the dangerous or sensitive statements from the original test suite, and a new CLPS is generated for the next iteration. The same process is pro- ceeded in the same way. This iterative process ends until there are no failing tests in the test suite and all statements on the CLPS become safe statements. We conduct an empirical study on several subject programs, and the results show that IPSETFUL can help identify most of the faults in the program with the given test suite. Moreover, it can save much effort in inspecting unfaulty program statements compared with the existing spectrum based fault localization techniques and the relevant state of the art technique.

Prioritizing Test Cases for Memory Leaks in Android Applications

Mobile applications usually can only access limited amount of memory. Improper use of the memory can cause memory leaks, which may lead to performance slowdowns or even cause applications to be unexpectedly killed. Although a large body of research has been devoted into the memory leak diagnosing techniques after leaks have been discovered, it is still challenging to find out the memory leak phenomena at first. Testing is the most widely used technique for failure discovery. However, traditional testing techniques are not directed for the discovery of memory leaks. They may spend lots of time on testing unlikely leaking executions and therefore can be inefficient. To address the problem, we propose a novel approach to prioritize test cases according to their likelihood to cause memory leaks in a given test suite. It firstly builds a prediction model to determine whether each test can potentially lead to memory leaks based on machine learning on selected code features. Then, for each input test case, we partly run it to get its code features and predict its likelihood to cause leaks. The most suspicious test cases will be suggested to run at first in order to reveal memory leak faults as soon as possible. Experimental evaluation on several Android applications shows that our approach is effective.

Test Case Generation Evaluator for the Implementation of Test Case Generation Algorithms Based on Learning to Rank

In software testing,the quality of test cases is crucial,but manual generation is time-consuming.Various automatic test case generation methods exist,requiring careful selection based on program features.Current evaluation methods compare a limited set of metrics,which does not support a larger number of metrics or consider the relative importance of each metric to the final assessment.To address this,we propose an evaluation tool,the Test Case Generation Evaluator(TCGE),based on the learning to rank(L2R)algorithm.Unlike previous approaches,our method comprehensively evaluates algorithms by considering multiple metrics,resulting in a more reasoned assessment.The main principle of the TCGE is the formation of feature vectors that are of concern by the tester.Through training,the feature vectors are sorted to generate a list,with the order of the methods on the list determined according to their effectiveness on the tested assembly.We implement TCGE using three L2R algorithms:Listnet,LambdaMART,and RFLambdaMART.Evaluation employs a dataset with features of classical test case generation algorithms and three metrics—Normalized Discounted Cumulative Gain(NDCG),Mean Average Precision(MAP),and Mean Reciprocal Rank(MRR).Results demonstrate the TCGE’s superior effectiveness in evaluating test case generation algorithms compared to other methods.Among the three L2R algorithms,RFLambdaMART proves the most effective,achieving an accuracy above 96.5%,surpassing LambdaMART by 2%and Listnet by 1.5%.Consequently,the TCGE framework exhibits significant application value in the evaluation of test case generation algorithms.

Improved metrics for evaluating fault detection efficiency of test suite

By analyzing the average percent of faults detected （APFD） metric and its variant versions, which are widely utilized as metrics to evaluate the fault detection efficiency of the test suite, this paper points out some limitations of the APFD series metrics. These limitations include APFD series metrics having inaccurate physical explanations and being unable to precisely describe the process of fault detection. To avoid the limitations of existing metrics, this paper proposes two improved metrics for evaluating fault detection efficiency of a test suite, including relative-APFD and relative-APFDc. The proposed metrics refer to both the speed of fault detection and the constraint of the testing source. The case study shows that the two proposed metrics can provide much more precise descriptions of the fault detection process and the fault detection efficiency of the test suite.

Method for test case selection and execution of web application regression testing

In order to improve the efficiency of regression testing in web application,the control flow graph and the greedy algorithm are adopted.This paper considers a web page as a basic unit and introduces a test case selection method for web application regression testing based on the control flow graph.This method is safe enough to the test case selection.On the base of features of request sequence in web application,the minimization technique and the priority of test cases are taken into consideration in the process of execution of test cases in regression testing for web application.The improved greedy algorithm is also raised resulting in optimization of execution of test cases.The experiments indicate that the number of test cases which need to be retested is reduced,and the efficiency of execution of test cases is also improved.

Generating test case specifications of web service composition using model checking

Model checking techniques have been widely used in verifying web service compositions to ensure the trustworthi- ness. However, little research has focused on testing web services. Based on the research of model checking techniques~ we propose a model checking based approach for testing web service composition which is described by using the web services choreography description language （WS-CDL）. According to worldwide web consortium （W3C） candidate recommendation, the WS-CDL specification provides a language for characterizing interactions between distinct web services using XML. Since the behaviors of web service composition are asynchronous, distributed, low-coupled and platform independent, we employ the guarded automata （GA） model for specifying the composition described in WS-CDL and using the simple promela interpreter （SPIN） model checker for detecting the collaborations of web services. Test cases can be transformed from counterexamples generated by SPIN using adequacy criteria. In this paper we apply the transition coverage criterion for generating counterex- amples. To illustrate our approach, we set ＂E-commerce service system＂ as an example for demonstrating how test cases can be generated using SPIN for compositions specified in WS-CDL.

Aware Time in Bug Fix—A Novel Automatic Test Case Selection for Prioritization of Version Control

The supreme goal of the Automatic Test case selection techniques is to guarantee systematic coverage, to recognize the usual error forms and to lessen the test of redundancy. It is unfeasible to carry out all the test cases consistently. For this reason, the test cases are picked and prioritize it. The major goal of test case prioritization is to prioritize the test case sequence and finds faults as early as possible to improve the efficiency. Regression testing is used to ensure the validity and the enhancement part of the changed software. In this paper, we propose a new path compression technique (PCUA) for both old version and new version of BPEL dataset. In order to analyze the enhancement part of an application and to find an error in an enhancement part of an application, center of the tree has been calculated. Moreover in the comparative analysis, our proposed PCUA- COT technique is compared with the existing XPFG technique in terms of time consuming and error detection in the path of an enhancement part of BPEL dataset. The experimental results have been shown that our proposed work is better than the existing technique in terms of time consuming and error detection.

Novel technique for cost reduction in mutation testing

Aimed at the problem of expensive costs in mutation testing which has hampered its wide use,a technique of introducing a test case selection into the process of mutation testing is proposed.For each mutant,a fixed number of test cases are selected to constrain the maximum allowable executions so as to reduce useless work.Test case selection largely depends on the degree of mutation.The mutation distance is an index describing the semantic difference between the original program and the mutated program.It represents the percentage of effective test cases in a test set,so it can be used to guide the selection of test cases.The bigger the mutation distance is,the easier it is that the mutant will be killed,so the corresponding number of effective test cases for this mutant is greater.Experimental results suggest that the technique can remarkably reduce execution costs without a significant loss of test effectiveness.

Component system regression testing method based on CTAM

The emphasis of component system regression testing is retesting of the event interaction between updated components and other components in a system.A component system regression testing method based on a new component testing association model （CTAM） is proposed.First,the modification-affected component groups are identified by the impact analysis on CTAM,and each component in this group is assigned with an influence degree.Then,previous test cases are selected according to the influence degree,to generate the minimal regression test suite.Compared with traditional methods,CTAM is derived from the statistic on the interactive events that occurred in previous test executions,and focuses on the complicated relationship between components,which is more applicable to the component system regression testing.

Real-time embedded software testing method based on extended finite state machine

The reliability of real-time embedded software directly determines the reliability of the whole real-time embedded sys- tem, and the effective software testing is an important way to ensure software quality and reliability. Based on the analysis of the characteristics of real-time embedded software, the formal method is introduced into the real-time embedded software testing field and the real-time extended finite state machine （RT-EFSM） model is studied firstly. Then, the time zone division method of real-time embedded system is presented and the definition and description methods of time-constrained transition equivalence class （timeCTEC） are presented. Furthermore, the approaches of the testing sequence and test case generation are put forward. Finally, the proposed method is applied to a typical avionics real- time embedded software testing practice and the examples of the timeCTEC, testing sequences and test cases are given. With the analysis of the testing result, the application verification shows that the proposed method can effectively describe the real-time embedded software state transition characteristics and real-time requirements and play the advantages of the formal methods in accuracy, effectiveness and the automation supporting. Combined with the testing platform, the real-time, closed loop and automated simulation testing for real-time embedded software can be realized effectively.

Software Test Case Generation with Adequacy Analysis on Scenario-Based Testing

This paper studies the software scenario testing, which is commonly used in black-box testing at present. In the paper, the workflow model based on task-driven, which is very common in scenario testing, is analyzed. According to test adequacy criteria in scenario testing, the model is designed to correspond test cases in the light of logic block(LB). The final test cases that conform to the test adequacy criteria can be obtained through test case combination and test case reduction. In the last part of the paper, example of actual workflow is to design the efficient test case. Therefore the method is proved to be effective.

Implementation of Hybrid Particle Swarm Optimization for Optimized Regression Testing

Software test case optimization improves the efficiency of the software by proper structure and reduces the fault in the software.The existing research applies various optimization methods such as Genetic Algorithm,Crow Search Algorithm,Ant Colony Optimization,etc.,for test case optimization.The existing methods have limitations of lower efficiency in fault diagnosis,higher computa-tional time,and high memory requirement.The existing methods have lower effi-ciency in software test case optimization when the number of test cases is high.This research proposes the Tournament Winner Genetic Algorithm(TW-GA)method to improve the efficiency of software test case optimization.Hospital Information System(HIS)software was used to evaluate TW-GA model perfor-mance in test case optimization.The tournament Winner in the proposed method selects the instances with the best fitness values and increases the exploitation of the search to find the optimal solution.The TW-GA method has higher exploita-tion that helps to find the mutant and equivalent mutation that significantly increases fault diagnosis in the software.The TW-GA method discards the infor-mation with a lower fitness value that reduces the computational time and mem-ory requirement.The TW-GA method requires 5.47 s and the MOCSFO method requires 30 s for software test case optimization.

Automated Test Case Generation from Requirements: A Systematic Literature Review

Software testing is an important and cost intensive activity in software development.The major contribution in cost is due to test case generations.Requirement-based testing is an approach in which test cases are derivative from requirements without considering the implementation’s internal structure.Requirement-based testing includes functional and nonfunctional requirements.The objective of this study is to explore the approaches that generate test cases from requirements.A systematic literature review based on two research questions and extensive quality assessment criteria includes studies.The study identies 30 primary studies from 410 studies spanned from 2000 to 2018.The review’s nding shows that 53%of journal papers,42%of conference papers,and 5%of book chapters’address requirementsbased testing.Most of the studies use UML,activity,and use case diagrams for test case generation from requirements.One of the signicant lessons learned is that most software testing errors are traced back to errors in natural language requirements.A substantial amount of work focuses on UML diagrams for test case generations,which cannot capture all the system’s developed attributes.Furthermore,there is a lack of UML-based models that can generate test cases from natural language requirements by rening them in context.Coverage criteria indicate how efciently the testing has been performed 12.37%of studies use requirements coverage,20%of studies cover path coverage,and 17%study basic coverage.

A hybrid algorithm based on ILP and genetic algorithm for time-aware test case prioritization

To solve the problem of time-awarc test case prioritization,a hybrid algorithm composed of integer linear programming and the genetic algorithm(ILP-GA)is proposed.First,the test case suite which cm maximize the number of covered program entities a d satisfy time constraints is selected by integer linea progamming.Secondly,the individual is encoded according to the cover matrices of entities,and the coverage rate of program entities is used as the fitness function and the genetic algorithm is used to prioritize the selected test cases.Five typical open source projects are selected as benchmark programs.Branch and method are selected as program entities,and time constraint percentages a e 25%and 75%.The experimental results show that the ILP-GA convergence has faster speed and better stability than ILP-additional and IP-total in most cases,which contributes to the detection of software defects as early as possible and reduces the software testing costs.

Modified Harris Hawks Optimization Based Test Case Prioritization for Software Testing

Generally,software testing is considered as a proficient technique to achieve improvement in quality and reliability of the software.But,the quality of test cases has a considerable influence on fault revealing capability of software testing activity.Test Case Prioritization(TCP)remains a challenging issue since prioritizing test cases is unsatisfactory in terms of Average Percentage of Faults Detected(APFD)and time spent upon execution results.TCP ismainly intended to design a collection of test cases that can accomplish early optimization using preferred characteristics.The studies conducted earlier focused on prioritizing the available test cases in accelerating fault detection rate during software testing.In this aspect,the current study designs aModified Harris Hawks Optimization based TCP(MHHO-TCP)technique for software testing.The aim of the proposed MHHO-TCP technique is to maximize APFD and minimize the overall execution time.In addition,MHHO algorithm is designed to boost the exploration and exploitation abilities of conventional HHO algorithm.In order to validate the enhanced efficiency of MHHO-TCP technique,a wide range of simulations was conducted on different benchmark programs and the results were examined under several aspects.The experimental outcomes highlight the improved efficiency of MHHO-TCP technique over recent approaches under different measures.