Computer Engineering / Bilgisayar Mühendisliği
Permanent URI for this collectionhttps://hdl.handle.net/11147/10
Browse
18 results
Search Results
Article Citation - Scopus: 1Model-Based Ideal Testing of Hardware Description Language (hdl) Programs(Springer, 2021) Kılınççeker, Onur; Türk, Ercüment; Belli, Fevzi; Challenger, MoharramAn ideal test is supposed to show not only the presence of bugs but also their absence. Based on the Fundamental Test Theory of Goodenough and Gerhart (IEEE Trans Softw Eng SE-1(2):156–173, 1975), this paper proposes an approach to model-based ideal testing of hardware description language (HDL) programs based on their behavioral model. Test sequences are generated from both original (fault-free) and mutant (faulty) models in the sense of positive and negative testing, forming a holistic test view. These test sequences are then executed on original (fault-free) and mutant (faulty) HDL programs, in the sense of mutation testing. Using the techniques known from automata theory, test selection criteria are developed and formally show that they fulfill the major requirements of Fundamental Test Theory, that is, reliability and validity. The current paper comprises a preparation step (consisting of the sub-steps model construction, model mutation, model conversion, and test generation) and a composition step (consisting of the sub-steps pre-selection and construction of Ideal test suites). All the steps are supported by a toolchain that is already implemented and is available online. To critically validate the proposed approach, three case studies (a sequence detector, a traffic light controller, and a RISC-V processor) are used and the strengths and weaknesses of the approach are discussed. The proposed approach achieves the highest mutation score in positive and negative testing for all case studies in comparison with two existing methods (regular expression-based test generation and context-based random test generation), using four different techniques.Article Citation - WoS: 3Citation - Scopus: 6Model-Based Ideal Testing of Gui Programs-Approach and Case Studies(IEEE-Inst Electrical Electronics Engineers inc, 2021) Kilincceker, Onur; Silistre, Alper; Belli, Fevzi; Challenger, MoharramTraditionally, software testing is aimed at showing the presence of faults. This paper proposes a novel approach to testing graphical user interfaces (GUI) for showing both the presence and absence of faults in the sense of ideal testing. The approach uses a positive testing concept to show that the GUI under consideration (GUC) does what the user expects; to the contrary, the negative testing concept shows that the GUC does not do anything that the user does not expect, building a holistic view. The first step of the approach models the GUC by a finite state machine (FSM) that enables the model-based generation of test cases. This is always possible as the GUIs are considered as strictly sequential processes. The next step converts the FSM to an equivalent regular expression (RE) that will be analyzed first to construct test selection criteria for excluding redundant test cases and construct test coverage criteria for terminating the positive test process. Both criteria enable us to assess the adequacy and efficiency of the positive tests performed. The negative tests will be realized by systematically mutating the FSM to model faults, the absence of which are to be shown. Those mutant FSMs will be handled and assessed in the same way as in positive testing. Two case studies illustrate and validate the approach; the experiments' results will be analyzed to discuss the pros and cons of the techniques introduced.Article Citation - WoS: 9Citation - Scopus: 11Test Input Generation From Cause-Effect Graphs(Springer, 2021) Kavzak Ufuktepe, Deniz; Ayav, Tolga; Belli, FevziCause-effect graphing is a well-known requirement-based and systematic testing method with a heuristic approach. Since it was introduced by Myers in 1979, there have not been any sufficiently comprehensive studies to generate test inputs from these graphs. However, there exist several methods for test input generation from Boolean expressions. Cause-effect graphs can be more convenient for a wide variety of users compared to Boolean expressions. Moreover, they can be used to enforce common constraints and rules on the system variables of different expressions of the system. This study proposes a new mutant-based test input generation method, Spectral Testing for Boolean specification models based on spectral analysis of Boolean expressions using mutations of the original expression. Unlike Myers' method, Spectral Testing is an algorithmic and deterministic method, in which we model the possible faults systematically. Furthermore, the conversion of cause-effect graphs between Boolean expressions is explored so that the existing test input generation methods for Boolean expressions can be exploited for cause-effect graphing. A software is developed as an open-source extendable tool for generating test inputs from cause-effect graphs by using different methods and performing mutation analysis for quantitative evaluation on these methods for further analysis and comparison. Selected methods, MI, MAX-A, MUTP, MNFP, CUTPNFP, MUMCUT, Unique MC/DC, and Masking MC/DC are implemented together with Myers' technique and the proposed Spectral Testing in the developed tool. For mutation testing, 9 common fault types of Boolean expressions are modeled, implemented, and generated in the tool. An XML-based standard on top of GraphML representing a cause-effect graph is proposed and is used as the input type to the approach. An empirical study is performed by a case study on 5 different systems with various requirements, including the benchmark set from the TCAS-II system. Our results show that the proposed XML-based cause-effect graph model can be used to represent system requirements. The developed tool can be used for test input generation from proposed cause-effect graph models and can perform mutation analysis to distinguish between the methods with respect to the effectiveness of test inputs and their mutant kill scores. The proposed Spectral Testing method outperforms the state-of-the-art methods in the context of critical systems, regarding both the effectiveness and mutant kill scores of the generated test inputs, and increasing the chances of revealing faults in the system and reducing the cost of testing. Moreover, the proposed method can be used as a separate or complementary method to other well-performing test input generation methods for covering specific fault types.Conference Object Citation - WoS: 10Citation - Scopus: 12Random Test Generation From Regular Expressions for Graphical User Interface (gui) Testing(Institute of Electrical and Electronics Engineers, 2019) Kılınççeker, Onur; Silistre, Alper; Challenger, Moharram; Belli, FevziGeneration of test sequences, that is, (user) inputs - expected (system) outputs, is an important task of testing of graphical user interfaces (GUI). This work proposes an approach to randomly generate test sequences that might he used for comparison with existing GUI testing techniques to evaluate their efficiency. The proposed approach first models CUI under test by a finite state machine (FSM) and then converts it to a regular expression (RE). A tool based on a special technique we developed analyzes the RE to fulfill missing context information such as the position of a symbol in the RE. The result is a context table representing the RE. The proposed approach traverses the context table to generate the test sequences. To do this, the approach repeatedly selects a symbol in the table, starting from the initial symbol, in a random manner until reaching a special, finalizing symbol for constructing a test sequence. Thus, the approach uses a symbol coverage criterion to assess the adequacy of the test generation. To evaluate the approach, mutation testing is used. The proposed technique is to a great extent implemented and is available as a tool called PQ-Ran Test (PQ-analysis based Random Test Generation). A case study demonstrates the proposed approach and analyzes its effectiveness by mutation testing.Conference Object Citation - WoS: 7Citation - Scopus: 8Models in Graphical User Interface Testing: Study Design(Institute of Electrical and Electronics Engineers, 2020) Silistre, Alper; Kılınççeker, Onur; Belli, Fevzi; Challenger, Moharram; Kardaş, GeylaniModel-based GUI testing is an important concept in Software GUI testing. Manual testing is a time-consuming labor and heavily error-prone. It has several well-accepted models that Software Testing community has been working and contributing to them for many years. This paper reviews different models used in model-based GUI testing and presents a case study with a proposed approach for how to convert several well-accepted models to ESG (Event Sequence Graphs) to generate test cases and execute them with an aim to consolidate past and future works in a single model. © 2020 IEEE.Conference Object Citation - WoS: 1Citation - Scopus: 1Mutation Operators for Decision Table-Based Contracts Used in Software Testing(Institute of Electrical and Electronics Engineers, 2020) Khalilov, Abbas; Tuğlular, Tuğkan; Belli, FevziThe Design by Contract technique allows developers to improve source code with contracts, and testing using contracts helps to identify faults. However, the source code of the program under test is not always available. With black-box testing, it is possible to generate contracts from specifications of the software. In this paper, we apply mutation analysis on a model of a given specifications, where mutants are initially gained by applying proposed in this paper certain mutation operators on corresponding model, and then mutated specifications are examined. © 2020 IEEE.Conference Object Citation - WoS: 10Citation - Scopus: 13Regular Expression Based Test Sequence Generation for Hdl Program Validation(Institute of Electrical and Electronics Engineers Inc., 2018) Kılınççeker, Onur; Türk, Ercüment; Challenger, Moharram; Belli, FevziThis paper proposes a test sequence generation approach for behavioral model validation of sequential circuits implemented in Hardware Description Language (HDL). In the procedure of test sequence generation proposed in this study, Regular Expressions (REs) are utilized to model the behavior of the System Under Test (SUT). First, the HDL program is converted to a Finite State Machine (FSM). Then, the obtained FSM is transformed to RE which is represented by a Syntax Tree (ST). In this way, the test sequence generation problem is simplified to the tree traversal algorithm in which symbol and operator coverage criteria are satisfied. The required tools for test sequence generation are provided to automatize the whole procedure of the proposed approach. Also, a running example, based on a real-life-like Traffic Light Controller (TLC), validates the proposed approach and analyzes its characteristic features.Article Citation - WoS: 7Citation - Scopus: 9Input Contract Testing of Graphical User Interfaces(World Scientific Publishing Co. Pte Ltd, 2016) Tuğlular, Tuğkan; Belli, Fevzi; Linschulte, MichaelUser inputs are critical for the security, safety, and reliability of software systems. This paper proposes a new concept called user input contracts, which is an integral part of a design-by-contract supplemented development process, and a model-based testing approach to detect violations of user input contracts. The approach generates test cases from an input contract integrated with graph-based model of user interface specification and applies them to the system under consideration. The paper presents a proof-of-concept tool that has been developed and used to validate the approach by experiments. The experiments are conducted on a web-based system for marketing tourist services to analyze input robustness of system under consideration with respect to user input contracts.Book Part Citation - WoS: 4Citation - Scopus: 8Advances in Model-Based Testing of Graphical User Interfaces(Academic Press Inc., 2017) Belli, Fevzi; Beyazıt, Mutlu; Budnik, Christof J.; Tuğlular, TuğkanGraphical user interfaces (GUIs) enable comfortable interactions of the computer-based systems with their environment. Large systems usually require complex GUIs, which are commonly fault prone and thus are to be carefully designed, implemented, and tested. As a thorough testing is not feasible, techniques are favored to test relevant features of the system under test that will be specifically modeled. This chapter summarizes, reviews, and exemplifies conventional and novel techniques for model-based GUI testing.Article Citation - WoS: 7Citation - Scopus: 11Exploiting Model Morphology for Event-Based Testing(Institute of Electrical and Electronics Engineers Inc., 2015) Belli, Fevzi; Beyazıt, MutluModel-based testing employs models for testing. Model-based mutation testing (MBMT) additionally involves fault models, called mutants, by applying mutation operators to the original model. A problem encountered with MBMT is the elimination of equivalent mutants and multiple mutants modeling the same faults. Another problem is the need to compare a mutant to the original model for test generation. This paper proposes an event-based approach to MBMT that is not fixed on single events and a single model but rather operates on sequences of events of length k ≥ 1 and invokes a sequence of models that are derived from the original one by varying its morphology based on k. The approach employs formal grammars, related mutation operators, and algorithms to generate test cases, enabling the following: (1) the exclusion of equivalent mutants and multiple mutants; (2) the generation of a test case in linear time to kill a selected mutant without comparing it to the original model; (3) the analysis of morphologically different models enabling the systematic generation of mutants, thereby extending the set of fault models studied in related literature. Three case studies validate the approach and analyze its characteristics in comparison to random testing and another MBMT approach.