Shulou(Shulou.com)06/03 Report--

A major innovation of thread testing for the testing industry is that decades after the emergence of white box testing theory, Shanghai Zero one Assembly Information Technology Co., Ltd. combined with more than a decade of painstaking research in testing theory, take the lead in putting forward the second generation coverage technology, which is definitely not a slogan. But ZOA's real understanding of white-box testing and deep understanding of standard third-party testing services is a technology that meets commercial standards after years of basic research and more than two years of research and development. Now let's review the classical testing theory:

1. Testing methodology

Black box function test method:

The black box function testing method regards the software to be tested as a "black box", regardless of its internal structure and what algorithm is used to implement the required functions. Instead, according to the functional requirements of the requirements, the corresponding test cases are designed (including the input data and condition settings of the test and the expected output of the software). Through the manual or automatic comparison of the output (including character output and image output) with the expected results, to verify whether the tested software can give correct results. In order to judge whether the software meets the requirements and whether it is consistent with the relevant parts of the specification and user manual of the software system.

The advantages of this method are:

(a) it can most directly and directly reflect whether the designed software meets the requirements.

(B) it can be done by manual testing even without the support of any testing tools.

Its shortcomings are as follows:

(a) this test method is difficult to find some special types of errors. For example, when the software under test does not provide any output information corresponding to a set of inputs-it may just change a certain working state, and if there is an error in the source code processing part, it is more difficult to find out.

(B) it is impossible to determine which test cases are valid or invalid (the so-called invalid does not mean that no test effect can be received when a test case is used alone, but that it does not contribute when used with a previously used test case, only repeating the completed tests of the previous test case)

(C) unavoidable blindness: when the software is modified, because you don't know which test cases can be tested to the directly modified module or the module affected by the modified module, so you have to run all the test cases from the beginning again, and run dynamically, which is very time-consuming and laborious.

White box structure testing method

The white-box structure testing rule is contrary to the black-box functional testing method: regardless of whether the software being tested meets the requirements and implements the designed function, it only pays attention to the internal structure of the software in order to design enough test cases. so that 100% or as many program components can be tested at least once, so as to expose the software errors as much as possible.

The advantages of the white box structure testing method:

(a) be able to find many software errors that cannot be found by functional testing

(B) each unit can be tested separately before the whole software system is completed.

(C) the test case can be minimized by analyzing the validity of the test case, so as to greatly shorten the response testing time and cost of the modified software.

(d) memory leak analysis can be performed at the same time

(e) Branch execution frequency can be analyzed at the same time.

(F) Software complexity analysis can be carried out simultaneously.

(G) data and variable analysis can be carried out simultaneously

(h) performance analysis can be performed at the same time

(I) dynamic error location and execution path tracing can be carried out at the same time.

The disadvantages of the white box structure testing method:

(1) it must be done through special testing tools, and it is necessary to insert piles (insert record points) on the copy of the user's software to record whether or how many times each branch / condition has been executed.

(2) it will slow down the running speed of the software being tested.

(3) it is necessary to increase the resource overhead of the software under test.

Misunderstandings about Software quality

Many managers of software development organizations and application software development departments mistakenly believe that they have done adequate functional testing (also known as "black box testing") of the software they have developed, thinking that "the quality of our software is fine!" However, after analyzing a large number of "fully functionally tested" software, experts found that about half of these software branches have never been executed!

Why is this? It turns out that the function description of the software is relatively easy, simple and rough, which cannot be explained in detail by the specific implementation logic structure within the software; but in order to achieve the same function, software can have many equivalent implementation methods. In particular, the realization of the software function is related to the programming language, the operating system environment, the database used and some third-party software. In fact, many branches of a software are not directly related to the functions of the software itself, but are used to deal with a variety of possible operating conditions. For example, how to protect open documents when the developed software is suddenly terminated (system power outage or user interruption), how to warn before the system resources are exhausted, how to deal with some files to be used accidentally, and so on. These program branches also have possible errors in writing and must be added to the test. This usually needs to be done through the structural testing of the program (also known as "white-box testing"), and white-box structural testing can only be carried out with the help of software testing tools.

In view of the quality misunderstanding of appeal, ThreadingTest not only judges whether its output (if any) is consistent with the expected value, but also determines whether its execution path is consistent with the expected value for a group of inputs during the test. In this way, even if the test output is consistent with the expected value, errors may be found-if the expected execution path is not consistent with the actual execution path. For example, when testing a calculator program, if the input is 2x2 and the measured result is 4, it may also be an error-if its execution path is not consistent with the expected value: the end result may be the output of the 2 × 2 path. Because TT can test situations with input but no output (at this time only to test whether its execution path is consistent with the expected path), it can be used in any development stage to achieve a veritable full-process test driver.

2. The second generation white box coverage technology.

Coverage technology is one of the basic technical means of software testing, but although there have been a variety of theoretical methods and commercial products for decades, it has not been promoted in the mainstream application field of testing, mainly for the following reasons:

(1) coverage results usually have to be re-accumulated after a rerelease, and testing for large programs is equivalent to returning all tests to zero.

(2) the usual scenario of software testing is to use testing tools to analyze the code, while software testing tools, especially white-box testing tools that can meet commercial standards, have been monopolized by several established software testing tools abroad. The price is high, and the reliability of information security is poor for aerospace and military level testing requirements.

(3) White-box testing operation is difficult, it is difficult for testers to understand, and it is difficult to promote it in the test team.

(4) White-box testing tools are stand-alone, so it is difficult to promote and use them in large-scale testing teams.

(5) coverage has nothing to do with test cases, usually coverage is the mixed result of performing a series of actions, while testers and developers usually need to know the code coverage of a function when locating the problem. None of these traditional white-box testing tools can support it.

(6) with the increasing proportion of mobile applications in the consumer and enterprise market, some established testing tools for mobile environments (android, iOS) obviously lack or even do not provide support.

The above reasons make it difficult for the first generation of coverage technology to be really popularized. In view of the defects of the first generation coverage technology, ThreadingTest proposed a new second generation coverage technology, and designed a new application function on the basis of the coverage method:

(1) there is no need to supervise test scenarios: the statistics of coverage can be collected by backend programs, which can be transparent to testers. Testers only need to run the program after insertion, turn on the automatic collection function of the program, and carry out routine testing without supervision. TT will automatically collect, analyze and store the test execution of the program in the database, and you can easily check the real-time coverage of the program with TT.

The real-time monitoring interface automatically collects the execution of the programs under test and makes statistics.

(2) Bi-directional traceability is an important tool for TT to achieve 100% coverage. Through bi-directional traceability, functional testers can find all untested branches after running all use cases, and confirm with the developer how to cover and supplement the use cases. Until the test of 100% coverage of critical modules is reached. For the more difficult to cover program logic, developers and testers can focus on code walkthrough and joint use case design. Developers and testers can fully interact around the coverage results, but before the emergence of threaded testing tools, due to the lack of coverage as a shared data, it is difficult for developers and testers to fully interact and collaborate. Because developers do not know the specific program execution logic corresponding to test cases, and testers do not know how to complete adequate testing.

Figure two-way traceability interface test cases and code before interacting around coverage

(3) support android mobile application testing based on Java language.

Picture the operation on the mobile phone, and the TT on the connected computer collects test information in real time.

(4) Cumulative coverage technology: if there are coverage results of multiple versions of the program under test, TT can merge the coverage of multiple versions and display them in one view.

Cumulative coverage display of multiple versions selected in the CallGraph diagram of the main interface of the figure.

(5) support in the program structure chart, control flow chart and other graphics to display coverage, testing and developers can clearly see the program coverage from multiple perspectives, you can view the overall coverage, you can also view the coverage of a single function, or even view the coverage implementation of a branch.

Map coverage display

Map coverage display

Map coverage display

Map coverage display

(6) support distributed testing, and the coverage generated by multiple testers' tests can be displayed in a unified view.

(7) realize the white box structure testing technology of US military standard DO-178B MC/DC, achieve 100% coverage, visualize the combination of complex conditions, and greatly improve the product quality.

Through the second generation coverage technology, the whole test can be run in a fully quantified environment, and the whole development and test team can see the contribution of the coverage of each use case to the overall test in real time. Dynamically adjust the whole testing process according to the growth of coverage and other indicators, and at the same time guide the supplement of use cases for the key modules with low cumulative coverage. We hope that the domestic professional white-box testing tool TT can really upgrade the white-box testing technology for testers to master and like, and promote China's software testing to a new level.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.