Shulou(Shulou.com)06/03 Report--

On effective updating of Test cases and Pesticide Paradox

In 2014, our team tried to promote one thing-reverse adding the problems in the back end of the product (customer, customer service, manufacturing, etc.) to the test case base and extending it to the test case base. Avoid repetitive problems-Liu Chuanzhi asked a contestant on an entrepreneurial program earlier that, as a boss, what is the first thing you have to deal with every day. The contestants said a lot according to their priorities and importance. Liu Chuanzhi said: you should give priority to dealing with recurring problems.

The reopening theory is the key skill of Lenovo, and this is only to borrow this meaning.

After trying to do this for a period of time, the reverse supplementary test cases that have been formed are extended to the relevant test case base, and then executed and checked in practice, there are roughly the following phenomena:

First, most of them are not implemented at all.

Second, a small part should be carried out selectively.

Third, a small part, rewritten and incorporated into the original test cases for execution.

There are many reasons for the first phenomenon, which are aboveboard and less aboveboard-I prefer to think of them as those mentioned below.

For the second and third phenomena, the question I was asked was: if it is not pulled out separately according to the format we have written, and there is an execution result, then it is impossible to count whether these new use cases have been implemented manually or by tools. The data is not available, so it is impossible to judge whether there is any optimization and progress in testing.

Let's put aside the targeted problems of complex execution and inspection for the time being, just from the test itself-- a problem arises. Should the steps of the problem, the defect scenario, and similar possible logic be written in the test case, and be executed repeatedly in subsequent projects?

The answer is not necessarily-test design is something that experts in a field do, not just a rigid description of one by one. Or to put it another way, test cases are at the heart of testing and are creative things, rather than having an absolutely correct methodology that can be done once and for all.

Give some different examples to show the complex relationship between appearance and essence:

First, the cause of the problem, what is its frequency?

EX1-- if the problem is because the developer commented a piece of code by mistake, or because of a variety of pen errors caused by defects, found that the code was modified and recompiled, the problem is solved.

Then the probability of this kind of problem is one-off. After this defect is fixed, the probability of recurrence is very small-unless the code is left behind by someone else, and then another development, and the courage to modify some of the old code. Then his team leader did not have a code review and submitted it directly. Then this problem can come to light again. Normally, in response to this situation, it is not necessary to write a few case, and subsequent projects are executed every time.

EX2-- has a resource, and multiple modules will be called, and the business logic of these modules are closely coupled, and the co-debugging has not been done well, even because of solving the defects, there have been many disputes, such as yours, mine, and so on.

Then there should be a probability of this kind of problem. After this defect is fixed, not only the operation caused by this defect needs to be added later, but even some methods of these modules calling resources have not been paid too much attention before, and the test design should be strengthened appropriately in the follow-up.

Second, how many components, branches and versions are involved in the question?

EX3-- in embedded devices, the word "mouth" cannot be displayed and is displayed as "mouth". The reason for the problem is that when the memory of the embedded device is small, the font may use a first-level font, so it is possible that all the second-level font text will display an exception.

2.1. Uniqueness:

If the whole company uses a unified font font. Then only update this font, the second-level font problem of all embedded devices will be solved, after this defect is fixed at once, it does not need to be included in the test case.

2.2. There are multiple branches:

There are a lot of outsourced projects to display different fonts and languages of different countries. In short, there are many branches of font.

2.A, if there is a good and comprehensive defect analysis and spread notification, we each repair, and do not need to be written into the test case, because it is an one-time behavior.

2. If there is a certain defect notification way, different tributaries can be perceived, but the timeliness is poor, then it should be solidified in the test case and executed for a period of time.

2.C, if there is no high degree of defect notification, based on a stream, follow-up development and maintenance, then be sure to write in the test cases, or even organize small special tests to focus on exposing different versions of the problems.

Third, is there a strong sequential dependency?

EX4-- if a problem is strongly related to the business logic order, it needs to go through the necessary steps such as 1, 2, 3, 4, 5, etc., before it will lead to an inevitable bug. From the tester's job, to find such bug (commonly known as divine bug) is the best performance that he knows business knowledge like the back of his hand, and he can even boast about it as his own anecdotes.

However, this kind of bug, after regression testing, you really don't have to form a test case and let every subsequent project be executed repeatedly-- the strong business sequence relationship has been repaired, and the follow-up will not occur naturally. Whether there is any other implicit logic and whether other branch state tests are needed is another matter.

Fourth, do you have the conditions for verification?

EX5-- all kinds of complex communication problems between foreign manufacturers.

This kind of bug, mostly on the spot, can only be repaired through packet analysis, code stream analysis, and then constantly replacing the temporary version. If it is the protocol standardization, it can be strengthened in the testing link, if it is the non-standard agreement produced by the rapid development of various manufacturers, no one can do it, but can only solve it on the spot.

So, you can write a note, device A, which needs to be connected to the XXX product of manufacturer An or the YYY product of manufacturer B for ZZZ function test. However, these test cases are not enforceable.

For this kind of interconnection problem, the best solution is to find a customer with a large number of equipment models, maintain a good customer relationship, and when you release a new product, bring your own equipment, and the joint adjustment will be done.

This example requires testing strategies and solutions for such problems, rather than bluntly replenishing test cases that cannot be executed.

The problems caused by EX6-- running for a long time, such as the aging of XX devices after three years of operation, or the loss of versions and files for no reason.

This involves reliability and flash repeated reading, fragmentation and Black Swan events, respectively.

To test this kind of problem, in order to simulate the effect of three years in a short time, we can only test the amount of equipment and derive the approximate stability through the formula. Written in the test cases, in the daily work, it is obviously impossible to achieve, it is better to honestly do special tests, concentrate manpower, equipment and so on. Solve this kind of problem at once.

The above is the first concept of test cases extracted from defects in reverse-- to learn from the problem and avoid repeating the same problem later, the thinking and logic are correct. But it definitely does not mean that compared with gourd painting, some problems may be one-off, some problems may be bigger problems, some problems you know but can only look at the probability and input-output ratio, or try to solve them through other methods.

The second concept has something to do with teams and people. The real operation of a team is often known only to insiders. By the same token, the real cause of the problem is often hidden within a team, so whether it is possible to write accurate test cases can only be handled by the team itself. This means that if the test case update is not triggered actively within its own department, but driven by a third-party department (quality department, process department), then it is doomed to get some products.

Fifth, the real cause of the problem will make your case completely different.

For example: the software client decodes no sound.

But if you add a test case: "after the software installation / update is successful, check whether the codec status is normal, and the expected result: the image and sound are normal." People who get this use case will think that the writer show is funny, such a basic thing has long been tested, but also seriously added, the end result is either not to implement, or brainless tick through.

But this most basic problem will appear again and again, and there is naturally something deep behind it.

EX7: compatibility problem. A certain audio format has been flipped and compatibility is not considered.

The early version of the audio code stream was sent, and the decoding failed, and this lack of sound is a standard compatibility problem-- so the supplementary test case should be written to test the compatibility with each version of the product to see if the audio is normal.

Does it look like you've got a real problem? But this use case is also a comprehensive use case in theory, and it cannot be executed in practice (refer to VI, the enforceability of test cases)-there may be more than 20 historical products and more than 20 historical versions. You use your mouth to interconnect, not to mention whether there are so many devices in the test environment, that is, when everything goes well, it will take several working days to replace the version and test it. In the context of test resources and time constraints, it is no wonder that this case will be executed.

Combined with the above point of view, to see whether the version of the codec component has changed, and then decide whether to perform a compatibility test between codec versions, and then select the product and version through the equivalence class. the correct solution is to let the test execution be executed in half an hour to an hour.

The problem that EX8:DLL is overwritten. The system first installed the product codec plug-in, and then installed other players (Storm Player, thousands of listening have this problem), the same name codec plug-in was overwritten, decoding failed.

The troubleshooting process may be rather complicated, but after a clear investigation, whether to write a test case will be included for execution every time: "first install our products, and then install Storm Player to encode and decode to see if it is normal. Expected result: video and audio is normal." Is this use case executable?

Look at the solution to the problem:

8.1. If the solution is sales avoidance-the server is installed independently, the software is installed in a standard version, and no other software is allowed to be installed, then the problem does not need to be solved at all. You only need to uninstall the non-permitted software and reinstall it.

8.2 if the solution is to uniformly change the path of dll from the system directory to the specified directory, so as to avoid the problem that dll is overwritten. Then the case needs to be executed for a period of time, and it should be clearly checked. After setup, check the XX file under the XX path to see if the check item has been updated successfully.

If the solution is not uniformly specified, each software team specifies its own directory, and the characteristics of dll are different, and multiple versions are used at the same time, then there must be conflicts in calling dll for multiple software of their own company, or to put it bluntly, some people do not even consider the dll search path "current directory-> system directory-> windows directory-> directory specified by the environment variable Path". So if this is to be exposed, you have to find a few people to set up a special test, or even check it periodically-- but once it is so bad, there are no uniform rules for all software products. People set up their own ideas behind closed doors, and simply think that customers will only install one product. If it were me, I would definitely go on strike-- the system department would sit down and make a rule, and we could change it together, once and for all. As a result, the problem is left to the back-end team, find a few people time-consuming, long-term to toss about. This is not to treat others as a person, nor to consider the overall cost of the project, or simply not to do their duty, why let the testers take the blame?

For a seemingly identical phenomenon, the actions that may be taken are completely different because of different causes. If you are not on the project team, you don't know the reasons like the back of your hand. Is it your problem to simply urge a certain person? On the contrary, it is easy to stimulate rebellious emotions and produce a lot of negative energy to promote the product as a whole.

Sixth, the enforceability of test cases.

An example has been given above. It is irresponsible to write test cases of exhaustion rate test casually.

A: I wrote to traverse all the interfaces, all the formats, clearly, why didn't you measure it?

B: have you calculated how much time your line actually needs to be tested? You write a sentence, I will toss about for a week.

When something went wrong, you said you thought that the executive was lazy, but with such a tight testing time, it was impossible to give a week to test such a basic function. Isn't test design supposed to do test priority, test equivalence class division, or even make risky non-test decisions based on customer usage probability?

Form a chart to illustrate the point.

The purpose of this table is not to memorize, but when you think, "when this problem arises, should we write a case and execute it all the time?" When, according to the actual characteristics of their own products, to make a correct analysis and judgment.

So back to the beginning of the problem, if it is in accordance with the cold specification agreement, all the problems must be incorporated into the defect management. In the face of complex and ever-changing realities, landing styles may be varied (no need, optional implementation, long-term routine coverage, short-term coverage, special focus solutions).

The various inspection methods of the third-party department may not be applied to the use of existing use cases one by one, and the instinct to seek advantages and avoid disadvantages is very troublesome to explain the reasons and solutions to people who do not understand the business. So often the actual product verifier, instead of trying to explain one, two, three, four needlessly, might as well just make a superficial statement and write a few test cases that look easy to understand, so that everyone will be more comfortable.

According to the concept of driving force 3.0, carrot sticks will stifle the enthusiasm and creativity of others, so the high-end behavior of intellectual positioning and writing enough test cases, through standardization and inspection, will often be turned into hydrological coping behavior-this is not the focus of this article, so far.

Regression test case update and optimization itself.

In addition to the reverse supplement of test cases extracted from defects, the benchmark library of test cases should also be regularly reviewed, modified and updated.

This is the insecticide paradox (pesticide paradox) in test cases-the more software is tested, the more immune it is to software testers. Or literally, if the field is treated with only one pesticide for a long time, then the bug will develop drug resistance, resulting in less and less effective, and finally can not kill the bug. Or another dimension to describe: the test case is a kind of data quantitative index, what you want to assess, you will inevitably get this quantitative result in the long run, but it may not be very helpful to the actual improvement of things.

It is possible that some test cases were designed to address the weaknesses of the product at that time. But after several project tests are completed, or even a few years later, the validity of these test cases tends to be zero.

1. It may be a code logic fix, and this kind of problem will never happen again.

2. it may be a software and hardware change, and the original test scheme needs to be adjusted.

3. It may be the priority adjustment of test cases caused by the change of function point priority and so on.

For example, in a test case, the version was required to be placed after the publisher, and after a re-download, an installation test was conducted to confirm the version number information of each module. This is an inevitable test step under the conditions at that time. Reasons first, there have been different decompression software and breakpoints continue to download tools, resulting in inconsistent file byte size problem; second, the original version is a folder, in which there are a variety of ini, exe, bin, setup files, it is easy to test the version and the release of the inconsistent phenomenon; so re-install to check the version after the behavior is very necessary.

But after a few years, there are more and more decompression software and better compatibility. It is more and more realistic to specify the decompression software and version number; the release file itself is not a folder, but one or more Zip packages, which avoids the risk of manual copy and paste of multiple files, resulting in artificial confusion; third, data verification is also done much better than before, and there is no need to use hillbilly method to verify manually.

Therefore, there is no doubt that this use case can be deleted. After all, it will take two or three hours to download, replace and look at the version number.

Constant test cases over the years, from the perspective of cost-effective work, this is ineffective work content. It's like the waiter standing at the top of the stairs, just standing there because of tradition, not knowing that at first it was just to remind everyone that the paint on the stairs was not dry.

In terms of testing responsibility and risk, this is the responsibility of the manager. In any case, there is no doubt that a constant test case means a test team that has not improved for many years.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.