Testing

Testing can take up to half the cost of development in industry. Bill Gates: "are we in the business of writing software, or test harnesses?"
Testing takes place at a number of levels:

Validation of the initial design

Module test after coding

System test after itnegration

Beta test/field trial

Subsequent litigation

...
Note that testing earlier is far less expensive than testing late.

Regression testing

The major advance in package software in the last ten years has been in testing.
Regression testing involves checking that the new version of the software gives the same answers as the old version.

A database is continually updated, containing every bug ever found. Unless all previous bugs are tested for bugs have a ~20% chance of reoccuring unexpectedly. It is best to test data given by real users, as this is what is most likely to be noticed.

Reliability growth models help us asses mean time to failure, the number of bugs remaining and the economics of further testing. It is found emprically. Based on these models the rate of software failure drops exponentially at first, then settles down to decrease at k/t.

Changing testers often makes new bugs visibile, as shown below

Failure to understand the environment in which the system will actually be deployed often leads to incorrect testing and failures. A military approach (a hostile review followed by prolonged field testing) can be very effective, but in industry corporate politics normally prevents a wide uptake of good practices.

Risk Reduction vs Due Diligence

Often one can never be certain about how many bugs are left. Organisations are highly averse to such uncertainty and prfer to avoid risk issues. Legal and cultural pressures replace risk reduction with "due diligence": a standard (but often insufficient) checklist is followed to ensure the legal pressures are satisfied but often not the problem itself.

Summary

A lack of consistent testing can cause programs to behave incorrectly, fail or be insecure. For example in March 2002 it was reported that software bugs in Britain's national tax system resulted in more than 100,000 erroneous tax overcharges. The problem was partly attributed to the difficulty of testing the integration of multiple systems.
Testing is important as there will inevitably be bugs in any program of significant length. These may be caused by a variety of factors at various stages in the development cycle. The earlier a defect is found the cheaper it is to fix -if a product has to be recalled due to a serious fault the cost can be enormous. Inadequate testing may mean no one notices a problem until a vital system crashes. Testing itself however can be very time consuming and expensive and there can never be testing of 100% of real world situations in any program of complexity. More safety critical and military projects will employ greater testing, through large amounts of simulation testing. Often mathematical proofs will be used to show that algorithms work in theory, complementing real world tests.
There are numerous testing techniques and models, application depends upon who the final system is for. Tests can be broadly split into black box tests (external, based on functionality) and white box testing (based on knowledge of the internal workings of the program, following branches etc.) Unit testing tests particular methods and is useful for the micro scale examination of individual parts of a program.
If the product is mass produced a company will generally factor the cost of testing against the possible cost of a recall. For such mass market software often beta testing is used, where large numbers of volunteers test the product and report errors before the final product is shipped. This will test common errors as the people testing are fairly normal users, but they lack the expertise to test extreme cases and to notice many bugs and their causes.
In a “test-driven software development model” unit tests are written first, and as more and more code of the main project is written less and less of the unit tests are failed.
In a clean room process probabilities are assigned to paths users can take both correct and incorrect, and the most common paths are tested more thoroughly. This method was employed by Ericson in their OS development, cutting errors down to 1 per 1000 lines of code, down from the industry average of about 25 per 1000 (Scientific American, 1994).
A good test engineer will have the ability to take the point of view of the customer, a desire for quality and attention and good judgment on where to focus testing efforts with scarce time resources.

Email to a friend | Printer friendly version | Link to this page | Terms of Use | Contact Unless otherwise noted, content on this site is licensed under Creative Commons Attribution 2.5
Software_Engineering/Testing.htm was last modified on 2007-01-02 14:06:38