2.0 Testing Fundamentals

• 2.0 Testing Fundamentals 
  • 2.1 Testing Overview
  • 2.2 Testing Stages
  • 2.3 Testing Techniques
    • 2.3.1 Static and Dynamic Testing
    • 2.3.2 Black Box and White (Glass) Box Testing
    • 2.3.3 Equivalence Partitioning
    • 2.3.4 Boundary Value Analysis
    • 2.3.5 Path Testing
  • 2.4 Testing Versus Inspections

1. A controlled/observed environment that will allow a test to be reproducible, so as to verify that the defect was corrected,
2. A set of sample input that is (generally) a small subset of the possible values,
3. Predicted results that are expected from the set of sample input, so as to verify correct or incorrect operation. The predicted result must be available before the execution of the test case.
4. An analyzed outcome, that compares the predicted and actual results for each execution of the test.

Tests are run to assess the level of quality in a software product. They determine what a system actually does, and to what level of performance. As well, testing helps to achieve software quality by finding defects, along with other activities in the software development process. Finally testing helps to preserve quality by enabling a modified system to be retested, ensuring that what worked before still works.

Glen Myers [Myers 1979] states that

1. Testing is a process of executing a program with the intent of finding an error.
2. A good test case is one that has a high probability of finding an as-yet-undiscovered error.
3. A successful test is one that uncovers an as-yet-undiscovered error.

By uncovering errors in the software, their cause can be determined and then a fix implemented. This results in a software product that contains fewer defects, achieving a greater level of quality. Additional tests may or may not uncover errors, which leads to the fundamental limitation of testing. Testing can only show the presence of a defect. It can never prove the absence of all errors.

As is well known, producing error free software is extremely difficult. The developer requires an assessment of the quality of the software before releasing it to customers. This assessment is derived from testing. If no or few errors are found, then the quality of the software is assumed good, providing some confidence that the product is working well.

Therefore, from this it can be seen that software testing has two main objectives:

1. To uncover errors in the software product before its delivery to the customer, (or the next state of development), and
2. To give confidence that the product is working well.

It is interesting to note that the two testing objectives result in what is known as the "Testing Paradox". If the first objective is to uncover errors in the software product, how can there be confidence that the product is working well, since it was just proved that it is, in fact is not working!

It must also be noted that test cases that pass (i.e. produce the expected results) do not, in and of themselves, improve the quality of the software product. Nothing has changed in the software under test, so given that there is no change there can be no improvement or degradation. Rather, these tests only improve our confidence in the software product.
 

At the lowest and simplest level is Unit Testing. Here, the basic units of the software are tested in isolation. (A unit is defined to be the smallest testable piece of software [Beizer 1990].) The objectives are to find errors in these units in either the logic or data.

Units are assembled into larger aggregates called components. When two or more tested components or units are combined, the testing done on the aggregate is called Component Integration Testing, or just Integration Testing. The tests done at this level look for errors in the interfaces between the components. As well, the functions, which can now be performed by the aggregate, that were not testable individually must also be examined.

After the all the components have been assembled, the entire system can be tested as a whole, and as such this is called System Testing. It might be argued that "System Testing" is just the final stage of component integration testing. However, at this stage the functional and/or requirements specification is used to generate the test cases. System Testing looks for errors in the end to end functionality of the system as well as errors in the non-functional requirements such as performance, reliability, and security.

The last stage of testing is Validation or Acceptance Testing. Here the system is handed over to the end-user or customers. The purpose of this testing is to give confidence that the system is ready for operational use, rather than to find errors. Thus, this is more correctly called a demonstration rather than a test.

Each level of testing addresses different views of the quality of the software product as shown in Table 1.

Table 1 Testing Levels verus Quality Views

A "Verification and Validation" software life cycle model is usually used to demonstrate the goals of the different testing stages. In this model, Verification and Validation is used to refer the testing activities. Verification is the actions in the testing that ensure a specification or function is correctly implemented. The activities that ensure the software that has been built is traceable to the original requirement specification are known as Validation. Boehm [Boehm 1981] states this in two simple sentences:

Verification: "Are we building the product right?".

Validation: "Are we building the right product?"

Figure 1 Verification and Validation Software Lifecycle Model
 

program functions, behaviours and performances. Therefore, testing techniques are not only performed on the functional areas but also non-functional areas of the software. The next few sections will examine several types of testing techniques.

Static analysis testing developed from compiler technology, resulting in a significant number of errors being uncovered. This analysis can examine the control and data flow, check for dead or unreachable code, and identify infinite loops, uninitialized or unused variables, and standards violations. Certain measures such as McCabe s Cyclomatic Complexity are calculated statically and provide an assessment of the testability of the software entity.

Dynamic test techniques can be classified into Functional or Structural techniques. These are often referred to as Black Box or White (Glass) Box testing, respectively, and are described in the next section.

1. test cases that reduce, by a count that is greater than one, the number of additional test cases that must be designed to achieve reasonable testing, and
2. test cases that tell us something about the presence or absence of classes of errors, rather than an error associated only with the specific test at hand.

The goal of black box testing is to find errors such as

1. incorrect or missing functions,
2. interface errors,
3. behaviour or performance errors, and,
4. initialization and termination errors.

White box tests are executed to verify whether the program is running correctly by considering the internal logical structure. Test items are chosen to exercise the required parts of the structure. The test cases designed for performing white box testing should follow the following criteria:

1. guarantee that all independent paths within a module are all covered at least once,
2. execute all the logical decisions for true and false,
3. exercise all loops at their boundaries and within their operational bounds, and
4. ensure internal data structures are validated.
   

One concrete white box testing technique is "Basis Path Testing", as proposed by [McCabe 1976]. There are also other techniques, such as Condition Testing [Tai 1989], Data Flow Testing, and Loop testing.

White box testing and black box testing are complementary techniques. Only using one of them is impractical in the real software testing world. They are typically combined to provide an approach that validates both the interface and the internal workings of the software.

Boundaries can exist on output data ranges as well. Tests should be designed to produce valid and (attempt to produce) invalid outputs.

Hidden boundaries such as maximum string lengths or data set sizes also need to be determined and verified that appropriate responses are generated by the software under test.

These leads to the potential conclusion that inspections may effectively address the software quality issues at the unit test and some portion of the integration test levels. However, as the complexity of the system under test increases, there is a threshold where static analysis must give way to the dynamic nature of testing. In other words, inspections and testing are complementary techniques to assess or improve the quality of the software product. Each should be employed where its return on investment is greatest, or where the alternate technique is incapable of being completely successful.