Testing Different Types of Systems

There are many aspects of a system that affect the way it should be tested for maximum effectiveness. We still take the view that testing is a search process. Different types of systems need to be searched in different ways and for different things. We will summarize some ideas that have been presented elsewhere in this book and provide a few new ideas as well.

Reactive Systems

Object-oriented techniques have been used heavily in building systems that are driven by the inputs of a user. One important characteristic of these systems is that there is a very large number of paths through the code. Each run is different from the last. The result is often a very complicated state machine.

One technique was illustrated in Chapter 7. The state machine can be decomposed along the lines of the inheritance relation and the "implements" relation. An interface can specify a state machine even if it is not implemented. Test cases are created from the decomposed state machines and then the test cases can be composed.

For reactive systems, it is usually possible to identify concurrent state machines. These often come from the inherited classes or composed objects. In particular, they come from the "listener" threads in the interface that are waiting for the user to stimulate a mouse event, keypress/release event, or other input event. These concurrent state machines can be tested separately first and then tested together.

A coverage metric that is especially useful for reactive systems is the "all events" level of coverage. This provides an input-side measure of coverage. As such it allows us to determine that the system doesn't do anything that it is not supposed to do. All events coverage for a system such as Brickles entails the events generated by mouse actions, events generated by the windowing system, and the operating environment. For both versions of Brickles, the mouse-move and button-press events are the same and the test cases can be used for all versions. A Java applet version adds events that relate to starting, stopping, and refreshing the applet. There should be test cases for each of these, as shown in Figure 9.13.

Figure 9.13. Events coverage test cases

Embedded Systems

Programs that run within a piece of machinery are referred to as embedded software. The software is working within the constraints of limited memory and constraints (sometimes stringent) on performance. A specific set of object-oriented design patterns apply to this type of system. The development environment is also special in that a simulator is often used as the initial execution environment and the program will be compiled with one code generator and then, later, another code generator. This first raises the question of how well the simulator conforms to the real environment, and then how different the two code generators are. Certain complex compilation issues, such as template instantiation in C++, result in very different object code from two different compilers.

When testing on two different compilers/environments, the type of testing used depends on the nature of the test cases. If the test cases are going to require extensive stubbing of other pieces of the environment, then it may be best to do a minimum amount of testing in the simulation environment and a maximum amount in the actual environment. If extensive stubbing is not needed, or perhaps if the entire system is being assembled in the simulation environment, then doing extensive testing in the simulation environment is preferred. There usually are more tools on the simulation side than the actual hardware side.

In either case, the reuse of test cases will be an important issue. The approach of writing the test cases in the language of the system implementation instead of a scripting language becomes useful at this point. The team must have language tools that work on both the simulator and the target hardware; therefore, they can apply the test cases in both environments.

Embedded systems tend to be more state driven and more safety critical than other kinds of systems. In many cases they cannot stop executing regardless of an error. This means that there is extensive recovery functionality associated with the system. This basically leads to a number of tests that inject faults into the system and evaluates whether the error recovery matches the specified behavior.

We use a technique from communication protocol testing to generate tests from the state machine for an embedded system. The technique is called an n-way switch cover [Chow87]. Consider the state machine shown in Figure 9.14. A very basic set of tests would cover every transition in the state machine at least once. These tests would determine whether any transitions are not provided in the implementation of the state machine. The next level of coverage would be to consider combinations of specific transitions in one of the concurrent sets of states with specific transitions in other concurrent sets.

Figure 9.14. State machine for a cell phone

These systems also often include a "self-test" mode like we described earlier in this chapter. Rather than thinking of this as a testing issue, we should think of it as a part of the system's behavior that should be tested in the same manner as any other behavior. A self-test capability is intended to identify portions of the system that are not working correctly. To test the self-test behavior, a tester must introduce an error in the configuration, modify file permissions, or remove a resource. Then a test case that invokes the self-test should be executed. The expected result is that the system fails.

Multitiered Systems

Multitiered and distributed systems are similar topics. A multitiered system is simply a distributed system with a particular architectural style. Often when this term is used, it refers to systems in which there is a pipeline-style architecture. The client has a GUI at the user's end and some amount of computational logic, an application server, or multiple layers of servers in the middle, and a database on the back end. The amount of computation that can be performed at the client level varies. A client and GUI that does a minimum amount of work is referred to as a thin client.

These systems encompass an amazing diversity of technologies. The GUI portion of the client may be Web-based and may be written in JavaScript. The exact form of the display may be changed depending on the data being received in XML format. The GUI might contain Enterprise JavaBeans and, if so, then the middle layer is an application server.

Distributed Systems

We have addressed this topic in Chapter 8 from an infrastructure perspective. From a requirements perspective, there is only a small difference between distributed and nondistributed systems. The difference in specification is whether the system is supposed to be able to do the following:

1. Seek an alternative server when the expected one cannot be found.

2. Notify its user and pause operation while waiting for the network problem to be corrected.

3. Shut down the system.

Our point here is that the system's specification should explicitly address these possibilities. Tests can then be developed to cover these possibilities.