Basic Differences

We want to consider some of the basic differences between sequential systems and these other models particularly from a testing perspective.

Nondeterminism

It is very difficult to exactly replicate a test run when the software contains multiple concurrent threads. The exact ordering is determined by the scheduler of the operating system. Changes in programs not associated with the system under test can affect the order in which threads of the system under test are executed. This means that if a failure is encountered, the defect is isolated and repaired and the test is repeated, we can't be certain that the defect is removed just because the error does not reoccur during a specific run.

This leads us to use one of the following techniques:

• Conduct more thorough testing at the class level. The design review of a class that produces distributed objects should investigate whether there is an appropriate provision for synchronization in the design for the class. The dynamic class testing should determine whether the synchronization is working correctly in a controlled test environment.

• Execute a large number of test cases while attempting to record the order in which events occur. This provides a higher probability that all possible orderings have been executed. The problems we are attempting to detect result from sequences of actions. If all possible sequences of these actions have been executed, the defects will have to be found.

• Specify a standard test environment. Begin with as clean a machine as possible including as few connections to networks, modems, or other shared devices as possible. Identify those applications that must run for the platform to be viable. Add a basic set of applications that would be running on the typical machine. Each test case should provide a description of any modifications made to this standard environment. This includes the order in which processes are started. Including a debugger in the standard environment allows the tester to verify the order in which threads are created, executed, and deleted. The larger the environment and the more it can be shared and networked, the more difficult it is to maintain consistency within that environment. Wherever possible there should be a testing lab in which machines are isolated (at least for the initial testing phases) from the rest of the corporate net and dedicated to the test process.

Additional Infrastructure

Many of the distributed object systems rely on an infrastructure provided by a third-party vendor. Over time, successive versions of this infrastructure will be released. A regression test suite should be created that tests the compatibility between the application and the infrastructure.

A second issue here is the reconfiguration of the system. Some infrastructures are self modifying and reconfigure themselves when the system reconfigures itself. Essentially a specific input data set can cause a different path to be executed because the previous path no longer exists. An analysis of the infrastructure should provide a set of standard configurations for the infrastructure and tests should be executed for each different one.

Partial Failures

A distributed system can find that a portion of its code cannot execute because of hardware or software failures on one of the machines hosting the system. An application running on a single machine does not experience this type of failure: it is either running or not. The possibility of partial failure leads us to include tests in which failures are simulated by removing or disabling network connections or by shutting down a node in the network. This can be implemented in the previously mentioned test lab.

Time-Outs

Networked systems avoid deadlock by setting timers when a request is sent to another system. If no response is received within a specified time, the request is abandoned. The system may be deadlocked or one of the machines in the network may simply be very busy and may be taking longer to respond than what is allowed by the timer. The software must be able to perform the correct behavior in the case when the request is answered and when it is not, even though that behavior may be very different in the two situations. Tests must be run with a variety of loading configurations on the machines in the network.

Dynamic Nature of the Structure

A distributed system is often built with the capability of changing its configuration, for example, where specific requests are directed dynamically, depending on the load on the various machines. Systems are also designed to allow a variable number of machines to participate in the system. Tests need to be replicated with a variety of configurations. If there are a set number of configurations, it may be possible to test them all. Otherwise, a technique such as orthogonal array testing system (OATS) can be used to select a specific set of tests and configurations.