State Testing

So far what you've been testing is the datathe numbers, words, inputs, and outputs of the software. The other side of software testing is to verify the program's logic flow through its various states. A software state is a condition or mode that the software is currently in. Consider Figures 5.8 and 5.9.

Figure 5.8. The Windows Paint program in the pencil drawing state.

Figure 5.9. The Windows Paint program in the airbrushing state.

Figure 5.8 shows the Windows Paint program in the pencil drawing state. This is the initial state in which the software starts. Notice that the pencil tool is selected, the cursor looks like a pencil, and a fine line is used to draw onscreen. Figure 5.9 shows the same program in the airbrush state. In this state, the airbrush tool is selected, airbrush sizes are provided, the cursor looks like a spray-paint can, and drawing results in a spray-paint look.

Take a closer look at all the available options that Paint providesall the tools, menu items, colors, and so on. Whenever you select one of these and make the software change its look, its menus, or its operation, you're changing its state. The software follows a path through the code, toggles some bits, sets some variables, loads some data, and arrives at a new state of being.

NOTE

A software tester must test a program's states and the transitions between them.

Testing the Software's Logic Flow

Remember the example in Chapter 3 that showed the infinite data possibilities for testing the Windows Calculator? You learned earlier in this chapter that to make the testing manageable, you must reduce the data possibilities by creating equivalence partitions of only the most vital numbers.

Testing the software's states and logic flow has the same problems. It's usually possible to visit all the states (after all, if you can't get to them, why have them?). The difficulty is that except for the simplest programs, it's often impossible to traverse all paths to all states. The complexity of the software, especially due to the richness of today's user interfaces, provides so many choices and options that the number of paths grows exponentially.

The problem is similar to the well-known traveling salesman problem: Given a fixed number of cities and the distance between each pair of them, find the shortest route to visit all of them once, returning to your starting point. If there were only five cities, you could do some quick math and discover that there are 120 different routes. Traversing each of them and finding the shortest route to all wouldn't be that difficult or take that much time. If you increase that to hundreds or thousands of citiesor, in our case, hundreds or thousands of software statesyou soon have a difficult-to-solve problem.

The solution for software testing is to apply equivalence partition techniques to the selection of the states and paths, assuming some risk because you will choose not to test all of them, but reducing that risk by making intelligent choices.

Creating a State Transition Map

The first step is to create your own state transition map of the software. Such a map may be provided as part of the product specification. If it is, you should statically test it as described in Chapter 4, "Examining the Specification." If you don't have a state map, you'll need to create one.

There are several different diagramming techniques for state transition diagrams. Figure 5.10 shows two examples. One uses boxes and arrows and the other uses circles (bubbles) and arrows. The technique you use to draw your map isn't important as long as you and the other members of your project team can read and understand it.

Figure 5.10. State transition diagrams can be drawn by using different techniques.

NOTE

State transition diagrams can become quite large. Many development teams cover their office walls with the printouts. If you expect that your diagrams will become that complex, look for commercial software that helps you draw and manage them.

A state transition map should show the following items:

• Each unique state that the software can be in. A good rule of thumb is that if you're unsure whether something is a separate state, it probably is. You can always collapse it into another state if you find out later that it isn't.

• The input or condition that takes it from one state to the next. This might be a key press, a menu selection, a sensor input, a telephone ring, and so on. A state can't be exited without some reason. The specific reason is what you're looking for here.

• Set conditions and produced output when a state is entered or exited. This would include a menu and buttons being displayed, a flag being set, a printout occurring, a calculation being performed, and so on. It's anything and everything that happens on the transition from one state to the next.

REMINDER

Because you are performing black-box testing, you don't need to know what low-level variables are being set in the code. Create your map from the user's view of the software.

Reducing the Number of States and Transitions to Test

Creating a map for a large software product is a huge undertaking. Hopefully, you'll be testing only a portion of the overall software so that making the map is a more reasonable task. Once you complete the map, you'll be able to stand back and see all the states and all the ways to and from those states. If you've done your job right, it'll be a scary picture!

If you had infinite time, you would want to test every path through the softwarenot just each line connecting two states, but each set of lines, back to front, round and round. As in the traveling salesman problem, it would be impossible to hit them all.

Just as you learned with equivalence partitioning for data, you need to reduce the huge set of possibilities to a set of test cases of workable size. There are five ways to do this:

• Visit each state at least once. It doesn't matter how you get there, but each state needs to be tested.

• Test the state-to-state transitions that look like the most common or popular. This sounds subjective, and it is, but it should be based on the knowledge you gained when you performed static black-box analysis (in Chapter 3) of the product specification. Some user scenarios will be more frequently used than others. You want those to work!

• Test the least common paths between states. It's likely that these paths were overlooked by the product designers and the programmers. You may be the first one to try them.

• Test all the error states and returning from the error states. Many times error conditions are difficult to create. Very often programmers write the code to handle specific errors but can't test the code themselves. There are often cases when errors aren't properly handled, when the error messages are incorrect, or when the software doesn't recover properly when the error is fixed.

• Test random state transitions. If you have a printed state map, throw darts at it and try to move from dart to dart. If you have time to do more, read Chapter 15, "Automated Testing and Test Tools," for information on how to automate your random state transition testing.

What to Specifically Test

After you identify the specific states and state transitions that you want to test, you can begin defining your test cases.

Testing states and state transitions involves checking all the state variablesthe static conditions, information, values, functionality, and so on that are associated with being in that state or moving to and from that state. Figure 5.11 shows an example of Windows Paint in the startup state.

Figure 5.11. The Windows Paint opening screen in the startup state.

Here's a partial list of the state variables that define Paint's startup state:

• The window looks as shown in Figure 5.11.

• The window size is set to what it was the last time Paint was used.

• The drawing area is blank.

• The tool box, color box, and status bar are displayed.

• The pencil tool is selected. All the others are not.

• The default colors are black foreground on a white background.

• The document name is untitled.

There are many, many more state variables to consider for Paint, but these should give you an idea of what's involved in defining a state. Keep in mind that the same process of identifying state conditions is used whether the state is something visible such as a window or a dialog box, or invisible such as one that's part of a communications program or a financial package.

It's a good idea to discuss your assumptions about the states and state transitions with your team's spec writers and programmers. They can offer insights into states that happen behind the scenes that you may not have considered.

THE DIRTY DOCUMENT FLAG

State variables can be invisible but very important. A common example is the dirty document flag.

When a document is loaded into an editor, such as a word processor or painting program, an internal state variable called the dirty document flag is cleared and the software is in the "clean" state. The software stays in this state as long as no changes are made to the document. It can be viewed and scrolled and the state stays the same. As soon as something is typed or the document is modified in some way, the software changes state to the "dirty" state.

If an attempt is made to close or exit the software in the clean state, it shuts down normally. If the document is dirty, users will get a message asking if they want to save their work before quitting.

Some software is so sophisticated that if an edit is made that dirties the document and then the edit is undone to restore the document to its original condition, the software is returned to the clean state. Exiting the program will occur without a prompt to save the document.

Testing States to Fail

Everything discussed so far regarding state testing has been about testing-to-pass. You're reviewing the software, sketching out the states, trying many valid possibilities, and making sure the states and state transitions work. The flip side to this, just as in data testing, is to find test cases that test the software to fail. Examples of such cases are race conditions, repetition, stress, and load.

Race Conditions and Bad Timing

Most operating systems today, whether for personal computers or for specialized equipment, can do multitasking. Multitasking means that an operating system is designed to run separate processes concurrently. These processes can be separate programs such as a spreadsheet and email. Or they can be part of the same program such as printing in the background while allowing new words to be typed into a word processor.

Designing a multitasking operating system isn't a trivial exercise, and designing applications software to take advantage of multitasking is a difficult task. In a truly multitasking environment, the software can't take anything for granted. It must handle being interrupted at any moment, be able to run concurrently with everything else on the system, and share resources such as memory, disk, communications, and other hardware.

The results of all this are race condition problems. These are when two or more events line up just right and confuse software that didn't expect to be interrupted in the middle of its operation. In other words, it's bad timing. The term race condition comes from just what you'd thinkmultiple processes racing to a finish line, not knowing which will get there first.

NOTE

Race condition testing is difficult to plan for, but you can get a good start by looking at each state in your state transition map and thinking about what outside influences might interrupt that state. Consider what the state might do if the data it uses isn't ready or is changing when it's needed. What if two or more of the connecting arcs or lines occur at exactly the same time?

Here are a few examples of situations that might expose race conditions:

• Saving and loading the same document at the same time with two different programs

• Sharing the same printer, communications port, or other peripheral

• Pressing keys or sending mouse clicks while the software is loading or changing states

• Shutting down or starting up two or more instances of the software at the same time

• Using different programs to simultaneously access a common database

These may sound like harsh tests, but they aren't, and the user often causes them by accident. Software must be robust enough to handle these situations. Years ago they may have been out of the ordinary but today, users expect their software to work properly under these conditions.

Repetition, Stress, and Load

Three other test-to-fail state tests are repetition, stress, and load. These tests target state handling problems where the programmer didn't consider what might happen in the worst-case scenarios.

Repetition testing involves doing the same operation over and over. This could be as simple as starting up and shutting down the program over and over. It could also mean repeatedly saving and loading data or repeatedly selecting the same operation. You might find a bug after only a couple repetitions or it might take thousands of attempts to reveal a problem.

The main reason for doing repetition testing is to look for memory leaks. A common software problem happens when computer memory is allocated to perform a certain operation but isn't completely freed when the operation completes. The result is that eventually the program uses up memory that it depends on to work reliably. If you've ever used a program that works fine when you first start it up, but then becomes slower and slower or starts to behave erratically over time, it's likely due to a memory leak bug. Repetition testing will flush these problems out.

Stress testing is running the software under less-than-ideal conditionslow memory, low disk space, slow CPUs, slow modems, and so on. Look at your software and determine what external resources and dependencies it has. Stress testing is simply limiting them to their bare minimum. Your goal is to starve the software. Does this sound like boundary condition testing? It is.

Load testing is the opposite of stress testing. With stress testing, you starve the software; with load testing, you feed it all that it can handle. Operate the software with the largest possible data files. If the software operates on peripherals such as printers or communications ports, connect as many as you can. If you're testing an Internet server that can handle thousands of simultaneous connections, do it. Max out the software's capabilities. Load it down.

Don't forget about time as a load testing variable. With most software, it's important for it to run over long periods. Some software should be able to run forever without being restarted.

NOTE

There's no reason that you can't combine repetition, stress, and load, running all the tests at the same time. This is a sure way to expose severe bugs that might otherwise be difficult to find.

There are two important considerations with repetition, stress, and load testing:

• Your team's programmers and project managers may not be completely receptive to your efforts to break the software this way. You'll probably hear them complain that no customer will use the system this way or stress it to the point that you are. The short answer is that yes, they will. Your job is to make sure that the software does work in these situations and to report bugs if it doesn't. Chapter 19, "Reporting What You Find," discusses how to best report your bugs to make sure that they're taken seriously and are fixed.

• Opening and closing your program a million times is probably not possible if you're doing it by hand. Likewise, finding a few thousand people to connect to your Internet server might be difficult to organize. Chapter 15 covers test automation and will give you ideas on how to perform testing such as this without requiring people to do the dirty work.