Test Tools
| As a software tester you'll be exposed to a wide range of testing tools. The types of tools that you'll use are based on the type of software that you're testing and whether you're performing black-box or white-box tests. The beauty of test tools is that you don't always need to be an expert in how they work or exactly what they do to use them. Suppose that you're testing networking software that allows a computer to simultaneously communicate with up to 1 million other computers. It would be difficult, if not impossible, to perform a controlled test with 1 million real connections. But, if someone gave you a special tool that simulated those connections, maybe letting you adjust the number from one to a million, you could perform your tests without having to set up a real-world scenario. You don't need to understand how the tool works, just that it doesthat's black-box testing. On the other hand, a tool could be set up to monitor and modify the raw communications that occurs among those million computers. You'd likely need some white-box skills and knowledge of the low-level protocol to effectively use this tool. NOTE This example brings up an important distinction between two types of toolsnon-invasive and invasive. If a tool is used only to monitor and examine the software without modifying it, it's considered non-invasive. If, however, the tool modifies the program code or manipulates the operating environment in any way, it's invasive. There are varying degrees of invasiveness and testers usually try to use tools that are as non-invasive as possible to reduce the possibility that their tools are affecting the test results. The next few pages will discuss the major classes of testing tools and how they're used. Some examples are based on tools that are included with most programming languages; others are commercial tools sold individually. You may find, however, that your software or hardware is unique enough that you'll have to develop or have someone else develop custom tools that fit your specific needs. They will likely, though, still fall into one of these categories. Viewers and MonitorsA viewer or monitor test tool allows you to see details of the software's operation that you wouldn't normally be able to see. In Chapter 7, "Testing the Software with X-Ray Glasses," you learned how code coverage analyzers provide a means for you to see what lines of code are executed, what functions are run, and what code paths are followed when you run your tests. A code coverage analyzer is an example of a viewing tool. Most code coverage analyzers are invasive tools because they need to be compiled and linked into the program to access the information they provide. Figure 15.1 shows an example of another type of viewera communications analyzer (or comm analyzer, for short). This tool allows you to see the raw protocol data moving across a network or other communications cable. It simply taps into the line, pulls off the data as it passes by, and displays it on another computer. In this example, you could enter a test case on Computer #1, confirm that the resulting communications data is correct on Computer #3, and check that the appropriate results occurred on Computer #2. You might also use this system to investigate why a bug occurs. By looking at the data pulled off the wire, you could determine if the problem lies in creating the data (Computer #1) or interpreting the data (Computer #2). This type of system is non-invasive to the software. In networking, such a monitor is known as a sniffer because, like an electronic nose, it sniffs the data from the air. Figure 15.1. A communications analyzer provides a view into the raw data being transferred between two systems.
The code debuggers that come with most compilers are also considered viewers because they allow programmers or white-box testers to view internal variable values and program states. Anything that lets you see into the system and look at data that the average user wouldn't be able to see can be classified as a viewer test tool. DriversDrivers are tools used to control and operate the software being tested. One of the simplest examples of a driver is a batch file, a simple list of programs or commands that are executed sequentially. In the days of MS-DOS, this was a popular means for testers to execute their test programs. They'd create a batch file containing the names of their test programs, start the batch running, and go home. With today's operating systems and programming languages, there are much more sophisticated methods for executing test programs. For example, Java or a Perl script can take the place of an old MS-DOS batch file, and the Windows Task Scheduler (see Figure 15.2) can execute various test programs at certain times throughout the day. Figure 15.2. The Windows Task Scheduler allows you to schedule when programs or batch files are to run on your PC.
Figure 15.3 shows another example of a driver tool. Suppose that the software you're testing requires large amounts of data to be entered for your test cases. With some hardware modifications and a few software tools, you could replace the keyboard and mouse of the system being tested with an additional computer that acts as a driver. You could write simple programs on this driver computer that automatically generate the appropriate keystrokes and mouse movements to test the software. Figure 15.3. A computer can act as a driver test tool to replace the keyboard and mouse of a system being tested.
You might be thinking, why bother with such a complicated setup? Why not simply run a program on the first system that sends keystrokes to the software being tested? There are potentially two problems with this:
When considering ways to drive the software that you're testing, think of all the possible methods by which your program can be externally controlled. Then find ways to replace that external control with something that will automatically provide test input to it. StubsStubs, like drivers, were mentioned in Chapter 7 as white-box testing techniques. Stubs are essentially the opposite of drivers in that they don't control or operate the software being tested; they instead receive or respond to data that the software sends. Figure 15.4 shows an example that helps to clarify this. Figure 15.4. A computer can act as a stub, replacing a printer and allowing more efficient analysis of the test output.
If you're testing software that sends data to a printer, one way to test it is to enter data, print it, and look at the resulting paper printout. That would work, but it's fairly slow, inefficient, and error prone. Could you tell if the output had a single missing pixel or if it was just slightly off in color? If you instead replaced the printer with another computer that was running stub software that could read and interpret the printer data, it could much more quickly and accurately check the test results. Stubs are frequently used when software needs to communicate with external devices. Often during development these devices aren't available or are scarce. A stub allows testing to occur despite not having the hardware, and it makes testing more efficient. NOTE You may have heard the term emulator used to describe a device that's a plug-in replacement for the real device. A PC acting as a printer, understanding the printer codes and responding to the software as though it were an actual printer, is an emulator. The difference between an emulator and a stub is that a stub also provides the means for a tester to view and interpret the data sent to it. A stub is a superset of an emulator. Stress and Load ToolsStress and load tools induce stresses and loads to the software being tested. A word processor running as the only application on the system, with all available memory and disk space, probably works just fine. But, if the system runs low on these resources, you'd expect a greater potential for bugs. You could copy files to fill up the disk, run lots of programs to eat up memory, and so on, but these methods are inefficient and non-exact. A stress tool specifically designed for this would make testing much easier. Figure 15.5 shows the Microsoft Stress utility that comes with its programming language development software. Other operating systems and languages have similar utilities. The Stress program allows you to individually set the amounts of memory, disk space, files, and other resources available to the software running on the machine. Figure 15.5. The Microsoft Stress utility allows you to set the system resources available to the software you're testing.
Setting the values to zero, or near zero, will make the software execute different code paths as it attempts to handle the tight constraints. Ideally, the software will run without crashing or losing data. It may run more slowly, or tell you that it's running low on memory, but it should otherwise work properly or degrade gracefully. Load tools are similar to stress tools in that they create situations for your software that might otherwise be difficult to create. For example, commercially available programs can be run on web servers to load them down by simulating a set number of connections and hits. You might want to check that 10,000 simultaneous users and 1 million hits a day can be handled without slowing response times. With a load tool, you can simply dial in that level, run your tests, and see what happens. Interference Injectors and Noise GeneratorsAnother class of tools is interference injectors and noise generators. They're similar to stress and load tools but are more random in what they do. The Stress utility, for example, has an executor mode that randomly changes the available resources. A program might run fine with lots of memory and might handle low memory situations, but it could have problems if the amount of available memory is constantly changing. The executor mode of the Stress utility would uncover these types of bugs. Similarly, you could make a slight change to the viewer tool setup shown in Figure 15.1 to create a test configuration as shown in Figure 15.6. In this scenario, the viewer is replaced with hardware and software that allows not only viewing the data on the communications line, but also modifying it. It becomes an interference injector. Such a setup could simulate all types of communications errors caused by data dropouts, noisy or bad cables, and so on. Figure 15.6. An interference injector hooked into a communications line could test that the software handles error conditions due to noise.
When deciding where and how to use interference injectors and noise generators, think about what external influences affect the software you're testing, and then figure out ways to vary and manipulate those influences to see how the software handles it. Analysis ToolsYou might call this last category of tools analysis tools, a best-of-the-rest group. Most software testers use the following common tools to make their everyday jobs easier. They're not necessarily as fancy as the tools discussed so far. They're often taken for granted, but they get the job done and can save you a great deal of time.
Of course, software complexity and direction change all the time. You need to look at your individual situation to decide what the most effective tools would be and how best to apply them. |
EAN: 2147483647
Pages: 233