Approaching the Task

The decision-making process that goes into deciding what devices to test with and how they should be tested is a fairly straightforward equivalence partition project.

What's important, and what makes the effort a success or not, is the information you use to make the decisions. If you're not experienced with the hardware that your software runs on, you should learn as much as you can and bring in other experienced testers or programmers to help you. Ask a lot of questions and make sure you get your plan approved.

The following sections show the general process that you should use when planning your configuration testing.

Decide the Types of Hardware You'll Need

Does your application print? If so, you'll need to test printers. If it has sound, you'll need to test sound cards. If it's a photo or graphics program, you'll likely need scanners and digital cameras. Look closely at your software feature set to make sure that you cover everything. Put your software disk on a table and ask yourself what hardware pieces you need to put together to make it work.

ONLINE REGISTRATION

An example of a feature that you can easily overlook when selecting what hardware to test with is online registration. Many programs today allow users to register their software during installation via modem or broadband connections. Users type in their name, address, and other personal data, click a button, and the modem dials out to a computer at the software company where it downloads the information and completes the registration. The software may not do anything else with online communications. But, if it has online registration, you will need to consider modems and network communications as part of your configuration testing.

Decide What Hardware Brands, Models, and Device Drivers Are Available

If you're putting out a cutting-edge graphics program, you probably don't need to test that it prints well on a 1987 black-and-white dot-matrix printer. Work with your sales and marketing people to create a list of hardware to test with. If they can't or won't help, check out the recent equipment reviews from PC Magazine or Mac World to get an idea of what hardware is available and what is (and was) popular. Both magazines, as well as others, have annual reviews of printers, sound cards, display adapters and other peripherals.

Do some research to see if some of the devices are clones of each other and therefore equivalentfalling under the same equivalence partition. For example, a printer manufacturer may license their printer to another company that puts a different cover and label on it. From your standpoint, it's the same printer.

Decide what device drivers you're going to test with. Your options are usually the drivers included with the operating system, the drivers included with the device, or the latest drivers available on the hardware or operating system company's website. Usually, all three are different. Ask yourself what customers have or what they can get.

Decide Which Hardware Features, Modes, and Options Are Possible

Color printers can print in black and white or color, they can print in different quality modes, and can have settings for printing photos or text. Display cards, as shown in Figure 8.6, can have different color settings and screen resolutions.

Figure 8.6. The display properties of number of colors and screen area are possible configurations for a display card.

Every device has options, and your software may not need to support all of them. A good example of this is computer games. Many require a minimum number of display colors and resolution. If the configuration has less than that, they simply won't run.

Pare Down the Identified Hardware Configurations to a Manageable Set

Given that you don't have the time or budget to test everything, you need to reduce the thousands of potential configurations into the ones that matterthe ones you're going to test.

One way to do this is to put all the configuration information into a spreadsheet with columns for the manufacturer, model, driver versions, and options. Figure 8.7 shows an example of a table that identifies various printer configurations. You and your team can review the chart and decide which configuration you want to test.

Figure 8.7. Organize your configuration information into a spreadsheet.

Notice that Figure 8.7 also has columns for information about the device's popularity, its type, and its age. When creating your equivalence partitions, you might decide that you want to test only the most popular printers or ones that are less than five years old. With the type informationin this case, laser or inkjetyou could decide to test 75 percent lasers and 25 percent inkjets.

NOTE

Ultimately, the decision-making process that you use to equivalence partition the configurations into smaller sets is up to you and your team. There is no right formula. Every software project is different and will have different selection criteria. Just make sure that everyone on the project team, especially your project manager, is aware of what configurations are being tested (and not tested) and what variables went into selecting them.

Identify Your Software's Unique Features That Work with the Hardware Configurations

The key word here is unique. You don't want to, nor do you need to, completely test your software on each configuration. You need to test only those features that are different from each other (different equivalence partitions) that interact with the hardware.

For example, if you're testing a word processor such as WordPad (see Figure 8.8), you don't need to test the file save and load feature in each configuration. File saving and loading has nothing to do with printing. A good test would be to create a document that contains different (selected by equivalence partitioning, of course) fonts, point sizes, colors, embedded pictures, and so on. You would then attempt to print this document on each chosen printer configuration.

Figure 8.8. You can use a sample document made up of different fonts and styles to configuration test a printer.

Selecting the unique features to try isn't as easy as it sounds. You should first make a black-box pass by looking at your product and pulling out the obvious ones. Then talk with others on your team, especially the programmers, to get a white-box view. You may be surprised at what features are even slightly tied to the configuration.

Design the Test Cases to Run on Each Configuration

You'll learn the details of writing test cases in Chapter 18, "Writing and Tracking Test Cases," but, for now, consider that you'll need to write down the steps required to test each configuration. This can be as simple as

1.	Select and set up the next test configuration from the list.
2.	Start the software.
3.	Load in the file configtest.doc.
4.	Confirm that the file is displayed correctly.
5.	Print the document.
6.	Confirm that there are no error messages and that the printed document matches the standard.
7.	Log any discrepancies as a bug.

In reality, the steps would be much more involved, including more detail and specifics on exactly what to do and what to look for. The goal is to create steps that anyone can run. You'll learn more about writing test cases in Chapter 18.

Execute the Tests on Each Configuration

You need to run the test cases and carefully log and report your results (see Chapter 19, "Reporting What You Find") to your team and to the hardware manufacturers if necessary. As described earlier in this chapter, it's often difficult and time-consuming to identify the specific source of configuration problems. You'll need to work closely with the programmers and white-box testers to isolate the cause and decide if the bugs you find are due to your software or to the hardware.

If the bug is specific to the hardware, consult the manufacturer's website for information on reporting problems to them. Be sure to identify yourself as a software tester and what company you work for. Many companies have separate staff set up to assist software companies writing software to work with their hardware. They may ask you to send copies of your test software, your test cases, and supporting details to help them isolate the problem.

Rerun the Tests Until the Results Satisfy Your Team

It's not uncommon for configuration testing to run the entire course of a project. Initially a few configurations might be tried, then a full test pass, then smaller and smaller sets to confirm bug fixes. Eventually you will get to a point where there are no known bugs or to where the bugs that still exist are in uncommon or unlikely test configurations. At that point, you can call your configuration testing complete.