An Overview of Configuration Testing

The next time you're in one of those computer superstores, look at a few software boxes and read over the system requirements. You'll see things such as PC with a Pentium 4 processor, 1024x768 32-bit color monitor, 32-bit audio card, game port, and so on. Configuration testing is the process of checking the operation of the software you're testing with all these various types of hardware. Consider the different configuration possibilities for a standard Windows-based PC used in homes and businesses:

• The PC. There are several well-known computer manufacturers, such as Dell, Gateway, Hewlett Packard, and others. Each one builds PCs using components designed themselves or obtained from other manufacturers. Many hobbyists even build their own PCs using off-the-shelf components available at computer superstores.

• Components. Most PCs are modular and built up from various system boards, component cards, and other internal devices such as disk drives, CD-ROM drives, DVD burners, video, sound, fax modem, and network cards (see Figure 8.1). There are TV tuner cards and specialized cards for video capture and home automation. There are even input/output cards that can give a PC the ability to control a small factory! These internal devices are built by hundreds of different manufacturers.

  Figure 8.1. Numerous internal components make up a PC's configuration.

  

  
  

• Peripherals. Peripherals, shown in Figure 8.2, are the printers, scanners, mice, keyboards, monitors, cameras, joysticks, and other devices that plug into your system and operate externally to the PC.

  Figure 8.2. A PC can connect to a wide assortment of peripherals.

  
  
  

• Interfaces. The components and peripherals plug into your PC through various types of interface connectors (see Figure 8.3). These interfaces can be internal or external to the PC. Typical names for them are ISA, PCI, USB, PS/2, RS/232, RJ-11, RJ-45, and Firewire. There are so many different possibilities that hardware manufacturers will often create the same peripheral with different interfaces. It's possible to buy the exact same mouse in three different configurations!

  Figure 8.3. The back of a PC shows numerous interface connectors for attaching peripherals.

  

  
  

• Options and memory. Many components and peripherals can be purchased with different hardware options and memory sizes. Printers can be upgraded to support extra fonts or accept more memory to speed up printing. Graphics cards with more memory can support additional colors and higher resolutions. The system board can have different versions of BIOS (its Basic Input Output System) and, of course, various amounts of memory.

• Device Drivers. All components and peripherals communicate with the operating system and the software applications through low-level software called device drivers. These drivers are often provided by the hardware device manufacturer and are installed when you set up the hardware. Although technically they are software, for testing purposes they are considered part of the hardware configuration.

If you're a tester gearing up to start configuration testing on a piece of software, you need to consider which of these configuration areas would be most closely tied to the program. A highly graphical computer game will require lots of attention to the video and sound areas. A greeting card program will be especially vulnerable to printer issues. A fax or communications program will need to be tested with numerous modems and network configurations.

You may be wondering why this is all necessary. After all, there are standards to meet for building hardware, whether it's for an off-the-shelf PC or a specialized computer in a hospital. You would expect that if everyone designed their hardware to a set of standards, software would just work with it without any problems. In an ideal world, that would happen, but unfortunately, standards aren't always followed. Sometimes, the standards are fairly loosecall them guidelines. Card and peripheral manufacturers are always in tight competition with one another and frequently bend the rules to squeeze in an extra feature or to get in a last little bit of performance gain. Often the device drivers are rushed and packed into the box as the hardware goes out the door. The result is software that doesn't work correctly with certain hardware configurations.

Isolating Configuration Bugs

Those configuration bugs can bite hard. Remember the Disney Lion King bug described in Chapter 1, "Software Testing Background"? That was a configuration problem. The software's sound didn't work only on a few, but very popular, hardware configurations. If you've ever been playing a game or using a graphics program and the colors suddenly go crazy or pieces of windows get left behind as you drag them, you've probably discovered a display adapter configuration bug. If you've ever spent hours (or days!) trying to get an old program to work with your new printer, it's probably a configuration bug.

NOTE

The sure way to tell if a bug is a configuration problem and not just a bug that would occur in any configuration is to perform the exact same operation that caused the problem, step by step, on another computer with a completely different hardware setup. If the bug doesn't occur, it's very likely a specific configuration problem that's revealed by the unique hardware used in the test.

Assume that you test your software on a unique configuration and discover a problem. Who should fix the bugyour team or the hardware manufacturer? That could turn out to be a million-dollar question.

First you need to figure out where the problem lies. This is usually a dynamic white-box testing and programmer-debugging effort. A configuration problem can occur for several reasons, all requiring someone to carefully examine the code while running the software under different configurations to find the bug:

• Your software may have a bug that appears under a broad class of configurations. An example is if your greeting card program works fine with laser printers but not with inkjet printers.

• Your software may have a bug specific only to one particular configurationit doesn't work on the OkeeDoKee Model BR549 InkJet Deluxe printer.

• The hardware device or its device drivers may have a bug that only your software reveals. Maybe your software is the only one that uses a unique display card setting. When your software is run with a specific video card, the PC crashes.

• The hardware device or its device drivers may have a bug that can be seen with lots of other softwarealthough it may be particularly obvious with yours. An example would be if a specific printer driver always defaulted to draft mode and your photo printing software had to set it to high-quality every time it printed.

In the first two cases, it seems fairly straightforward that your project team is responsible for fixing the bug. It's your problem. You should fix it.

In the last two cases, things get blurry. Say the bug is in a printer and that printer is the most popular in the world, with tens of millions in use. Your software obviously needs to work with that printer. It may take the printer vendor months to fix the problem (if it does at all) so your team will need to make changes to your software, even though the software is doing everything right, to work around the bug.

In the end, it's your team's responsibility to address the problem, no matter where it lies. Your customers don't care why or how the bug is happening, they just want the new software they purchased to work on their system's configuration.

OF PURPLE FUZZ AND SOUND CARDS

In 1997 Microsoft released its ActiMates Barney character and supporting CD-ROM learning software for kids. These animatronic dolls interacted with the software through a two-way radio in the doll and another radio connected to a PC.

The PC's radio connected to a seldom-used interface on most sound cards called an MIDI connector. This interface is used for music keyboards and other musical instruments. Microsoft assumed the connector would be a good choice because most people don't own musical devices. It would likely not have anything plugged into it and would be available for use with the ActiMates radio.

During configuration testing, a typical amount of bugs showed up. Some were due to sound card problems, some were in the ActiMates software. There was one bug, however, that could never quite be pinned down. It seemed that occasionally, randomly, the PC running the software would just lock up and would require rebooting. This problem occurred only with the most popular sound card on the marketof course.

With just weeks left in the schedule, a concerted effort was put together to resolve the problem. After a great deal of configuration testing and debugging, the bug was isolated to the sound card's hardware. It seems that the MIDI connector always had this bug, but, being so seldom used, no one had ever seen it. The ActiMates software exposed it for the first time.

There was a mad scramble, lots of denials and finger pointing, and lots of late nights. In the end, the sound card manufacturer conceded that there was a problem and promised to work around the bug in updated versions of its device driver. Microsoft included a fixed driver on the ActiMates CD-ROM and made changes to the software that attempted to make the bug occur less frequently. Despite all those efforts, sound card compatibility problems were the top reason that people called in for assistance with the product.

Sizing Up the Job

The job of configuration testing can be a huge undertaking. Suppose that you're testing a new software game that runs under Microsoft Windows. The game is very graphical, has lots of sound effects, allows multiple players to compete against each other over the phone lines, and can print out game details for strategy planning.

At the least, you'll need to consider configuration testing with different graphics cards, sound cards, modems, and printers. The Windows Add New Hardware Wizard (see Figure 8.4) allows you to select hardware in each of these categoriesand 25 others.

Figure 8.4. The Microsoft Windows Add New Hardware Wizard dialog box allows you to add new hardware to your PC's current configuration.

Under each hardware category are the different manufacturers and models (see Figure 8.5). Keep in mind, these are only the models with support built into Windows. Many other models provide their own setup disks with their hardware.

Figure 8.5. Each type of hardware has numerous manufacturers and models.

If you decided to perform a full, comprehensive configuration test, checking every possible make and model combination, you'd have a huge job ahead of you.

Say there are approximately 336 possible display cards, 210 sound cards, 1500 modems, and 1200 printers. The number of test combinations is 336 x 210 x 1500 x 1200, for a total in the billionsway too many to consider!

If you limited your testing to exclude combinations, just testing each card individually at about 30 minutes per configuration, you'd be at it for about a year. Keep in mind that's just one pass through the configurations. It's not uncommon with bug fixes to run two or three configuration test passes before a product is released.

The answer to this mess, as you've hopefully deduced, is equivalence partitioning. You need to figure out a way to reduce the huge set of possible configurations to the ones that matter the most. You'll assume some risk by not testing everything, but that's what software testing is all about.