Preparing a Suitable Test Environment

If you are testing a system for a corporate environment, it's a good idea to have a dedicated suite of test machines. As a result, machines are available for end users to try out the new system without being an inconvenience to you, and they can also focus on the task at hand by being away from their own work environment. More important, it means that you are not running the software on a machine that might contain other versions of the system (or at least some of its components) that you are developing.

The nature, size, and variety of the test environment will inevitably depend on the size of your organization. A large corporation will conceivably have dedicated test rooms containing a dozen or so test machines. This setup will not only offer the scope to test the software under different conditions but will also allow for a degree of volume testing (several different users using the same network resources at the same time, particularly if you have developed a product that accesses a shared database). If you work for a small software house or you are an independent developer, chances are you will not be able to provide yourself with many testing resources.

Most software these days has one of two target markets. The software is either intended for some form of corporate environment, or for commercial sale. Corporate environments can normally provide test environments, and if you work for a small software house or you are an independent developer, you will probably be expected to perform system testing on site anyway. (Obviously your user-acceptance tests must be on site.) If, however, there is no mention of this during your early contract negotiations or project planning, it is worth asking what sort of test facilities your employer or client will be able to provide for you. It's better to arrange this at the outset rather than muddle your way through a limited test.

Test Machine Configurations

If you are testing a system for a corporate environment, it is worthwhile having two different types of test machine configurations. A "plain-vanilla," or basic, configuration gives you a benchmark with which to work. A second configuration that contains a typical corporate build will highlight any problems you might encounter. Let's examine them in more detail.

The plain-vanilla configuration

In this configuration, a plain-vanilla machine is configured solely for the purpose of testing your new system. Preferably, the hard disk will have been formatted to remove everything relating to previous configurations. The machine should then be loaded with the following:

• The version of Windows you are testing against.

• The minimum network drivers that you need to get your configuration to work. By this, I mean that if your corporate environment runs on a TCP/IP-based protocol, check that the machine is not also running NetBEUI or IPX/SPX.

• The build (and only that build) of the system that you are testing.

This test will allow you to assess the performance in a pure environment. Whatever problems arise during this test are either straightforward bugs in your system or are fundamental problems in the way that you are trying to work with the Windows environment. By testing in such an uncontaminated environment, you can be sure that the problems are between you and Windows and that nothing else is causing any problems that arise at this stage.

I have a personal reason for being so particular about this point. A few years back, I was writing a client/server system using a non-Microsoft development tool. The product itself was buggy, and it was difficult to get any form of stable build from it at all. Eventually, everything that went wrong was blamed on the development tool. Because we concentrated on trying to get some common routines built first, my co-developer and I did not attempt to hook up to the Microsoft SQL Server service for a couple of weeks. When we did try, it wouldn't work. We both blamed the tool again. Because we had seen it done in a training course, we knew that it should work. Therefore, we reasoned, if we tried doing the same thing in different ways, we eventually would find success. We didn't. Only when we came to try something else that involved a connection to SQL Server did we find that it was the current Windows configuration that was at fault. We decided to reload Windows to get a plain-vanilla environment, and, sure enough, we got our database connection. As we reloaded each additional component one by one, we found out that the antivirus terminate-and-stay-resident (TSR) program that we were both using was interfering with the SQL Server database-library driver! When we changed to a different antivirus tool, the problem went away.

The corporate configuration

Having gained a benchmark against what works and what doesn't, you can then repeat the tests against a typical corporate environment. For example, your company might have several standard configurations. The base environment might consist of Windows 95, Microsoft Office (a mixture of standard and professional editions), a couple of in-house products (an internal telephone directory application that hooks up to a SQL Server service somewhere on the network), and a third-party communication package that allows connectivity to the corporate mainframe. Typically, additional install packs are created that add department-specific software to the base environment. For example, the car fleet department will probably have an off-the-shelf car pool tracking system. Allowances need to be made in your testing platforms to take into account more diverse variations of the corporate base environment, but only if the software that you have developed is likely to run in this environment, of course.

In a perfect world, there would be no problem running your new system in these environments. However, inconsistencies do occur. Products produced by such large companies as Microsoft are tested so widely before they are commercially released for sale that issues such as machine/product incompatibility are addressed either internally or during the beta test cycle. (Indeed, the various flavors of Windows currently available do contain the occasional piece of code that detects that it is running on a specific piece of hardware or with a specific piece of software and makes allowances accordingly.) One of these inconsistencies can be attributed to executable file versions. For example, different versions of the WINSOCK.DLL file are available from different manufacturers. Only one of them can be in the Windows System or System32 directory at any time, and if it's not the one you're expecting, problems will occur.

Another problem that can arise in some companies—as incredible as it seems—is that key Windows components can be removed from the corporate installation to recover disk space. Many large corporations made a massive investment in PC hardware back when a 486/25 with 4 MB of RAM and a 340 MB hard disk was a good specification. These machines, now upgraded to 16 MB of RAM, might still have the original hard disks installed, so disk space will be at a premium. This is less of a problem nowadays with the relative cheapness of more powerful machines, so if your organization doesn't suffer from this situation, all is well, but it is a common problem out there. I am aware of one organization, for example, that issued a list of files that could be "safely" deleted to recover a bit of disk space. Apart from the games, help files for programs such as Terminal and the object packager (ever use that? me neither), there was also the file MMSYSTEM.DLL. This file is a key component of the multimedia system. In those days (Windows 3.1), very few of the users had any multimedia requirements, so the problem went unnoticed for a while. The fix was obviously quite straightforward, but it still would have caused problems. If your attitude is "Well, that's not my problem," you are wrong. You need to be aware of anything that is going to prevent your system from running properly at your company, and if a show-stopping bug is not discovered until after the rollout, you'll be the one who looks bad, no matter who you try to blame.

A good indication of the amount of effort that went into producing a build of the first version of Windows NT can be found in the book Show-Stopper: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft, by G. Pascal Zachary (Free Press, 1994). Not only is it an interesting read, it describes well the role of the testing teams within a large development environment—larger than most of us will be exposed to during our careers, I dare say. But the book conveys very well the necessity of structure and discipline that must be maintained in large developments.