Identifying and Listing the Test Environments

What hardware and software must the system run on and with? Usually this information comes directly from the requirements documentation. Possible sources of these environmental requirements are development, in-house operations, and the client's operations group.

So far in the book, we have been discussing two of the main scenarios: the software that will deploy on a large, fairly static in-house system and the commercial software that will deploy to a large market and run in many environments. Many other scenarios are possible-for example, firmware, a small software product that will run in a finite number of environments.

Whatever your situation, you need to identify and list the environments that you will test during your effort. Each different type of scenario presents its own issues and challenges.

It doesn't matter whether the system is large or small. What is important is that in the real world, the test environment is never as big as production. However, in order to conduct meaningful testing, the integration environment must be in synch with production-or, at least, as close as possible. Thus, involvement with operations is vital, since that is where the expertise is. Hopefully, they will have adequate budget to create and support all the test environments that are needed.

I am a firm believer in the settling-pond approach to test environments. The settling-pond approach comes to us from wastewater management, a different but similar branch of engineering. The settling-pond approach uses a series of ponds to clean and purify wastewater. In the first pond, the heavy waste is encouraged to settle to the bottom, where it is pumped out. The top layer of fluid, now sans solids, is allowed to flow gently into the next tank for further cleaning. Cleaning may involve various activities, including the addition of aerobic entities and sanitizing chemicals. Each tank is its own environment, and each accomplishes a progressively more stringent clarification of the effluent by settlement. The settling-pond approach in testing is basically the same thing. It uses several test environments, each one cleaner than the last and more a mirror of the actual production environment. By the time the code gets to production, it has been well cleaned.

For example, at Prodigy, we performed module testing in a small, simulator-driven environment, and we performed our first integration tests in a shared integration environment. There was a lot of new code in the integration environment, so it was a fairly unstable place. Code might work one day and not the next. Once the application was thought stable and had passed its exit criteria, it was migrated to a cleaner integration environment where only tested, stable code was allowed. Finally, when it was ready to go to production, it was moved to a mirror environment for rehearsal and customer acceptance testing.

I still use this approach with all my Web-based applications. It is a very convenient way to test the application in the full production environment but still keep it private. It is also a good way to protect a functioning Web environment from new code. I make a backup mirror or, perhaps more correctly, a fall-back mirror, before new code is moved into production.

Note 

Developers should not be allowed to control, modify, or even test in the tester's test environment. They must have their own.

Testing a Large System

If you are dealing with a large system, as in our real-world shipping example that we explored in Chapter 4, then the environmental definitions and requirements are probably already included in the PDRs. Operations is often way ahead of testers in the planning process. In this real-world shipping example in particular, the operations staff actually was the source of the environment information. They had purchased the equipment and scheduled its live dates before a test group was assembled. Development did not have the information; they relied completely on operations to supply them with the correct environments. In other large system projects that are deployed on the client's system, only the client will be able to tell you the environment requirements, and those requirements are probably different at each customer site.

If you are working on commercial software that will run in the real world, a separate analysis and definition step will probably be required when you enumerate the test environments for each customer. In my experience, performing customer-specific environmental evaluations has traditionally started by the software vendor publishing a list of supported hardware, software, and platform information to the customer. Then someone installs the software, and, finally, someone bulldozes through the system, fixing configurations, conflicts, and bugs as they are encountered.

Figure 7.2 shows the environmental definition for the real-world example. This screen shot is the Environment Catalog view of the billing inventory shown on the Web site, taken from the data shown in Table 7.2. Everything to the right of the column labeled EDI is an environment in the system-for example, EIS, FAS, HRIS, IIDS, Mercury, TMC, TWS, TWSNet, and TYMs. The X's at the left side of a column indicate that this particular project touches this environment in some way.

Figure 7.2: Environment details provided during the second-level interviews.

This Web technology makes it very easy for the team to publish and maintain this information in a controlled environment. Any members of the team can be given authoring permission that lets them add, change, or delete any of this list information. Authors can even add columns to the list. The list concept also makes it very easy to sort or filter by any column. So it is very easy to pull a subset of items together-for example, all the projects that touch a certain system. When we combine this list with the list of tests, we can compile the list of tests that touch a system. We use this capability to build test suites on the fly as needed.

Testing Multiple Environments

If testing is automated, the entire test suite can be run against each environment each time code is turned over. In the Tester's Paradise sample application, a sample automated test effort found at the companion site to this book, www.testersparadise.com, environment testing is not mentioned directly in the overall count of tests because the test suite is automated and the test plan called for the test suite to be run simultaneously in all the test environments-hence, the assumption of 100 percent test availability. (The Tester's Paradise application is discussed further in Chapters 10, 12, and 13.)

Many projects today are funded by the organization that wants the product. So, you must first verify and validate the system in that first environment. Once the first customer is running, add the next environment to test; you already have the MITs tests in your suite.

If testing is manual, the prioritization of environments is critical, since there may not be sufficient time or resources to run the entire test suite against every environment. Consider Table 7.4. For example, if the test inventory contains 50 tests and takes 40 hours to run, every test environment you add to the test effort will require an additional 40 hours of test time, plus the time it takes to set up the test environment. In addition, when code fixes come in, they need to be tested in each of the environments.

Table 7.4: Environment Description Matrix

ENVIRONMENT NAME	SERVER HARDWARE	OPERATING SYSTEM AND DATABASE	NUMBER OF USERS
C486-66-30O	2 Pentium 4 2.8-GHz, 120 GB HD	Windows XP Information Server, SQL Server	1
C486-33-30W	2 Pentium 4 2.0-GHz, 80-GB HD	Windows 2000 Advanced Server, SQL Server	1-50
C486-33-30W	2 Pentium 4 2.0-GHz, 80-GB HD	Windows 2000 Advanced Server, Oracle	1-100
C486-33-30S	To Be Determined 40-MB HD	Linux, MySQL	1-25
C486-33-30O	To Be Determined 40-MB HD	Unix, RDBMS	1-100
C486-33-30O	To Be Determined 40-MB HD	Unix, Informix	1-100
C486-33-30O	To Be Determined 40-MB HD	Unix, Oracle	1-100

Analysis and enumeration of test environments can be conducted using the same techniques as data sets, discussed in Chapter 13, "Data Analysis Techniques." An environment matrix is usually the easiest way to publish and track the environments that will be tested. A simple environment matrix is shown in Table 7.4. In this table, hyperlinks serve to link the environment name to the detail layers of the description. There are many ways to prepare an environment matrix. I present a different type in Chapter 8, "Tools to Automate the Test Inventory."