Section 8.4. Test Environment and Performance Testing


8.4. Test Environment and Performance Testing

It is important for the project manager to get involved in setting standards for verifying that software has sufficient performance. It may seem like this is a detail that can be left up to the testers, but it can't. In fact, it's absolutely necessary that the project manager work with the team to try to forecast how the product will be used.

Performance requirements really should be in place as early as the scope definition phase. When the project manager is writing (or helping to write) the vision and scope document for the product, she should ask questions about the environment in which the product will be deployed:

  • How many users will be using the software?

  • How many of those users will be using it concurrently?

  • Does the software need to be available 24 hours a day, 7 days a week?

  • Are there peak usage times?

  • How much data will be stored in the database?

  • What physical hardware will the software be running on?

  • What version of the operating system will be used?

  • Is security a concern?

  • Will the software need to run under multiple environments (OS, hardware, etc.)?

  • Will the software need to be updated or maintained after it is rolled out?

All of these things will affect how the software is tested. Even though all of the details are not known when the vision and scope document is written, all of these issues should at least be raised, and any known information should be recorded. That way, the requirements analysts will make sure to discover the answers to these questions and include them in the software requirements specification as nonfunctional requirements (see Chapter 6). The testers, in turn, will take those nonfunctional requirements and use them to build a test environment and plan a set of performance tests.

Most performance testing requires automated tools. Sometimes these tools can be bought off the shelf; at other times, they must be developed separately by the programmers. Performance requirements should be known as early as possible in order to properly plan for them. It is a very common mistake for an organization to demand performance requirements, yet fail to budget for licensing fees for performance testing tools (which can be expensive), a hardware environment that mirrors the production environment (which costs as much as the production environment itself), and, most importantly, the time for the testers to set up this environment. This is another reason the project manager must be highly involved in planning for the performance testingit can be a very costly operation, but it is one that must be fought for if the software is to perform adequately.

Even when the software under test does not have to meet demanding performance requirements, it can be challenging to plan for, budget, and set up an adequate test environment. If the users will use multiple hardware or operating system platforms, each of those must be replicated in the test environment. If web-based software has a specific network configuration (routers, load balancers, separate network segments, firewalls, DMZ, security, etc.), then the test environment must have all of the same equipment. If not, the testers will never be able to replicate the real-world conditions under which the software might break.

It is very common to shortchange the test environment by using virtual machines, by using one machine for both the database and the web server, by getting rid of load balancers, or by introducing other differences between the test environment and the production environment. That amounts to a waste of time when it comes to performance testing. Maybe the team will find some memory leaks and bugs, but much of the time, performance problems are in configuration, not in code. This is often overlooked, and it can lead to large problems and user dissatisfaction when the software is rolled out.

It is important that the test environment be completely separate from production. This is especially true of software that has already been released to the clients (especially web-based software, or software that depends on a central database located in the same office as the programmers and testers, or that relies on their network). It is surprisingly easy to take down a production system during testing if the test environment is not completely separate. For example, it may seem "safe" to save money by sharing only a firewall, load balancer, or router. Yet there are serious kinds of bugs that may occur only in the test version, which will make it effectively impossible for production requests to reach the software. If this happens, the software testers will be blamed for the production downtimeeven if it was caused by a bug introduced by the programmers!

The primary reason people typically shortchange their test environments, or do not perform adequate performance tests, is money. It is expensive to perform these tests, and often difficult to convince senior managers that they need to double their hardware budget. The project manager must fight for adequate resources for performance tests, and plan this hardware need from the very beginning (rather than tack it on at the end). The truth is that if it is too expensive to set up an adequate test environment, then the organization simply cannot afford to produce this software and should seek another solution.



Applied Software Project Management
Applied Software Project Management
ISBN: 0596009488
EAN: 2147483647
Year: 2003
Pages: 122

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net