Software Testing

Time and time again, applications have worked great in a test or staging environment, but failed miserably when deployed in the production environment. Usually this is because the testing environment and production environment are often configured differently, especially with respect to network and host security.

Host Security

The application might have been tested on a server that had not been secured or hardened at all, and it broke in production when it couldn't perform the same actions on the hardened system that it could on the unsecured test server. When you are testing an application, do so on a server that is as close to the production configuration as is possible. The application and associated services should run with the correct privilege levels, and file and directory rights should be set properly. Many, many applications aren't designed or intended to run following your security standards; this sort of testing might be the only way to identify those problems before putting the application into production.

Too often, applications are tested successfully on a test system with full administrator privileges, and then the application support person is surprised that the application does not function properly when it runs with limited privileges. It is amazing how much software is written to require privileged access, such as root or Administrator, when running. Although it's normal for applications to require many privileges during installation, nearly all should run with reduced privileges. If an application such as DNS runs as root or Administrator in production and it is compromised, an attacker could potentially gain administrative access to your server through the vulnerability in that application. Other applications require wide-open file-sharing rights, or files set to Full Control privileges on Windows boxes, for example.

In some cases, you can coerce an application that expects such privileges into working properly without them. You might be able to get the application to run with limited administrator rights. Alternatively, you might be able to set directory rights that are more restrictive than the application supposedly requires, but less restrictive than what you would normally use. Just because the manual for an application claims to need certain privileges doesn't mean it really does. For example, an application might need Full Control to a directory on a Windows workstation, but only for one function that the users didn't need. Hopefully, the application can be deployed with limited rights while still giving the users the functionality they need. Unfortunately, this isn't always possible.

Ultimately, you and the application owners must decide whether the application is so critical to meeting business needs that it's necessary to deploy it even though it introduces substantial vulnerabilities. You should implement additional network security measures to further monitor activity and restrict access to the machine in question. If the application has multiple tiers and the security weakness exists in a single tier, you might be able to isolate that particular tier so that only the other tiers of the application can access that box and no other host can contact it directly. It's vitally important to do everything you can to protect this box.

Network Configuration and Security

Another problem that happens repeatedly is that applications are tested without production network security measures in place. During testing, all the components of the software solution are often on servers on the same subnet; firewalls or packet filters aren't involved. Then the application is transferred to a production environment, and the application owner quickly discovers that the network configuration and security measures prevent the application from working.

In an ideal world, the production environment would be duplicated in the test environment; however, this is typically not done due to financial limitations. Still, it can be costly not to discover a major architecture problem until the application is deployed to production. Delaying the rollout of a system for days or weeks while a firewall is upgraded or replaced, or the software architecture is otherwise redesigned, could cost a company substantial revenue. In such situations, it's certainly advisable to test in advance with a duplicate of your production firewall and other perimeter security components. Perhaps you can justify the purchase of spare equipment to your organization because that equipment can be used both for application testing and for a spare should your production equipment experience a failure.

If your organization simply cannot afford additional network equipment, another option is to do application testing using the production equipment. A separate screened subnet can be established for the test servers to shield them from all but your test traffic. This is a reasonable solution in some cases, particularly with applications that you fully expect to work, without making changes to your network configuration. However, if you test the application and discover it's not working in your production network environment, how easily will you be able to troubleshoot the problem and adjust your network? For example, changing rules on the production firewall on the fly to fix such a problem can be extremely dangerous. Such work should be done only in a test environment.