Common Testing Mistakes

When it comes to testing, an organization can make a number of mistakes that can negatively impact the security and quality of an application. Try to avoid these traps:

• Testing too little, too late

• Failing to test and retest for security

• Failing to factor in the cost of testing

• Relying too much on beta feedback

• Assuming third-party components are safe

Testing Too Little, Too Late

As humans, particularly as computer-savvy humans, we like to break things down into logical steps. A logical step after completing all of the code is to start testing. You should avoid the temptation to wait until after code is complete to start testing. Testing, or more specifically the testing process, should begin at exactly the same time as development begins. For example, you should have a test plan put together before code is written. As instructed earlier, if you don’t have a test plan and you are writing code, you should drop everything after completing this chapter and start working on your test plan.

To alleviate the last-minute rush to get everything tested, you should consider breaking up the development cycle into several short-term stages or milestones, where at the end of each milestone the features targeted for that milestone are stabilized and confirmed to be secure by means of testing.

Failing to Test and Retest for Security

You could fail to test for security altogether, or you could fail to test for certain security issues that you are not aware of. Not testing for security at all is downright irresponsible, especially if your application is deployed and used in a shared environment. Not testing for security issues you are not aware of is acceptable (and expected) as long as the security issue is indeed unknown and not in the realm of widely known (and understood) security issues—such as those presented in this book and in other sources of information such as www.cert.org. For example, if your application is vulnerable to buffer overruns, cross-site scripting, and SQL-injection attacks, you should be testing for all these issues.

Unknown security issues by definition are impossible to test for. Only until a new security threat reveals itself in some way should you be expected to include tests for that particular threat. In fact, the initial onslaught of high-profile, Web-based viruses occurred in part because Web components were never tested for threats not known at that time, such as buffer overruns and cross-site scripting attacks. For this reason, you could argue that the initial round of viruses exploiting unknown vulnerabilities were unavoidable—just as we have little ability to stop new forms of attacks exploiting new or currently unknown vulnerabilities in the future.

The on-going challenge is to keep informed of new security issues as they emerge. As you become aware of new security threats, you should retest your application for those threats. If you have already shipped your application, you should continue to test it for new threats. And if you uncover a vulnerability, you should supply a fix that mitigates the threat.

Another common mistake is to not retest your application or component when the application is retasked for a new purpose. For example, if you have a component acting as a middle-tier business object, you should retest the component if it’s ever made available in some other form such as a Web service on the Internet. Anytime an application or component is placed in a new environment, it’s subject to a new host of threats. Be sure to retest your application’s vulnerability to those new threats.

Failing to Factor In the Cost of Testing

Predicting when all the necessary code needed for an application will be written is generally easier than predicting when the application will be secure and free of all critical flaws. The development schedule is often used as the basis for the product schedule, and code complete is often mistaken for application complete. You should try to think not in terms of a development schedule, but rather in terms of the product schedule. Make sure you have budgeted enough time for stabilizing the application throughout the application-development cycle. Spend as much or more time predicting the necessary test time as you spend predicting the necessary development time—and don’t forget about set up. Chapter 10 presents techniques for securing your built components. You will need to factor in time for build changes and set up as well.

Relying Too Much on Beta Feedback

Don’t count on beta testers to find all the bugs in your application, particularly security bugs. Customers have many reasons to want to get their hands on your product early for evaluation purposes, the least of which is to test it for you. Unless a customer is either generously volunteering his services as your tester or is hopelessly blocked or confused in trying to use your product, you won’t get valuable feedback—such as feedback telling you that your authentication system is ineffective or that your application is prone to denial-of-service attacks. Beta customers will largely assume you know about these problems and that you’ll fix them before the final release. The only time you can count on getting useful feedback is after someone has put up his hard-earned cash, downloaded your product or taken it home, and tried to use it. Essentially, you’ll hear about the problems once it’s too late.

Assuming Third-Party Components Are Safe

Many companies have learned the hard way not to trust third-party components. Early incarnations of shopping carts provided by third-party companies contained a number of serious security flaws. For example, some shopping-cart implementations relied on client-side script to validate parameters, such as price, that went into the shopping cart. Hackers could easily sidestep the client- side script validation, change the price fed into the shopping cart, and reward themselves with a handsome discount.

Do not rely on third-party components to do their part to keep your application safe. Obtaining third-party components might speed up your development time, but in no way does it speed up your testing time. You cannot push off the responsibility of testing the third-party component—as used in your application—to the third-party company. It is your responsibility to test that your application interacts in a secure manner with all third-party components, such as middle-tier components, custom controls, or Web controls.