Caveats

BVA isn't always enough. There can be problems and times when you reduce the data set too far, as I have just pointed out. Here are some of the things to keep in mind as you plan your data testing.

Hidden Boundaries

Boundary value analysis provides us with an invaluable data reduction technique for test data selection. The risk of missing a significant error in some other part of a number line is very small when you use good boundary analysis techniques-unless there is another boundary in your test range that you don't know about. For example, when we tested the logon application at Prodigy, we used boundary value analysis to determine the test set. The testing went well, and the application was sent to production. But soon after it went live, we started experiencing user profile corruption problems.

The Prodigy Services ID was made up of four alpha characters, two digits, and one alpha character-for example, HFDC15A. Non-alpha and nonnumeric characters were not allowed, for example, spaces and punctuation characters.

The testers tested all the forbidden characters in the first four slots, and they tested several permutations of punctuation and alpha characters in the numeric slots 5 and 6, and so on. The one thing they did not think to test was a space in the fifth slot. Of course, almost immediately a clever user did perform this test when he or she was trying to log on the first time.

First-time logons were very special events back in the days when we were using data tape, DASD, to store our information. It turned out that a failed first attempt ruined the entire user profile. So the user would need to call customer service and receive a new ID before he or she could join the service.

This was a very hard bug to isolate and reproduce, because users who encountered it could not remember exactly what they had done. And as far as they were concerned, they hadn't done anything wrong.

We finally found the bug by building a complete test suite that systematically tried every character on the keyboard in each slot. When the space was typed into the fifth slot and the logon was submitted, an error was generated and the user profile was corrupted. With this evidence in hand, we were able to find the problem.

What Was Really Going On

• Bug no. 1. Unknown to us, there was a boundary at the fifth character-a different subroutine. A numeric validation routine evaluated slots 5 and 6, and it did not have the logic to check for a space character. A simple omission.

• Bug no. 2. When the space character was sent to the logon application in the mainframe that verified the user profile, which did not have logic to protect it from an illegal character (that was supposed to be handled at the client), the mainframe process overwrote the profile with the bad information and corrupted the record.

• Bug no. 3. The error message sent to the user told them that they had committed a security breach. This, of course, was not quite correct.

Boundary value analysis automatically picks the most important data points. Ranking is not usually necessary. This really cuts down the number of tests that you have to run while still minimizing the risk of missing some point of failure.

Web Services

Back in the late 1980s, users typing data into fields on a GUI was the unknown territory. All the validation code, the processing code, and the messages were brand-new and not to be trusted. The network protocols and interfaces were nowhere near as stable as what we have today. Server technology was in its infancy. However, we could at least be sure where our application logic was coming from. We could make bold statements like, "The application had been tested and was ready to go to production." I fear this will not be the case in the coming years.

Software as a service has been threatening to become a part of the application landscape for over 20 years. It was inevitable that some day it would become acceptable and profitable. Slowly but surely, Web-based software subscription services are becoming a reality in the Internet landscape. For example, look at eBay and PayPal. Taken together, these are a first-generation retailer Web service, robust and thriving. Anyone can become a shopkeeper. Next, consider the now-ubiquitous virus checkers and operating systems that automatically go "home" to check for updates in the background, and you don't need to know about it. There is Quicken Online, TurboTax-the list is getting longer every day.

Most of us have come to appreciate at least a few of these services even if we don't entirely trust the direction they are taking us. The caveat I see has more to do with the current "next generation" of Web services. These XML-based programs can live on any Web server and advertise their services in globally available search engines in the Universal Description Discovery and Integration (UDDI) servers. Programs can automatically locate and identify these Web services through their UDDI listings, which define the entire contract for the Web service, including what they provide and how you have to compose your requests to get these offerings.

Developers envision this to be the next opportunity for the independent development entrepreneur. Since these Web services are not high profile like eBay, they are background processes for the most part, and they will be hard to track. These contracts are all made programmatically behind the scenes. No humans need to be involved. The users have no idea where the actual processing is being done. It may be, for example, that your travel company contracts with a reservation Web service to get flight information, a Tester's Paradise Web service to find hotels, and so on.

How can we be sure that the Tester's Paradise Web service is legitimate? How can we be sure where they are sending our data?

How can we enforce a service-level contract when we can't be sure we are still using the Tester's Paradise Web service?

I don't have simple answers for any of these questions. I have already had a couple of bad experiences trying to debug problems when a third-party Web service was involved. I don't look forward to testing this next generation of Web services. If they really catch on, we testers will probably have to go back to an IP traffic monitor to even be sure who or what the application is talking to.

I mention this here because it has a profound impact on the verity of the applications and the very boundaries we take for granted, that we think are static and dependable. If you don't know where your data is coming from, or what module is performing the business logic calculations in your application, if these processes can be replaced without warning or sign, we cannot assume the things will work the same tomorrow as they did today.