Case Study: The 401(k) Web Project

I recently reviewed a new Web application for a fiduciary trust company. The company had been assured by their eXtreme developers that testers are unnecessary in Web projects. Since the risk of project failure would be measured in millions of dollars in lost business, they thought it a good idea to risk a few hundred on a second opinion. This turned out to be a good risk for them and a really good example for this chapter.

The Application

The only information I was given was that this was a long-awaited Web-based application that would provide 401(k) retirement plans for self-employed small-business people. Up until this new product line, the benefits of a 401(k) retirement plan had been the exclusive province of large businesses. Not only had there not been any opportunity for self-employed persons to have a 401(k) program, but there was also a significant number of now self-employed persons whose 401(k) retirement plans had been frozen in the 401(k) plan of their last major employer. People with their 401(k) plans in this situation couldn't contribute to them or borrow from them. These assets were effectively unavailable to their owners until they rolled them into an IRA, which could accept contributions but from which they could not borrow. The other option was to close the fund and remove their money, and pay a significant penalty and income tax on the amount.

This new product would allow self-employed people to set up and administer their own 401(k) plan. It would also allow them to transfer in their lame duck 401(k) holdings, and contribute to and borrow against the plan. The response to the plan offering was expected to be huge.

My Testing

No specific requirements were stated, so I adopted the fundamental risk requirements for all fiduciary dealings: No money can be created or destroyed (or lost or prematurely withdrawn). The main risk, if any of these conditions should occur, could include financial instability, audits by government agencies, criminal proceedings, and high-stakes litigation. I approached the assignment armed only with my ID, my newly issued password, my Internet-connected PC, and a blank document open in Word.

My first test was to log on and enroll in the plan. The logon failed to complete because of a security breaching procedure in the application code. This objectionable procedure was rejected by the security settings in the browser, creating an endless logon loop where the logon succeeded but no other page could be displayed. This caused the system to redisplay the logon page. However, no error message was displayed, and normal users would have been completely stymied. And there were more errors.

Within a few hours, I had identified three fatal errors in the application and two fatal errors in the process. In four hours of testing, I logged 20 issues. Of these, 15 required a call to customer service to resolve, and 5 were of a nature that would cause the customer to lose confidence in the institution. I found all of these issues by simply trying to enroll in the program.

Reporting My Results

It was a simple matter to keep track of the number of minutes spent on the phone with CS and the developers-215 minutes-and, of course, my time spent testing, talking, and reporting. The total came to 8 billable hours. I created an inventory of the features that I tested, the logic flow map of enrollment, and a bug report complete with pictures. The logic flow map clearly showed two of the fatal errors. These could have been detected in a design review.

Most of the issues were first reported over the phone, but I prepared a report enumerating all of them, which was widely circulated among the developers and the agents.

Reproducibility and Savings Estimates

The thing that makes this effort so important for demonstrating risk and its associated costs is that all of my issues were 100 percent reproducible. They were going to occur during every enrollment. So every customer was going to experience them. Consequently, calculating the value of testing or the risk of not testing in terms of dollars saved was doable, and the results were impressive and defensible.

Developers have long disputed this kind of cost savings statement on the basis that you can't be sure how many users will be affected. If you have to wait until you have collected actual usage stats from production before you can justify an estimate, the gun is a long time cold and it is hard to get a hearing.

Calculating the Value of Testing

The first step was to calculate the cost of their test effort. There were no expenses, so the invoice was for time only:

The cost of my testing: 8 hours × $125 per hour = $1,000

The next step was to calculate how much cost was incurred by the company for supporting the customer, me, through this process. I asked what their support costs were for call-in customers on a per-minute basis; the average estimated cost was $5 per minute. That figure includes all the costs associated with handling a call-floor space, telecommunications, salaries, benefits, and so on. This gives a typical cost per (computer-literate, Web-savvy) customer of:

215 minutes per customer × $5 per minute = $1,075 per customer

So, they would probably make back their investment on testing the first time a customer did not have to call customer service to get through the process of setting up their plan. This estimate doesn't take into account all the potential customers who would not become customers because of the number of issues they would have to resolve in order to set up their plan. Nor does it take into account the ones who would simply fail, give up, and go away without calling customer service.

Next, I asked how many customers they expected to sign up in the first month. (I ignored the cost to fix the bugs and assumed for the sake of estimation that all the bugs I found would be fixed within the first month.) The company had planned a mass mailing to 40,000 targeted prospective customers. They expected a record response to the long-awaited new plan, so estimates went as high as 20 percent response to the mailing, or 8,000 customers for the plan. (A 5 to 10 percent response would be the normal estimate.) So, I calculated a range from a 5 percent response, or 2,000 customers at the low end, to a 20 percent response, or 8,000 customers at the high end.

For simplicity, I will round off the estimated customer support cost per customer to get through the plan setup stage (calculated in the preceding text) to $1,000 per customer. The savings in customer support alone, due to testing this one feature, is somewhere between

$1,000 × 2,000 customers = $2,000,000

and

$1,000 × 8,000 customers = $8,000,000

Note 

Conversely and conservatively, the risk of not testing is somewhere between $2 million and $8 million in the first month.

If no testing is done, this range represents the budget that they had better have ready for customer support if they want the product to succeed.

Without a test effort to warn them of these problems in advance, the company would have discovered these issues in public and in volume. Yet they had not thought it necessary to spend even 1 percent of this potential loss on a test effort. This is typical of the I-feel-lucky approach to software development that I talked about in Chapter 3. It is also proof that this approach to software development is still very much in use.

Managers at the company didn't believe that a test effort would be worthwhile. They didn't believe because testers don't normally measure these things, and so the value of testing remains a mystery at best. Testers must learn how to demonstrate the value of their efforts, and they must pursue every opportunity to demonstrate this value.

Some Thoughts on This Project

This was a small application. I walked into it cold and broke it, many times. Or rather, I discovered many places where it was broken or disconnected, and I reported them to the maker. There was no planning and no estimation done. I just started exploring. The effort certainly saved the company millions of dollars. But there are some problems in this example.

The first problem is that in a larger project this would not be possible. While I could manage the complexity of this one function in my head and in my notes, in a larger project, there are too many complexities to succeed using this method. Like my friend in the maze in Chapter 11, "Path Analysis," I need a string to help me find my way. The next point is that this was an ad hoc effort; a one-time deal (also covered in Chapter 11). Again, ad hoc efforts don't work on a large scale, which brings me to the next problem.

The biggest thing wrong with this whole scenario is that it gave management the wrong message, even though it was unintentional. I cannot believe the naïveté and myopia that encumbers the managers who somehow reasoned that by testing one-tenth of the application, I had discovered most or all of the bugs and they were now safe to proceed with the deployment.