10 Thou shalt implement and always adhere to a production acceptance process

#10 Thou shalt implement and always adhere to a production acceptance process

Mentioned several times already in the first nine commandments , our last commandment centers around the use of the Web-Centric Production Acceptance (WCPA) process presented in Chapter 13. In our book we focus on the WCPA as tailored for web-centric applications. The WCPA is really a superset of the Client-Server Production Acceptance (CSPA) process presented in Managing the New Enterprise. Most of the WCPA will be useful even if you are not yet designing web-centric applications. Production acceptance takes an application from the early stages of development and into a production status. However, planning for WCPA really begins at the first stages of the software development process. This is where we first start getting users involved, and keeping them involved throughout the development process through the use of customer project teams . At the same time, the development team needs to start getting IT operations involved. The WCPA shows that it is never too early to start getting both users and operations involved.

One of the reasons we developed the WCPA is to serve as a communications vehicle. All too often, development organizations are isolated from the business groups who will use their application and the operations group that will run and maintain them. Proper use of the WCPA will help promote and instill good communication practices. Just as iterative development is a key software development process, so to is iterative and ongoing communications with operations and with users.

This commandment is important because, without a closely followed WCPA, your business may lose valuable revenue or even customers because your web-centric application does not function as expected. Perhaps one of the earliest examples of a WCPA can be traced to Netscape' web server when they first opened for business in mid-1994. When designing their web site, Netscape engineers studied the web server load of their competitor at the time, the Mosaic web site at the National Center for Supercomputer Applications (NCSA). The NCSA site was receiving approximately 1.5 million web "hits" a day at the time. Netscape engineers thus sized their web site to initially handle 5 million hits a day, over three times the NCSA site' capacity. Even that aggressive sizing, it turned out, was not sufficient as the site exceeded 5 million hits a day during its first week of operation. Luckily, Netscape had planned their web site architecture to be scalable, and were able to add additional hardware capacity to handle the load.

The success of the WCPA process is also related to the robustness of your software system's architecture. An even greater percentage growth than Netscape's occurred at AT&T when they first offered their customer Internet service, AT&T Worldnet. AT&T had expected to sign up 40,000 customers for their Worldnet Internet service during its first month of operation and had designed the site accordingly . During its first month of operation, Worldnet registered 400,000 new subscribers, ten times the expected amount. Luckily for AT&T, they had architected the system for growth and put in place the equivalent of a WCPA process. As a result, all new subscribers were able to start receiving service with few complaints of busy signals on dial-in attempts (in contrast to some other well-known Internet services).

A great example of what happens when you don't follow a complete WCPA process occurred at a major U.S. bank during 1998. The bank was planning an upgrade to its Internet home banking service. Prior to the upgrade, the bank used a load balancing scheme to distribute users to a number of front-end web servers, all of which connected to the bank' mainframe back-end systems. As part of the upgrade process, the bank was planning to let all users change their login ID and password, thus allowing more individual flexibility than the previous bank-generated login ID scheme. The first time a customer logged in after the upgrade, he would be required to select a new login ID and password or confirm keeping the old login ID. This process was delegated to a separate new web server in order to not interfere with any of the software on the existing load-sharing servers. The bank, of course, did have some production acceptance processes in place and tested the entire process completely prior to deploying the upgrade. Unfortunately, they did not adequately test the performance of the entire web-centric system. On the first day of production, users started to complain of extremely long access times. Unfortunately, the bank had not taken into account the potential bottleneck of funneling all users through the single server while they were queried for potential login ID changes.

While no process can guarantee a new production system will function 100% correctly, web-centric applications require new kinds of planning like that covered in the WCPA. Not only are user loads on the Internet much more unpredictable, web-centric applications typically involve the interaction of a much larger number of software and hardware systems.