Maintenance

Maintenance must be done when bugs are found in the software by vendor staff or client users. An enterprise resource planning (ERP) software vendor with 400 software engineers in its development organization may have 80 or 100 staff in its support organization. Many of these support staff are domain experts who advise client user staff on the best use of the firm's software. They must be sensitive to situations in which the software is not performing to the client's requirements. Most of their work is of a true support nature, referred to as "hand holding" or, in really messy cases, "diaper changing." In some fraction of cases it amounts to documenting errors in such a way that they can be repeated in a laboratory test setting. Many, if not most, of the support situations are in the broader context of the vendor's software, as modified or customized by the client. It runs with a database management system (Oracle, DB2, Informix, Microsoft SQL Server), on an operating system (Microsoft Windows XP, OS/400, UNIX, Linux). It suddenly exhibits a glitch because a change in some component of the support software exhibits an unexpected and unintended inconsistency. In more complex corporate global networked environments, multiple vendor ERP software applications run on machines from multiple manufacturers, each with a different operating system. A minor upgrade or bug fix from one vendor can make ripples throughout the whole pond. While the client firm running such a system has long since had to assume system responsibility, it may need vendor software expert backup at a moment's notice. This kind of support goes far beyond calling someone and asking, "What do I do if...?" Usually one or two people from the client firm have the unpublished telephone numbers of one or two experts from the software vendor firm, plus the implicit authorization to call one of them to travel around the world at a moment's notice to help in an emergency.

In any case, what is learned in these situations must be shared first within the vendor's support and development groups and then with all the potentially affected client users of the software. Bug and change reports are posted on vendor Web sites subscribed to by clients or often are broadcast to client MIS groups to be inserted. This happens only after they have been thoroughly tested to make sure that they cause no further "ripples" themselves by introducing unintended consequences. This may mean testing every combination of computer version, database management system (DBMS) software, and operating system. This is clearly a situation that calls for the Taguchi Methods to test a combination of parameters for a combination of noise factors all in one go with special sensitivity to any possible interactions.

Case Study 19.2: Field Maintenance of Software Systems^[2]

The ability to perform major bug fixes over the Internet, and even "ship" software over the Internet, has a downside backlash on final testing before shipping by software vendors. Of course, every attempt is made to fix known bugs, but at this stage the remaining bugs are mostly unknown and soon to be discovered by the new users. The authors of this case study were inspired by a paper titled "Evaluation of Signal Factor and Functionality for Software" by Genichi Taguchi.^[3] If a Taguchi final vendor test and debugging session is carried out using the method of this case study, it can become the basis for later field maintenance of the software on the many customers' premises. The approach is to allocate items of concern or signal factors to an L₁₈ or L₃₆ orthogonal array, run the software for a combination of signal factors in each row of the array, and assign a result of 0 or 1, depending on whether the result is normal or an error. Later, you can calculate a variance or interaction to identify which combination of factors was most likely to appear as a bug.

In this case, the process began with the beta version of the firm's software, because the beta version was known to have several bugs. This allows the process to start with a known "bug baseline," allowing the developers to certify its efficacy. As the software is maintained in the field, the new bug reports are added to the corrected software profile, and the experiments are rerun to provide a continuing maintenance profile. This is especially important in maintenance, because new bugs are often a result of the use of two ostensibly "bug-free" features used in combination with a particular data set. The Taguchi orthogonal array method is used in lieu of complete testing of all combinations. Therefore, evolving an L₁₈ or L₃₆ array over time to discriminate in favor of testing certain combinations (profiling?) is a powerful maintenance technique. As signal factors, the authors chose eight factors that can be selected by users and allocated them to an L₁₈ array. Because this is a case study of a company's actual software product, naturally the authors chose not to tell their readers what bugs their software exhibited and what multiple bug profiles they should search for in maintenance.

Sidebar 19.2: Maintaining Sophisticated Software Functionality Out of Existence

In 1961 Sperry Univac announced the Univac 1107 computer, the largest, fastest, most powerful commercially available machine of its day. It came with a capable but rather "plain vanilla" system software ensemble provided by the manufacturer. However, buyers of such major mainframes already had come to expect a powerful, interactive, open-shop, userfriendly operating system, FORTRAN IV, COBOL, ALGOL 60 compilers, and a sophisticated assembly-language processor.

The inventor of FORTRAN, Roy Nutt, had left United Aircraft and helped found Computer Sciences Corporation, a firm on the West Coast that wrote such software for computer manufacturers and the Department of Defense. They were contracted to write a state-of-the-art FORTRAN compiler. They chose to write it in a new 1107 assembly language, which they also contracted to prepare. This language, called SLEUTH II (because it replaced the manufacturer's SLEUTH I), was a remarkable language, perhaps the best assembler ever written, and certainly the best assembler ever made for writing compilers. Among other advanced features, it allowed recursive programming (at the assembly language level!) and even supported mutual recursion between two separate programs. In addition, it had numerous other abstruse features that were not all that well documentedat least, not well enough for new maintenance programmers, who didn't even know what recursion was, let alone mutual recursion.

SLEUTH II had been written by the best programmers in the world for their own use, with little consideration for the non-computer-literate masses.

One of the authors had occasion to use SLEUTH II as a scientific consultant to Sperry Rand International while on a five-year assignment in Europe. He employed these tricky features to develop programming languages for aircraft structural design applications at the Technical University of Stuttgart^[4] involving hyper-matrices, or matrices with matrices as elements down to several levels. He eventually returned to the United States and took up a new position as scientific consultant in the Univac Scientific Computing Division in Roseville, Minnesota. Here he tried to run some of these programs to test the double-precision floating-point arithmetic capability of the Univac 1108, successor to the 1107. Unfortunately, they no longer worked. Naturally the maintenance staff for SLEUTH II was consulted. It was discovered that because the programmers didn't understand what these features were, they simply "maintained" them out of existence. Because their managers didn't understand the features either, they not only allowed them to do it, but they institutionalized the newly devolved versions as corporate standards. Thus was a work of genius lost forever.