Cost of Software Quality


Philip Crosby, in Quality Is Free,[3] argues that the cost of quality (CoQ) is "the expense of nonconformancethe cost of doing things wrong." Some call it the "cost of poor quality" (CoPQ) or the "price of nonconformance" (PoNC). Determining CoQ is important, because it reveals how much you expend on prevention, appraisal, inspection, discovering, analyzing, scrapping, and correcting defects, faults, and errors. Software process maturity and quality levels lag hardware by a factor of 20 to 1. The cost of poor quality (CoPQ) related to software bugs, development cost overruns, botched implementation, and canceled projects is astronomical. A recent study puts one aspect of CoPQ, canceled projects, in excess of $300 billion in the United States alone.[4]

Cost of software quality (CoSQ) can be defined as the total cost of conformance and nonconformance to the customer's quality requirements. It consists of various direct and indirect expenses incurred when preventing, appraising, testing, discovering, analyzing, and fixing software faults, including maintenance. As we discuss in Chapters 1 and 2, some 40% of software development cost is spent on testing alone to remove errors and ensure adequate quality. Further, some 80 to 90% of the total software life-cycle cost (LCC) goes into software maintenance to fix, adapt, and expand the delivered program to meet users' changing and growing needs.[5]

Benefits of Cost-of-Quality Analysis

Identifying, understanding, and managing the cost of software quality are crucial components of a DFTS initiative for the following reasons:[6]

  • Cost-of-quality analysis helps the software team focus on discovering, and correcting, the root causes of software defects.

  • Root-cause analysis allows the project personnel to determine how the development process can be improved to prevent further defectsand thus improve quality.

  • This visibility also provides a vehicle for effective communication across all functions supporting the project, regardless of the organizational structure.

Traditional cost-evaluation metrics, generally associated with manufactured products, have focused on internal costs of rework, rejects, and returns; therefore, they have focused just on "failure-related activities."[7] They emphasize inspection and cost minimization by taking corrective measures following inspection. Although the need for corrective measures cannot be eliminated, an enlightened approach emphasizes preventive interventions so that corrective measures are unneeded in the first place, or can be minimized.

It should be emphasized that quality can be "free" only if the cost of quality improvement tasks is less than the consequent cost of not undertaking such tasks. This is possible only if adequate quality tasks are actually performed and are targeted at failure reduction. Such reduction can be dramatically enhanced by employing ways and means to move failure prevention, detection, and removal activities closer to the front end of the development processto the upstream stages of requirements planning, concept development, and design. Only in this way can the cost of performing quality tasks become less than the cost of not performing quality tasks.[8]

Evaluating the costs of not performing quality tasks is always challenging, because so many intangible factors are related to customer expectations and competitive factors that are in play in a particular industry. Often, conservative estimates are the only way to come up with a workable figure. This issue is discussed further when we cover Taguchi's Quality Loss Function (QLF) later in this chapter. Two sets of measures are involved in cost benefits related to quality tasks: the costs associated with quality tasks, and the cost savings from performing quality tasks.

Cost of Quality Tasks

Figure 4.2 shows the relationships between the cost components of various quality tasks. They can be summarized as follows:

cost of performing quality tasks

= cost of conformance + cost of nonconformance

cost of conformance

= preventive costs + appraisal costs

cost of nonconformance

= cost of internal failure + cost of external failure

cost of internal failure

= cost of upstream internal failure + cost of downstream internal failure


Figure 4.2. Cost Breakdown of Software Quality Tasks


It's important to understand this cost structure and its implications for overall quality and product cost for three reasons:

  • Emphasizing interventions upstream ensures much lower downstream and overall costs than would be the case otherwise.

  • These costs are interrelated. For example, high external failure costs may be due to lack of attention to upstream preventive measures. Thus, external failure costs may be avoided or reduced drastically as a result of taking appropriate preventive measures upstream.

  • Increased expenditure on inspection and testing without addressing the root cause of failure at upstream stages may just increase the cost of quality without any long-term improvement in overall quality.

The following example illustrates the points we want to make:

Experts estimate that 60 to 90 percent of the total quality costs are the result of internal and external failure and are the responsibility of, but not easily controllable by, the management ... an increase in prevention usually generates larger savings in all other cost categories. ... In a typical scenario, the cost of replacing a poor-quality component in the field might be $500; the cost of replacement after assembly might be $50; the cost of testing and replacement during assembly might be $5; and the cost of changing the design to avoid the problem might be only 50 cents.[9]

The objective should be to reverse this process. The majority of quality costs should be incurred at prevention, some in appraisal, perhaps a few in upstream and downstream internal failures, and virtually none in external failure. Thus, the software developer should first attempt to reduce external failure costs to zero by investing in preventive and appraisal activities to design bug-free software in the first place. Then he or she should discover sources of design and coding failure, if any, and take corrective actions consequently. As quality improves, failure costs decrease, and the amount of appraisal can be reduced with the shift of emphasis to prevention activities.[10] Determining cost of quality is a first step in identifying and removing the root causes of failure.

Classification of Cost of Software Quality

The profile of software quality costs differs from that of manufactured products. The costs reflect the manpower intensity, service orientation, and more intense customer interaction of a software development process. Because much of the work is intellectual, it lacks the visibility of a manufacturing process. The required visibility must be "created" by appropriate documentation. Just like in other service organizations, user interaction, or the lack thereof, is a crucial element of overall costs. CoSQ can be classified into five major categories:

  • Prevention costs

  • Appraisal costs

  • Upstream internal failure costs

  • Downstream internal failure costs

  • External failure costs

Prevention Costs

This category includes various costs associated with equipment, systems, and manpower associated with preventive upstream activities to ensure that nonconforming software products are not designed in the first place. These costs can be broken down as follows:

  • Quality planning and development costs: Personnel costs of quality professionals involved in developing new systems.

  • Information systems costs: The costs associated with equipment and organization of data and measurements required for making various quality-related decisions.

  • Training and administrative costs: Training, administrative, and communication-related costs.

  • Quality initiative implementation costs: Various costs associated with implementing quality initiatives.

  • Risk assessment: Cost of risk appraisal, such FMEA.

Appraisal Costs

This category includes various costs associated with activities carried out to ensure that the software designed and developed meets customer requirements. This includes salaries, equipment, measurement and analysis, customer interaction, and equipment costs. The activities involved are

  • Review and inspection

  • Test and evaluation

  • Verification and validation costs

  • Risk appraisal at downstream stages

Upstream Internal Failure Costs

Includes all costs related to manpower and equipment incurred in detecting and correcting defects and faults associated with requirements identification, design, and coding. This involves self-debugging your own unit and components. All this may also require interacting with the customer and user domains and carrying out risk analysis.

The formal testing by the independent test team is done in the following stage and is identified as part of downstream internal failure costs. Failure costs escalate after self-debugging, because it involves the following steps:

  • The test engineer researches and reports the failure.

  • The coder receives the failure report and fixes the fault.

  • The release engineer comes up with a new release.

  • The system administrator installs the new release in the test environment.

  • The tester tests the new release to confirm the fix and to check again for regression.

It helps a great deal if self-debugging can detect and fix faults.

Downstream Internal Failure Costs

This category includes all costs related to manpower and equipment incurred in detecting and correcting defects and faults associated with review, test and evaluation, and verification and validation. Testing at this stage involves an independent test team. As just mentioned, the internal failure costs in the downstream stages may be much higher than in the previous stage. These activities also require interacting with the customer and user environments and carrying out risk analysis.

External Failure Costs

Here the costs may be even higher than those incurred at the two internal failure stages just discussed. The faults in this case are found by the customers and not by self-debugging or by the internal test engineer. The whole cycle of fault fixing has to be undertaken. The costs include all manpower as well as equipment-related costs associated with detecting and correcting faults and complaints following release to customers. This category also includes

  • Testing and fault-correction costs following complaints, returns, or any other usage-related problems

  • Warranty and litigation costs following customer usage that are covered by warranty and product liability clauses

As stated earlier, the cost of software quality can be minimized dramatically by taking preventive measures, and corrective measures, if needed, upstream. (Table 4.2 lists essential activities at various stages.) The investments made in upstream preventive measures provide a much higher payoff than development models that rely on corrective measures downstream, or worse, corrective measures following failure during customer usage. External failures involve numerous unforeseen and intangible costs. In extreme cases, the potential damage may cripple the business. In addition to a sound preventive system, provisions must be made for costs related to insurance, product liability, and legal defense. The best strategy is, of course, prevention.

Table 4.2. Conformance and Nonconformance Quality Activities at Various Software Development Stages
 

Cost of Conformance

Cost of Nonconformance

Cost Classification

Prevention Costs

Appraisal Costs

Upstream Internal Failure Costs

Downstream Internal Failure Costs

External Failure Costs

Quality activities associated with the cost

Activities designed to prevent faults throughout the development process, especially internal upstream

Activities to ensure conformance, especially internal downstream

Activities at upstream stages to self-debug before downstream stages

Activities at downstream stages to debug before release to customers

Activities to fix errors after release to customers

Typical breakdown of quality activities

Training

Management

Infrastructure

Preventive quality initiative

DFTS, including QFD, Robust Design, FMEA, poka yoke, customer appraisal, requirements, design and code review, customer appraisal and review (CAR), and training

Review

Inspect

Test

Evaluate

Verify

Validate

CAR

Corrective measures

Design

Review

Detect

Design and coding corrections

CAR

Corrective measures

Review

Inspect

Test and evaluate

Validate and verify

Coding and design corrections

CAR

Detect

Corrective measures

Returns

All internal activities for the first four stages

CAR

Detect

Corrective measures during integration, expansion, and maintenance

Complaints compliance

Expedited delivery

Legal issues


Cost-reduction potentials are realized only if quality-related activities are based on clear analysis and understanding of root causes of faults, which a CoSQ analysis helps reveal. A CoSQ quality matrix points out quality problems and associated costs, as shown in Table 4.3. It is important not to get into a futile discussion about a particular item's cost categorization. Certain items could be considered either a development activity or a quality-related one. It is best to exclude debatable items in CoSQ reporting. Similarly, a cost item may belong to one category or another, depending on a particular perspective. However, major cost items are not difficult to categorize. In all such cases, it is best to be consistent so that improvements and opportunities for improvements are clearly identified.

Table 4.3. Establishing a CoSQ Reporting System

Phase Quality Tasks

Requirements Costs

Design Costs

Code Costs

Test, Evaluation, Validation, and Verification Costs

Integration Costs

Maintenance Costs

Total Costs

Explanation of Cost and its Implications

Prevention Costs

        

Quality planning and management

        

Information systems for quality

        

Training for quality

        

Quality initiatives such as DFTS (QFD, TRIZ, Pugh, risk analysis, Taguchi Methods)

        

Design review

        

Customer appraisal and review

        

Any other preventive measures

        

Appraisal Costs

        

Inspect

        

Test and evaluation

        

Validate and verify

        

Equipment for the preceding

        

Design review

        

Risk analysis

        

Customer appraisal and review

        

Any other appraisal measures

        

Upstream Internal Failure Costs

        

Detect design/coding correction

        

Risk analysis

        

Customer appraisal and review

        

Any other upstream failure-related measures

        

Downstream Internal Failure Costs

        

Test and evaluation

        

Detection and coding/design correction

        

Risk analysis

        

Design review

        

Validate and verify

        

Customer appraisal and review

        

New release

        

Any other downstream failure-related measures

        

External Failure Costs

        

Returns

        

All internal activities in the first four stages

        

Complaints

        

Expedited delivery

        

Legal issues

        

CAR

        

Corrective measures (during integration, expansion, and maintenance)

        

New release

        

Any other external failure-related measures

        

Total Costs

        


CoSQ reporting is the first step in identifying the root causes that need to be addressed by the software developer and outsourcing firms if they are involved. Various tools such as process mapping, cause-and-effect diagrams, and brainstorming can be used to identify the root causes (see Chapter 6).

Establishing a CoSQ Reporting System

The first requirement of a CoSQ program is to establish a reporting system that identifies costs incurred due to poor quality. Establishing such a reporting system is a major initiative and must be carefully planned. Evans and Lindsay[11] have summarized a 12-step implementation plan based on the recommendations of the Institute of Management Accountants:

1.

Obtain management commitment and support. The ideal milieu in which to launch such a system is created only if it's initiated by top management. If it's prompted by accounting or quality personnel, it is wise to secure management commitment. An estimate of the cost of poor quality and its sheer magnitude may encourage management's commitment and support. It is advisable not to embark on such a system without management support. It is also critical that accounting executives support and "own" the system. The accounting department should be involved in and responsible for CoQ operations for several reasons: They have access to primary data, they are used to evaluating processes in monetary terms, they are seen as neutral, and their results are trusted. Moreover, their involvement ensures that they will be active participants in the process to make it work.

2.

Establish an installation team. A credible team is essential in establishing such a system. It should consist of software professionals, test and sales engineers, quality professionals, and an accounting cadre. It should be headed by a senior accounting executive as champion. It is important to provide the best resources and personnel for such an initiative. If it is part of a DFTS initiative, a trained black belt who is well-versed in CoSQ and DFTS may be an excellent candidate to coordinate this assignment.

3.

Select an organizational segment as a prototype. It is best to launch such an initiative as a prototype in a small software project in the early planning stages. It can be launched as a stand-alone initiative, but the payoff is substantially higher if it is part of a larger initiative, such as DFTS.

4.

Obtain the cooperation of users and suppliers of information. It is critical that both users and suppliers of CoSQ information have a stake in the initiative's success and are adequately represented on the implementation team. Management should provide encouragement and incentives to ensure its success.

5.

Define quality costs and quality cost categories. It must be understood that quality data is secondary or derivative data, not the primary fiduciary data. It is important that various costs are defined properly (see Tables 4.2 and 4.3) and are clearly understood. Explanations should be provided for various conformance and nonconformance costs and their implications. Such documentation identifies necessary follow-up.

6.

Identify quality costs within each category. A brainstorming meeting of users and providers of information is an effective way to recognize the costs incurred at various phases of the software life cycle. The objective is not to seek consensus but to come to agreement with a view to identifying quality cost items correctly.

7.

Determine the sources of quality cost information. Certain data may not be readily obtainable or may not be useful in the form available. In such cases, considerable effort may be needed to uncover quality costs from aggregate accounting data. But a good estimaterather than rigorous accuracymay be all that is required at this stage. Accuracy of data collection as well as estimates improve as the system is implemented, used, and documented correctly over a period of time.

8.

Design quality cost reports and graphs. Reports and graphs should be user-friendly in terms of details, the type of information needed, and timely availability. They should be accurate and have a simple presentation. In this regard, the value of appropriate visual aids cannot be overemphasized.

9.

Establish procedures to collect quality cost information. Best practices from within and across industries should be identified. Involvement of accounting professionals can be extremely useful. Only individuals who understand the process of data collection and who have been trained in the CoSQ process should be responsible for collecting cost information. Proper procedures should be established and supported by adequate forms, personnel, and computer systems.

10.

Collect data and prepare and distribute reports. The team should ensure that this step is carried out in a manner that is based on a sound reporting system developed as a result of following the preceding steps diligently. By doing so, the team will inspire confidence and prepare the organization to use data and facts to manage its quality initiatives.

11.

Eliminate bugs from the system. The whole system should be monitored by running trial runs. People must learn and develop confidence in collection and use of data. The issues related to reliability of data should be addressed. The focus should be on the value of data from the users' perspective, with a view toward helping them improve the quality of the software development process.

12.

Expand the system. As soon as the system is up and running, this practice can be expanded to other segments of the organization in a planned manner. It is important to maintain proper records for the existing reporting system so that best practices can be established internally. The reporting system should be evaluated regularly for continuous improvement.

Payback from Investment in Quality

Quality initiatives are actually investments in future benefits and cost savings. They are like investments in preventive health care with a view to enjoying the benefits of good health (the quality). They also prevent catastrophic health-care expenses (costs) that may have to be incurred if prevention is ignored. The benefits or savings to be realized are not always apparent. For, example although certain internal cost savings are identifiable, costs of external failures associated with regulation, litigation, bad publicity, loss of customer or market share, and bankruptcies are difficult to estimate. Equally, the impact of quality on future revenue and business growth is difficult to estimate. Dollar estimates are just thatestimates. Being obsessed with absolutely measurable costs and benefits is counterproductive to any quality initiative. This mind-set of absolute accuracy and measuring everything often becomes a major irritant. If past records are available, they should be used. But it is impossible to get all the data on costs and benefits. It is best to take a practical approach toward these matters: make a conservative estimate of costs and benefits, and take appropriate measures upstream to prevent external failures. As a CoSQ system is established and up and running, such data is available more readily.

The payback from investment in various quality tasks can be expressed as follows:

Payback from quality investments = benefits realized investments in quality tasks

The Return on Quality Investments (ROQI) is calculated as follows:


Discounted Cash Flow Approach

ROQI normally is discounted using Net Present Value (NPV)/Internal Rate of Return (IRR) analysis. Another important Discounted Cash Flow (DCF) metric is the payback period. ROQI with discounted return is


Value of CoSQ Analysis

Quality initiatives very often emanate from quality professionals. Management may not be fully aware of the benefits to be realized. A good CoSQ analysis invariably reveals surprising, even shocking, facts to upper management. They are likely to understand and respond to such revelations because CoSQ is expressed in the language they understand bestdollars. It uncovers the cost of not paying attention to quality and helps identify activities that need to be addressed. A CoSQ metric program thus plays a critical role in successful execution of a DFTS initiative from both quality and cost perspectives.

It may be added that normally an appropriate quality task saves more than it costs. At times, a quality task may cost more than it saves, but it may satisfy a critical customer requirement. Customer requirements and not cost alone should be the determining factor in initiating a quality task. Equally, customer requirements may necessitate cost, quality, and schedule optimization, a basic precept of DFTS. The important thing about DFTS is that it helps you understand customer requirements throughout the software life cycle. Such understanding of various customer requirements provides the need and opportunity for robust optimization. We discuss this in Chapters 16 and 17.

Pitfalls of a CoSQ Program

A CoSQ initiative is too critical to be allowed to go astray. It is therefore important to be aware of the following potential pitfalls:

  • Lack of management support: Ideally the initiative should be initiated by top management, with specific objectives of improving quality and customer satisfaction. If it is initiated by quality professionals or the accounting department, they must make sure that top management is committed to it. Without management's support, the initiative is likely to fail and should not be attempted.

  • Treating the CoSQ data as financial data: CoSQ is not regular financial data that is prepared for financial reporting. The quality of CoSQ data is determined by whether it leads to addressing the root causes of failures effectively rather than by their absolute accuracy. Absolute accuracy of cost of quality is difficult to ensure and may be costly because many intangibles exist. Furthermore, you're not looking for absolute accuracy.

  • Emphasizing cost reduction, not customer requirements: The emphasis should always be on improving product or process quality, and meeting customer requirements and not treating CoSQ primarily as a cost-reduction exercise.

  • Using the initiative for performance appraisal: The initiative should not be used as an instrument for personnel evaluation, because it is not. It will end up giving you wrong information if used for that purpose. As a rule, it is precarious to use any kind of quality management data for individual performance appraisal. Effective quality improvement is an organizational process and is the outcome of many cooperating individuals. Rewarding them individually results in local optimization. Further, as Deming has said, performance evaluation, merit rating, and annual reviews have devastating effects on individuals and quality.




Design for Trustworthy Software. Tools, Techniques, and Methodology of Developing Robust Software
Design for Trustworthy Software: Tools, Techniques, and Methodology of Developing Robust Software
ISBN: 0131872508
EAN: 2147483647
Year: 2006
Pages: 394

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net