| The most common financial evaluation emphasizes just the dollar savings or ROQI following the introduction of a quality initiative, as discussed earlier in this chapter. This is absolutely inadequate, even from a financial perspective. Due emphasis should be given to other financial metrics, such as cash flow, sales growth, productivity gains, and Return on Equity (ROE), that quality improvements create in successive years. Equally important are nonfinancial metrics with regard to customer satisfaction, process improvement, and leadership development that are critical in creating sustainable value. A set of financial and nonfinancial metrics should be used for evaluation in an ongoing manner, rather than using just ROQI following the initial deployment. Sound development practice over a period of time creates sustainable results. The improved results are also made possible by addressing root causes of deficiencies that robust nonfinancial metrics reveal. So both are important: financial and nonfinancial metrics, one no more than the other. Such an approach based on Kaplan and Norton's balanced scorecard leads to improved decision-making and problem-solving.[18] Metrics for DFTS Evaluation The following metrics can be used to evaluate the DFTS intervention in various stages of the software life cycle: Financial performance: ROQI, CoSQ, cash flow, sales growth, profitability, lifecycle cost, market share, productivity, ROE Customer satisfaction: Software trustworthiness (including reliability, safety, security, maintainability, and customer responsiveness, as well as cost and delivery time) Software process capability: Improvement in understanding user domain, dealing with software complexity, schedule deviation, delivered defects, test defects, final testing time, user-reported defects, development cost, process predictability Leadership and quality of work life (QWL): Individual capability, team and leadership development, hiring and retaining talent, innovation capability, QWL
These measures are obviously interrelated. A broad evaluation enables sound performance monitoring, which helps you initiate appropriate corrective measures with a view to improving the DFTS process. In the absence of nonfinancial metrics, organizations may not be able to measure the impact on customer satisfaction, process improvement, and leadership capability. Many organizations get into the trap of just chasing financial returns following a quality initiative, without ascertaining its impact on other aspects. Such an approach inevitably leads to missed opportunities. Establishing a Financial Evaluation Framework for a DFTS Initiative The following list discusses some broad issues for financial evaluation of a DFTS intervention that you must keep in mind: Obtain management commitment and support. The top management team, including the CEO, CFO, and senior executives in charge of software development and DFTS, must support the establishment of financial metrics for the software development process. The value of establishing various financial metrics should be obvious to them before they commit resources and use them in decision-making. Do not use just ROQI. ROQI plays an important but essentially limited role. First, it helps in the initial decision-making to introduce DFTS. Second, it identifies the initial investment. Because initial investments are relatively small, organizations need to consider other financial objectives, such as growth and market share, that may be strategic. Other financial metrics also need to be considered in financial evaluation: cash flow, profitability, life-cycle cost, productivity, ROE, and CoSQ. Establish the cost of various quality-related activities in the software development process. Most software organizations do not have them, but it is essential to identify them to create useful and reliable financial metrics. Establish a financial evaluation team. The evaluation exercise should be headed by an accounting professional and should be adequately supported by the DFTS black belt, master black belt if available, DFTS champion, and software developers who are participating in the initiative. It is important to provide the best resources and personnel for this task. Provide training and incentives. All these personnel need to undergo training as part of the DFTS implementation process. (Chapter 5 discusses training and incentive issues.) Select financial measures and the DCF approach. Decide which of the financial metrics you will use. For ROI, select the appropriate Discounted Cash Flow (DCF) approach between IRR, NPV, and payback period that will be applied.[19] These require that future savings or revenue streams be forecast and a system for tracking costs be established. Life-cycle costs (see Figures 4.3 and 4.4) should be taken into account, not just internal development costs. Estimating various savingsespecially on failure costs, one of the main areas of cost savings and quality improvementis not easy. You must use historical data whenever possible. It is important to be conservative on estimates of future savings. You will have an opportunity to revise the estimates as the project progresses (which should be done in any case). Start with a small software project. It is advisable to launch a DFTS initiative on a small software project. This approach lets you introduce change without major disruption. (However, a roll-out plan for expanding DFTS to the rest of the organization should be carefully developed and communicated. All important development and learning should be documented for future learning and communicated to sustain interest and support.) Integrate the CoSQ team. If a CoSQ team exists, it may be advisable to integrate them in one unified team to establish financial information required for DFTS. Care should be taken to ensure that the team does not duplicate work and compile data that is readily available elsewhere. Identify savings and costs. Determining the savings from a DFTS intervention is an important step in the evaluation process. The net savings in an activity from DFTS can be calculated from the following equation: net savings in an activity = costs without DFTS costs with DFTS Here all the direct and indirect recurring costs are identified. If historical and valid data is available, costs without DFTS can be established relatively easily; otherwise, an estimate of various costs should be made. It may be easier to cost certain activities than others. The all-important external failure costs are the most difficult to estimate. These include costs that have not yet been encountered but that are plausible. It is advisable to estimate on the worst case. Monitor and improve. It is important to monitor for accuracy of data and value and validity of metrics from user perspectives. Steps should be taken to improve the cost and quality of data collection and presentation. A tracking system should be established to keep records. Expand the system as needed. After the system is debugged and documented, it can be expanded to the rest of the organization systematically.
Chapter 22 illustrates an excellent CoSQ program at Raytheon's Electronic Systems (RES) Group. It captures the essence of a sound CoSQ initiative and is highly recommended reading for software developers and quality professionals. Chapter 3 discussed software quality metrics. Chapters 5 and 20 cover individual, team, and leadership capability metrics. Chapters 6 through 21 discuss various software trustworthiness issues. They all contribute to delivering trustworthy software. A software organization embarking on a quality initiative must rely on a balanced set of financial and nonfinancial metrics covering various spheres. Many quality issues do not lend themselves to quantitative evaluation. "Everything must be measured before we move" is a wrong mind-set. Often, sound estimates may be just fine. The idea that a sound process creates value by itself should not be ignored. Most significantly, the prime focus should always be on meeting customer requirements. In the absence of that, financial evaluation may not even be carried out. In any case, you need nonfinancial metrics to understand and respond to customer needs, improve the software development process, and fortify your organization's software development capability. A sole focus on financial results is inevitably dysfunctional. As Walter A. Shewhart said, "Price without quality has no meaning." Such clarity and the implied self-discipline are essential for an organization embarking on a robust software development process. Without such commitment, it is difficult to see how the much-needed internal transformation for customer and process focus can be pulled off. |
