12.5 Study Design and Background Facts

---

In the study, RTI identified a set of quality attributes and used them to construct metrics for estimating the cost of an inadequate testing infrastructure. Two in-depth case studies were conducted, one in the manufacturing sector (transportation equipment) and one in the service sector (financial services).

For the analysis of transportation equipment industries, data were collected from 10 vendors of computer-aided design/manufacturing/engineering (CAD/CAM/CAE) and product data management (PDM) software, and from 179 users, primarily automotive and aerospace companies.

Approximately 60 percent of the automotive and aerospace manufacturers surveyed reported significant software errors in the previous year. Respondents who experienced errors reported an average of 40 major and 70 minor software bugs per year in their CAD/CAM/CAE or PDM software systems.

The total cost impact on these manufacturing sectors from an inadequate software-testing infrastructure is estimated to be $1.8 billion, and the potential cost reduction from feasible infrastructure improvements is $0.6 billion. Users of CAD/CAM/CAE and PDM software absorb approximately three-fourths of the total impact, with the automotive industry representing about 65 percent and the aerospace industry representing 10 percent. Software developers experience the remaining onefourth of the costs.

For the analysis of financial services, data was collected from four developers of financial electronic data interchange (FEDI) and clearinghouse software as well as the software embedded in routers and switches that support electronic data exchange, and from 98 software users, primarily banks and credit unions. Approximately two-thirds of the software users surveyed reported experiencing major software errors in the previous year.

Respondents that did have major errors reported an average of 40 major and 49 minor software bugs per year in their FEDI or clearinghouse software systems. Approximately 16 percent of those bugs were attributed to router and switch problems, and 48 percent were attributed to transaction software problems. The source of the remaining 36 percent of errors was unknown. Typical problems encountered due to bugs were increased person-hours used to correct posting errors, temporary shut down leading to lost transactions, and delay of transaction processing.

The total cost impact on the financial services sector from an inadequate software-testing infrastructure is estimated to be $3.3 billion. Potential cost reduction from feasible infrastructure improvements is $1.5 billion.

Software developers absorb about 75 percent of the economic impacts. Users experience the remaining 25 percent of costs, with banks accounting for the majority of user costs.

The annual cost to these two major industry groups from inadequate software infrastructure is estimated to be $5.18 billion. Based on similarities across industries with respect to software development and use and, in particular, software-testing labor costs, RTI projected the cost to the entire U.S. economy. Using the per-employee impacts for the two case studies, an extrapolation to other manufacturing and service industries yields an approximate estimate of $59.5 billion as the annual cost to the nation of inadequate software testing infrastructure.

Thus, if all software bugs could be identified and removed instantly (in real time), the combined economic benefits to the two industry groups and to the economy would be $5.85 billion and $59.5 billion, respectively. Realizing that such a “perfect infrastructure” is not attainable, industry experts were asked for estimates of a plausible reduction in delayed identifi-cation and removal of software errors. Based on this information, a “feasible improved infrastructure” scenario was constructed. For this scenario, software developers were asked to estimate the potential cost savings associated with enhanced testing tools, and users were asked to estimate cost savings if the software they purchase had 50 percent fewer bugs and errors. This improved infrastructure scenario is estimated to result in a combined annual benefit of $2.10 billion to the two industry groups studied, and $22.2 billion to the U.S. economy.

---