INTRODUCTION OF REVIEWS AND THE NAH SYNDROME

10.4 INTRODUCTION OF REVIEWS AND THE NAH SYNDROME

Reviews are, in many ways, counterintuitive. A programmer cannot understand how a review by a group of people can be more effective than testing. When human effort is the most critical resource in a project, it is not easy to accept the position that the highly manpower-intensive review process can make the overall process more productive and improve quality. As a result, convincing people to use reviews is one of the most difficult process deployment tasks. One SEI report indicates that only 22% of software organizations employ some form of inspections.⁸

Clearly, hard data are invaluable for proving the case. A fair amount of published data supports the claim that reviews can be cost-effective and can improve quality substantially. Nevertheless, such published data from organizations around the world often fail to convince engineers that reviews can be good for their organization. One reason for this skepticism is the Not Applicable Here (NAH) syndrome: People believe that reviews are good for other organizations but that the situation in their company is different and thus the reviews are not applicable.⁹

If inspections are to be deployed in a project, the NAH syndrome must be overcome. Managers as well as developers must be shown that inspections can provide real benefits. By definition, data from other organizations cannot be used to overcome the NAH syndrome. Instead, data from within the organization itself must be used to build a case for inspections. An experimental setup is therefore needed that can be quickly deployed in real-life scenarios to evaluate the suitability of inspections in the organization.

In an attempt to overcome the NAH syndrome, Infosys used a simple experiment, described in the next section. The experiment is general and simple and can be conducted easily by any organization. Further details of this endeavor appear in a paper I co-authored.⁹

10.4.1 The Infosys Experiment

Infosys's experiment focused on code inspections because developers relate to coding and it is usually the source of the greatest number of errors. Historically, inspections started with code and only later were extended to design, requirements, test plans, and other items. Once you build a strong case for code inspections and they are deployed, the advantages of these inspections will become obvious and will build a case for inspection of other work products.

For Infosys's experiment, six system enhancement requests (SERs) on a banking product were selected. Six developers were assigned one SER each. These developers were first trained in the inspection process; for practice, they were asked to inspect the implementation of an earlier SER that had been seeded with defects.

The personnel were divided into two groups of three developers. Each group formed an inspection team. Before submitting the SER, each developer was asked to implement the SER, compile his code, and do some self-testing. Once submitted, the SER went through two independent paths: inspections and unit testing. During the experiment, an inspection team inspected the code of the three SERs developed by the members of the group. In each inspection, the author, rather than the moderator or the reader, acted as an inspector. In parallel, the SERs were unit tested independently by the module leader for the domain to which the SER belonged. The flow diagram in Figure 10.4 shows the basic experiment steps.

Figure 10.4. Steps in the Infosys experiment

For each of the two paths, the effort spent and defects found were recorded. These data were then used to evaluate the cost-effectiveness and quality of the reviews. If the sets of defects found by these two approaches were not the same and if one set was not a subset of the other, then the claim could be made that inspections do find a different set of defects than unit testing and that adding inspections will be beneficial. Understanding the cost implications is more difficult (and is where most doubts arise). This cost was estimated based on past data for system testing.

10.4.2 Data from the Experiment

Table 10.5 gives the sizes of the SERs, the total effort, and the number of defects found in the two paths. Clearly, the inspection route identified more defects than did the unit testing route. This finding was consistently observed for all the SERs. Overall, inspections caught about 2.5 times as many defects as unit testing did, although inspections consumed more effort than testing did. The number of defects detected per person-hour, however, are similar for inspection and unit testing; both detected about 1.9 defects per person-hour.

Now let's look at the nature of the defects found by the two approaches, shown in Table 10.6. In almost all categories, inspections caught more defects than did unit testing, particularly for categories related to quality attributes such as maintainability, portability, and so on (as might be expected given that testing generally focuses on errors in functionality). The data also show that even in logic and interface defects (the focus of testing), inspections do better than unit testing. From these data, the case for adding inspections to improve the error detection capability is clear and convincing.

Table 10.5. Effort and Defect Data
	Inspections			Unit Testing
Size SER	Total Effort		Total Number of Defects	Total Effort	Total Number of Defects
	(LOC)	(Hours)		(Hours)
1	968	8.0	8	2.0	4
2	432	5.0	8	1.5	3
3	85	4.0	4	1.5	1
4	667	6.5	26	1.5	7
5	50	12.5	3	1.5	0
6	408	2.5	5	2.5	5
Total	2,610	27.5	54	10.5	20

 

Table 10.6. Defect Distribution
Defect Type	Inspections	Unit Testing	Common Defects
Data	3	1	0
Function	4	2	0
Interface	14	11	7
Logic	12	5	4
Maintainability	11	0	0
Portability	5	0	0
Other	5	1	1
Total	54	20	12

With these data, the cost per defect is about the same. However, reviews offer cost benefits. From past experience and data, it is known that it takes about 4 person-hours to identify and remove a defect during system testing. If a defect escapes system testing, it takes about 2 person-days (17 person-hours) to identify and remove it.

Testing rarely catches maintainability and portability type defects. We assume that all logic, interface, function, and data defects that are not caught by unit testing are found later. The number of such defects in Table 10.6 is 3, 4, 14, and 12, respectively. After eliminating common defects, which are also caught by unit testing, the number is 3, 4, 7, and 8. If all these defects are caught in system testing, then the system testing cost will increase by 22 * 4 = 88 hours, or about 11 person-days. If 75% of these errors are caught in system testing and 25% are caught later, the additional cost in system testing is 0.75 * 22 * 4 = 66 hours (9.5 person-days); the additional cost of fixing defects found later is 0.25 * 22 * 2.5 = 11 person-days. That is, if no inspections are done, an additional 20.5 person-days will be spent in fixing the extra defects.

Thus, the cost saving due to inspections is 11 person-days if all defects are caught in system testing and 20.5 person-days if 25% of the defects are not caught in system testing. The cost of the inspections, which yielded these savings, is about 3.5 person-days. The case is clear: If we spend one additional day in code inspection, for this product we can expect to save three to six days in defect fixing later in the development cycle!

The experiment lasted only about two weeks, but it had a substantial effect. The results convinced developers and managers alike that inspections should be tried. The data from the experiment also indicated that the inspections would offer fewer benefits if the code was simple or small (in smaller SERs, the benefit was unimpressive). The banking product team therefore made the policy decision to classify the SERs in three categories (simple, medium, and complex) and to consider formal inspections for all complex modules.