Diagnosing Evolution and Test Infection


Based on the structural observations derived from the 254 integration versions, we can make statements about the system's evolution and the influence on this evolution of the techniques used during development. We have extracted information about the following subset of system properties, which is an adaptation of the descriptions found in [Mattsson+1999].

  • The size of each package and subpackage is the number of classes it contains. Only top-level classes (no inner or nested classes) are used because they reflect the behavior of the system to the outside world.

  • The change rate is the percentage of classes in a particular package that changed from one version to the next. To compute the change rate, two versions of a class are needed. The relative number of the changed classes represents the change rate.

  • The growth rate is defined as the percentage of classes in a particular package that have been added (or deleted) from one version to the next. To compute the growth rate, two versions are compared, and the numbers of the added and deleted classes are computed. The relative number of the new classes (that is, the difference between added and removed classes) represents the growth rate.

  • The test coverage rate is the percentage of classes that are covered by test classes. Given the convention to name a test class with the suffix "Test," we can count the number of test classes for a given package. The test coverage rate is the number of test classes divided by the number of system classes in the package subtree (that is, the number of classes without the test classes).

One important detail for calculating the system properties is the way we deal with package restructurings. Given a class in version n of the system, we first look for that class in version n 1 with exactly the same package qualifier. If this class is not found, we look for the class in version n 1 without the package qualifier. Because of the nature of the development method of making small iterations and increments, we are likely to find those classes that are only moved to another package but not renamed. Now we can identify the predecessor of a given package in version n 1 by looking for the packages from which the classes in the package of version n originate.

System Observations

Figure 15.1 shows the historical development of the size of JWAM for the 254 observed integration versions. Around version 19, you see an irregularity of shrinking framework size. Here, a library that had once been part of the framework was deleted completely. At integration version 33, another outdated library was deleted. The other exceptionally high change in the system size is at integration version 219, where a large number of old test cases and old examples were integrated at once. Except for those singularities, a more or less linear growth of the framework's size can be seen over the observed period.

Figure 15.1. Overall size

graphics/15fig01.jpg

Mattsson and Bosch see a linearly growing size as a sign of the maturity of a framework. The change rate and growth rate of such a system should fall more or less linearly. They explain any nonlinear behavior in the change rate of the overall system as a major architectural change (that is, the introduction of online capabilities into a batch-oriented system). In JWAM, the nonlinear change rate and growth rate curves stem from the fact that we observed every integration version of the system not only the released versions. Thus, in the context of investigating test-infected code, we are more interested in the frequency of the change and growth rate peaks than in their absolute height.

Figure 15.2 shows the change and growth rate history for the whole system. Compared with the size history, this diagram shows the individual development steps (the dynamics of the development process). A change rate average of 0.005 indicates the iterative development in small steps.[5] A growth rate average of 0.002[6] indicates the incremental[7] development in small steps. A general observation is that a change occurs more often than growth in the system. This indicates that after or before the addition or deletion of new methods or new classes, some refactoring steps are performed. This matches perfectly the test-infected development style.

[5] A change rate of 0.005 means that five classes are changed, given an average overall system size of 1,000 classes.

[6] A growth rate of 0.002 means that two out of 1,000 methods are added or deleted.

[7] Cockburn distinguishes between incremental and iterative development. "Incremental development is a staging strategy in which portions of the system are developed at different times or rates, and integrated as they are ready. . . . Iterative development is a rework scheduling strategy in which time is set aside to revise and improve parts of a system" [Cockburn1998].

Figure 15.2. Overall change and growth rate

graphics/15fig02.jpg

In this study, we introduced the system property test coverage rate. This can be seen in Figure 15.3 for the overall system. Except for the singularities explained in the discussion of the size history for the overall system, the test coverage rate history exhibits more or less constant growth. For example, starting with a test coverage rate of less than 0.2, it reaches more than 0.5 at the end of the observed period.[8] This is an indication of the growing maturity of the application of the techniques for framework development.

[8] A test coverage rate of 0.2 means that only two out of ten system classes are covered by a test class.

Figure 15.3. Overall test coverage rate

graphics/15fig03.jpg

A test coverage rate of about 0.5 doesn't seem to be very sophisticated for a development process that claims to be a form of XP, because to be successful, XP requires a very high test coverage rate. As shown in Figure 15.4, we have to differentiate between the different top-level packages to make a more qualified diagnosis. This figure shows the test coverage rate for the individual top-level packages.

Figure 15.4. Test coverage rate of top-level packages

graphics/15fig04.jpg

The framework core in de.jwam exhibits more or less healthy growth in the test coverage rate, from below 0.7 to 1. This fits with the nearly ideal size history for the framework core shown in Figure 15.5. For the de.jwamexample package, the development team seems to have realized the importance of test cases for the examples during framework development.

Figure 15.5. Size of top-level packages

graphics/15fig05.jpg

Starting from a test coverage rate of 0, it ends at 0.4. At the same time, the package de.jwamexample is the only other top-level package that reveals nearly linear growth, as shown in Figure 15.5. The development team seems to have understood the value of a suite of examples and the importance of ensuring their high quality. Application developers could base their development on the examples provided along with their associated test cases.

The developers seem to have realized the importance of a test suite for the de.jwamx package (containing additional components on top of the framework core), which exhibits a growing number of test cases. But the number of test cases still didn't reach a level that could ensure healthy behavior in the size history curve for this package, as can be seen in Figure 15.4. The packages de.jwamalpha and de.jwamdev have by far the worst test coverage rate history. The developers obviously didn't see the need to put the same amount of effort into the evolution of their development tools as they did for the rest of the framework. The de.jwamalpha package is planned to be a test area for new ideas. In XP terms, these new ideas are spike solutions that do not have to be developed with the same care as the rest of the system. Development of a proper test suite for the new ideas is deferred until those ideas are incorporated into the base system. This is completely valid for XP and does not exhibit a fallacy in the development process. Figure 15.5 confirms this behavior for the size history of the two packages.

The differences between the top-level packages can also be traced in the change rate history shown in Figure 15.6. For the framework core, the development seems to be very close to the ideal: Many small peaks with a high frequency indicate many iterative steps. In the other top-level packages, the change rate peaks are less regular. Here, the development is performed in fewer and bigger steps.

Figure 15.6. Change rate of top-level packages

graphics/15fig06.jpg

Diagnosis and Recommendations

Generally, the development team seems to be on the right track for applying the two techniques. The analysis of the code shows the positive effects of building and maintaining a good test suite on the evolution of the system. The iterative and incremental development steps are most clearly seen in the most mature parts of the system. The distribution of effort in the development resources seems to be effective in the sense that a complete test suite is maintained only for those parts that have to be of high quality (that is, the framework core). The experimental parts of the framework (that is, the alpha package) have a poor test coverage rate, a nonlinear size history, and a few big change and growth rate peaks (as opposed to many, equally distributed small peaks for the framework core).

It is a good idea to provide framework users with a set of examples that come with a whole suite of tests specific to those examples. These examples could show the users how to write tests for typical uses of the framework and could help improve overall quality.

However, the analysis also exhibits some possible weaknesses in the system development. The size history of the overall system shows some nonlinearity. This seems to indicate large steps and big changes in the development. That would lead us to conclude that these steps did not happen in an XP-like style. But the current system in use demonstrates that the method used in this chapter also has its weaknesses. The changes mentioned affected a lot of classes, but they were not "big" changes in the sense of XP. The changes were made in a few minutes, maybe one hour, and did not affect many other parts of the system. So we would say they were not big or complicated changes. This "quality of the change" is not measured by the measurement method used here.



Extreme Programming Perspectives
Extreme Programming Perspectives
ISBN: 0201770059
EAN: 2147483647
Year: 2005
Pages: 445

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net