13.5 A Network-Based Measurement System

It should be eminently clear, at this point, that measuring the evolution of a software system is a very complex process. There is an astonishing amount of data that generates from this measurement process. Teams of software developers are constantly modifying the code base. Testers are continually testing the evolving systems. Requirements are constantly changing. For a large system, the total data flow can easily amount to several megabytes per week. These data must be organized and managed in a database so that they can be converted into information for software development managers, software test managers, and software quality assurance staff.

We have developed a system called Darwin, after the father of evolution, that is designed to manage the measurement data for evolving software systems. The Darwin system is a very good example of how a measurement system can be integrated into a software development system. Darwin is a software engineering management system for tracking the progress of evolving software systems. It is a repository for all of the engineering data surrounding software development and software test. It is a Web server that permits developers and testers to interact with the system. It is an analytical tool that provides the measurement capabilities for both software development and software test. All management aspects of the software development and testing process are to be maintained by the Darwin system.

There are two criteria that must be satisfied for the measurement program to be a success. First, the measurement process must be automated. Second, it must be made invisible. Software developers have little or no experience with static software measurement. If they are confronted with the measurement, they will simply fail to do it. Software testers have little or no experience with dynamic software measurement. If they are also confronted with the additional task of measurement, they too will resist the additional effort.

At the core of the Darwin system is the software measurement database. All information surrounding the daily changes made to a system and the activities surrounding the test of that system will be recorded in the database. All measurement processes are automated and transparent. The Darwin system also provides a framework for software failure tracking and for software fault resolution. Finally, Darwin has an integrated software requirements tracking capability to aid in the regression test process.

13.5.1 Network Appliance

The Darwin software system was designed to reside in a network appliance (NA) computer attached to the local intranet connecting systems analysts, software developers, software testers, software managers, and software quality staff. The measurement database, the metric tools, and all the ancillary support software for Darwin are serviced from this machine. To interact with the system, a user will visit a Web site on the server and will then be linked through Web pages provided by the appliance to the appropriate application. This NA approach greatly simplifies the problem of instituting a measurement program, in that it obviates the need to port the software measurement infrastructure from one platform to another.

Different people in the development group will interact with the NA in different ways. A software manager will query the database to retrieve the latest status of the evolving software system. A build manager will inform the NA of the modules and versions of the modules for each of the builds. A software tester will retrieve test cases for the test process from the NA. A test manager will query the test database for the status of the test process to date.

Of greatest importance to the entire development process is the cogency of the information. A development manager can see the status of the development process essentially in real-time. Specific reports on change activity can assist in the process of setting milestones and reviews. A test manager can evaluate the status of a test process, again in near-real-time. The NA is a visible aid to developers, managers, and testers. All of the engineering data about the evolving software is managed from this location.

13.5.2 The Measurement Infrastructure

On the static measurement side, Darwin is designed to track changes to code and faults reported against specific code modules. The measurement of specific changes to the source code system can be a very complex process, in that a typical large software system may have many developers working with the same code base at the same time. It is possible to track changes to code at the individual version level. However, experience has shown that this kind of resolution is seldom, if at all, necessary. What is of primary interest to developers and managers is the code that constitutes a build.

At the point of a new build, a build manager will develop a new build index vector. This build index vector will determine which source code elements actually go to the build. On completion of the build index vector, Darwin can be requested to measure the source code for the build. Currently, the Darwin NA can supply measurement tools for C and C++ with the Extended C Metric Analyzer (ECMA), or for Java with the Java Metric Analyzer (JMA). The measurement tools are a component of the Darwin NA.

With the contents of the build index vector, Darwin will systematically retrieve the appropriate source code elements from the configuration control system with the updated versions of this code. Each source code module will then be measured by the appropriate measurement tool and the raw metrics for that module will be incorporated into the database. It is important to observe that the only modules that are measured by the system are those that will result in executable code. All header files and compiler directives will have been resolved before the measurement process begins.

Just as important as the process of measuring code is the process of maintaining specific fault reports. Each fault report will report on exactly one fault on one managed document. These fault reports are served from the NA as Web pages and are tracked by Darwin. For each change to a system, there can be exactly two reasons for the change: a fault has occurred or there has been a change in the program specifications. Darwin must track all program document changes relative to faults.

On the dynamic measurement side, Darwin will track failure reports and test execution profiles. As we are working on a developing system, faults will be introduced. Some of these faults will result in the failure of the system. This failure will be recorded in the failure reporting system. The specifics of this report will vary from one installation to another but will typically contain information similar to that listed below in the Failure Report Relation for a typical Darwin database. As the failure report is examined and mapped to one or more specific problems in the requirements, design, or code, the appropriate fault reports are completed. Each failure report is resolved by the creation of one or more fault reports.

To monitor the code when it is executing, some languages (such as C and C++) will have to be instrumented with software probes prior to their test so that their execution can be profiled. Other language environments, such as Java, already support profiling as a built-in feature of the runtime environment. The Darwin NA currently has the capability of instrumenting both C and C++ with our Clic toolset. It also can link to and receive information from the Java Virtual Machine Profile Interface.

From a test outcomes measurement standpoint, the result of a test is a test execution profile. As each test is completed, the test measurement database is updated with the test execution profile and the information surrounding the test, such as the test case number, the tester, the test date and time, etc.

The entire Darwin NA measurement infrastructure was designed to make the measurement process both automatic and invisible. We have listened to the sad stories of legions of software developers, software development managers, and software testers. Their stories seem to indicate that they are some of the world's most harried employees. The mere thought of adding incremental responsibilities to their work list brings instant resistance. Software engineering measurement will not occur in this environment unless it is made painless and transparent.

13.5.3 The Software Engineering Maintenance Problem

The Darwin NA plays another significant role in the software development environment. It also serves a maintenance function. The NA will serve as a repository for the software requirements and also for the test plan and specific test cases.

There are two distinct classes of requirements that must be separately maintained. First, there are the operational requirements. These address what the software will do from a user's perspective. These operational requirements are a mutually exclusive set of distinct operations that the system is capable of performing. The second set of requirements includes the functional requirements or the design requirements. They address how the software will perform each of the operations. Again, this is a set of mutually exclusive functionalities that express exactly the activities that the system will perform to implement each of the operations.

The most important aspect of the requirements database is that each of the operational requirements is linked to the functionalities that will implement it. Further, each of the functional requirements is linked to the program modules that will implement it. This linkage will ensure complete requirements traceability. Starting at any one operation, it is possible to know what program modules actually implement that operational requirement. Similarly, beginning at the module level, it is possible to identify all of the functionalities that invoke that module and all of the operations that invoke those functionalities. With Darwin, we can easily identify the modules that have had the greatest code churn between builds. To know that fact and to be able to design a test case that would specifically test those modules is something very different. With the requirements traceability made manifest in the requirements database, we can easily develop test cases around the operations that will cause those modules to be expressed.

The second aspect of maintenance that is also performed within the Darwin database is the maintenance of the test cases. As each test case is designed, it is placed into the database. This gives a complete overview to everyone in the development and test organizations as to the completeness of the test case design. Further, as each test case is executed, the evaluation of that test case from the standpoint of functional coverage, test effectiveness, and test efficiency can be evaluated instantly.

13.5.4 The Darwin Reporting System

The Darwin system is equipped with a set of standard SQL generated reports that reflect the more typical database inquires that will be made by most managers. Darwin tracks changes in source code across builds. We have shown that change activity in the examples of the Space Shuttle PASS (Primary Avionics Software System) and the Cassini project. Software systems can be characterized for reporting purposes in terms of changes to the structure of the code in the attribute domains and also in terms of the fault burden of the system as measured by the system fault index (FI), code churn, and code deltas. This gives us considerable insight into where existing faults might be and continuing processes that are potentially introducing new faults. These data are useful for prioritizing the software review processes. We can devote our software review team energies to those regions of the code where the greatest problems are likely to be. In a sense, this is like one hand clapping, as indicated earlier.

The dynamic code measurements from the test activities give us a good, clear idea of how the test activity is distributed across the modules that comprise the system. In conjunction with the functional requirements, it will provide solid information as to the extent of coverage of the set of functional and operational requirements in the system to date. It will also give some limited insight into the ability of the test activity to enter regions of code where faults might be. This dynamic measure assessment is the other hand clapping.

When we combine the static measurement capability with the dynamic measurement capability in the report generation system of the Darwin NA, we will have both hands clapping. The static measures identify the potential fault problems; the dynamic measures indicate what code has executed. Together, they tell us how well the test activity has focused on the problems that are likely to be in the code.

Identifying the extent to which the potential faults have been exposed is but one part of the problem. It is all well and good to learn that modules 27, 105, and 231 were fault prone and were not executed; it is quite another to know what it will take to execute them. With the requirements traceability features built into the Darwin database, the answer can be developed from a simple SQL statement. This is so because functional specifications are linked to specific program modules. Operational specifications are linked to functional specifications. It is a very simple process to construct a new test suite that will operationally exercise modules 27, 105, and 231.

A case does not need to be made for the reporting capabilities of the Darwin system. This reporting capability is inherent in any database system. What is important is that the Darwin NA contains the relevant information to make informed decisions about the software development process and the software quality/testing process.

13.5.5 The Darwin Database

The structure of a schema for a generic database for Darwin is outlined in Exhibit 10. The strong influence of development work on this database at JPL is evidenced by the module naming conventions. This is not intended to be a universal database for all potential applications. It can easily be tailored to suit the needs of individual organizations.

Exhibit 10: Generic Database for Darwin

 DOMAINS People    Name      Last      First      MI    Employee Code    Position Code    Activity Code Process    Event       Failure          Failure ID          Severity          Priority          Apparent Cause          Resolution       Fault          Location          Functions Affected          Etiology of Error          How Discovered          Fault Classification          Fault Resolution Code       Change          Change Request Number          Modules Affected          Reason          Design Element    Staff Hours    Software Process Methodology    Testing       Test Suite       Test Iteration Product    Software Module Name       CSCI       CSC       CSU    Operational Requirement Number    Operational Requirement Text    Functional Requirement Number    Functional Requirement Text    Build Number    Version Number    Static Metrics       Metric Primitives          Comments          Executable Statements          Nonexecutable Statements          Total Operators          Unique Operators          Unique Nonoverloaded Operators          Function Operators          Total Operands          Unique Operands          Nodes          Edges          Paths          Maximum Path Length          Average Path Length          Cycles          System Fan-Out Explicit          Unique System Fan-Out Explicit          C++ Library Fan-Out          Unique C++ Library Fan-Out          System Fan-Out Implicit       Derived Metrics          Fault Index          Code Delta          Net Fault Change    Dynamic Metrics      Module Frequency Count Environment    Building    Work Area Organization    Department    Project    Group Time    Date       MM       DD       YY    Time       HH       MM       SS RELATIONS Operational Requirement    Operational Requirement Number    Operational Requirement Text    Implemented_by_Functional_Requirement Vector (Array    [1..m] of Functional Requirement Number) Functional Requirement    Functional Requirement Number    Functional Requirement Text    Implemented_in_Module Vector (Array [1..m] of Software    Module Name) Operational Profile    Operation (Array [1..m] of Profile) Failure Report    Program (CSC)    Build Number    Tester (Employee Code)    Date    Time    Test Suite    Test Iteration    Failure       Failure ID       Severity       Priority       Apparent Cause       Resolution Module Fault Resolution    CSCI    CSC    CSU    Version Number    Fault Classification    Failure ID    Fault Resolution Code Module Measurement Element    CSCI    CSC    CSU    Version Number    Metric Primitives    Fault Index Module Descriptor Element    Module Name (CSU)    Program Name (CSC)    System Name (CSCI)    Developer (Employee Code)    Start Date (Date)    Finish Date (Date) Module Change History    Module Name (CSU)    Change Request Number    Date    Developer (Employee Code)    Developer Time (Staff Hours)    Unit Tester (Employee Code)    Test Time (Staff Hours) Module Fix History    Module Name (CSU)    Fault Resolution Code    Date    Developer (Employee Code)    Unit Tester (Employee Code) Test Metric Report    Program (CSC)    Build Number    Test Suite    Test Iteration    Date    Module Frequency (Array [1..n] of Module Frequency Count) Baseline Build Data    Build Number    Program (CSC)    Date    Mean Vector (Array [1..m] of real)    Std Vector (Array [1..m] of real)    Eigenvalues (Array [1..k] of real)    Transformation Matrix (Matrix [1...k;1...m] of real) Build Metrics    Program (CSC)    Module (CSU)    Build Number    Code Delta    Net Fault Change Build Modules    Module Name (CSU)    Build Number    Version Number Developers    Name       Last       First       MI    Employee Code    Position Code    Activity Code Testers    Name       Last       First       MI    Employee Code    Position Code    Activity Code Change Request    Change Request Number    Date    Reason

There are two parts to the schema. The first part, the Domains, defines the attributes that we will keep in the database. The Relations show how the entities will be created to convert the flow of data into the database into meaningful information. In each relation the key fields are underlined. This sample schema is the underlying framework for a Darwin system currently in operation at Sun Microsystems and at the JPL.