10.6 Instrumenting for the Profiles

The instrumentation techniques discussed to this point relate to the capture of execution frequency data. To capture execution frequency data, a vector equal in length to the number of modules (or functionalities) is allocated and initialized to zero. We will call this vector the execution frequency vector, EFC. Each time a module is called, a corresponding element in the EFC is incremented. We can monitor and record the entire operation of a system from the time it is initiated until it is terminated in the EFC. Alternatively, we can periodically save the contents of the EFC and reinitialize its elements to zero. This will yield a sequence of EFCs. Each EFC will represent a certain number of epochs. We could, for example, elect to store the contents of the EFC every 1000 epochs.

The data in the EFC is collected independent of the knowledge of what the system might be doing. It speaks only to the execution of the code points that were captured during a particular observation interval. In a sense, there is considerable confounding in these data. An EFC that is generated by observing an arbitrary number of epochs (e.g., 1000) may contain module execution data from the execution of one functionality, it may span from one functionality to another, or it may contain data from several functionalities. Extracting a signal from this noisy data is very difficult. It would be far better to collect execution data that relates to the specific activities, functionalities or operations, of the system.

10.6.1 Module Profile

A module profile is the multinomial probability distribution for the execution of modules on our system under a given operational profile. To create a module profile, we will construct an EFC equal in length to the number of modules in a system. Each element of the EFC will represent the execution frequency of exactly one module. Each module in the system will be instrumented to increment this frequency when the module is first entered. As per our earlier discussion on software instrumentation, the definition of a module must first be nailed down. A module, for example, may be a block of code that is the destination of a CALL machine instruction. It may also be a block of code that is entered through a conditional branch instruction.

A module profile can be derived from an EFC by dividing each entry in the observation profile by the total number of epochs represented in the EFC. From the Dirichlet distribution we observe that our estimate for the probability of execution p_i of the i^th program module is the ratio of the number of times this module has executed to the total epochs of observation to wit:

where α_i is the current module frequency count and α₀ is the total epoch count.

10.6.2 Functional Profile

It is relatively easy to collect data at the module level. We have either source code or machine code that we can easily modify to insert the necessary software instrumentation. We cannot instrument for functionality if this has not first been specified. This will always be a liability. The number of software systems currently in service or in development throughout the world for which complete and current functional specifications are maintained could be easily tallied on the fingers and toes of a single person. This is a particularly astonishing fact when the amount of safety-critical or mission-critical software currently in service is considered. It is possible, however, to retrofit systems with sufficient instrumentation to recapture their underlying functionality (Hall, ^[6] Elbaum ^[7]).

To collect data to construct a functional profile, the software system must be instrumented to do this. Unfortunately, functionalities do not really physically exist in code. Therefore, they cannot be directly instrumented. Let us observe that each functionality, by definition, must contain a nonempty set of program modules M_u that are uniquely associated with that functionality. Further, there must be a nonempty set of program modules that are indispensably associated with the functionality. This means that there must be at least one element in the intersection set M_d = M_u ∩ M_i. Any one of the modules in the intersection set M_d may be instrumented and will represent the expression of the functionality.

In this instrumentation strategy, there will by an EFC equal in length to the number of system functionalities, one entry for each functionality. Exactly one of the modules in M_d will be instrumented. Whenever a new functionality is expressed, the instrumented module associated with this functionality will cause the EFC element associated with this functionality to be incremented. The basic problem with this instrumentation strategy is that there may be a significant latency, in epochs, between the initiation of the functionality and the registration of this event in the EFC. It might well be that the instrumented module falls at the end of a long sequence of module executions that constitute the set of modules M^(f) that might execute when this functionality is invoked.

Perhaps the best way to instrument for the functional monitoring is to introduce this measurement task into the design stage of software development. In this case we will insist that a module m_i ∈ M_d be the first in the sequence of a modules that will express the functionality. In essence, each functionality will excise a subgraph from the execution call graph for the program. There will always be a root module for this subgraph. In this instrumentation strategy we will insist that this module be an element of M_d.

As was the case for the module profile, the functional profile can be derived from the EFC for the functionality. In this case, the elements of the EFC represent the frequency of execution of each functionality.

10.6.3 Operational Profiles

If the number of programs worldwide that have viable functional specifications is very small, then the subset of these that have usable operational specifications as well is even smaller. It is, however, relatively easy to retrofit a viable set of operational specifications from a set of functional specifications. Each operation is a legitimate operation if and only if it has at least one distinct functionality. That is, each operation must have some unique functionality. There must be some services performed by the system that are distinctly associated with a particular user operation.

From Chapter 9 we identified the set of functionalities that were distinctly associated with each operation o_i. Further, if we insist that each operation has at least one distinct functionality, then there exists a set of modules within this set of functionalities that is unique to that operation. As we observed before, when a program is executing a module m where , it is quite clear that the user is expressing operation o_i. Thus, if we instrument any one of the elements of there will by an EFC equal in length to the number of user operations, one entry for each operation. Whenever a new operation is invoked by the user, the instrumented module associated with this operation will cause the EFC element associated with this operation to be incremented. Again, the basic problem with this instrumentation strategy is that there may be an even more dramatic latency, in epochs, between the initiation of the operation and the EFC update to reflect that activity.

A more direct instrumentation strategy could be derived in the same manner that we developed for instrumenting the functionalities. Let us observe that the set of distinct functionalities associated with each operation are mutually exclusive functionalities. In this set of functionalities, there is one functionality that will always be executed first. Just as we did with the instrumentation of the functionalities, we can design the system so that one particular module is invoked whenever this initial functionality is expressed. By instrumenting just this one module at the head of a call graph structure for the operation, we can then collect the operation EFC data in a timely manner.

As was the case for the functional profile, the actual operational profile can be derived from the EFC for the system operation. In this case, the elements of the EFC represent the frequency of execution of each operation. Remember that our design quality standards dictated that we have some estimate for the prior distribution of the operation profile before we designed the system in the first place. Our subsequent observations on new operational profiles as the system is placed into service will serve either to validate our initial prior distribution of the operational profile or to indicate a need to revisit the design and testing process should a new operational profile emerge.

^[6]Hall, G.A., Usage Patterns: Extracting System Functionality from Execution Profiles, Ph.D. thesis, University of Idaho, Moscow, ID, 1997.

^[7]Elbaum, S.G. and Munson, J.C., Software Black Box: An Alternative Mechanism for Failure Analysis, Proceedings of the 2000 IEEE International Symposium of Software Reliability Engineering, San Jose, CA, November 2000.