3.4 Measurements (sampling)

< Free Open Study >

How does one measure a system or component performance? This is the main problem facing the computer systems performance analyst. To determine how to measure, when to measure, or what to measure, the analyst must first know all of the events of interest in the system and the relationship these events have with each other. The events, as we saw earlier, represent all of the real actions that occur in the computer system under study. These events form a hierarchy of relationships, where the finer, granular events are used to construct the coarse-grained events in the system. Even with these definitions, however, we do not know enough to begin measurements that will have meanings. We must know all the possible conditions that hold for events in our system and when they can be valid. Given a set of possible events and their values, we can describe a valid "state" for the computer system under study.

State is an important concept when considering any computer system. The state, S, is defined as the set of all events in our system along with valid values for their condition within the defined state. This can be described as follows:

(3.1)

where each of the events must have all component events valid, and their own values must define valid states. For example, a state for a central processing unit may be defined as being composed of the following events and values:

The program counter address held in the program counter
The instruction held in the instruction register
The status and value of the index register
The status and values in the condition control register
The value in the arithmetic logic unit temporary registers
The value on the data bus
The value on the address bus
The value in the memory data register

Once we have definitions for the events and the state of the system, we can then begin to discuss the concept of measuring quantities within the system. There are three primary types of measures: A, B, and C. They can be described as follows:

Type A looks to count a number of items over a given time period. For example, we may be interested in how often the CPU receives a new instruction during each second. This would represent the instruction speed of the processor, given the mix of instructions presented to the CPU.

Type B measures all state variables (valid events and their values). A representation of this type of measurement may be to extract all of the values for all internal registers and devices at the beginning of an instruction execution cycle.

Type C measures the fraction of time the system is within a state. An example of this measure may be to see what the fraction of time is that the system is executing load instructions versus all other kinds of instructions during the measured period of time.

It is not sufficient to simply determine the kind of event one wishes to measure and the values representing this event. One must also be able to recognize that a specified state has been reached and that all events and status variables for the state are valid. In addition to recognizing that a valid state has been reached, one must also be able to determine if we are at an end or transitional point within a state. These are not easy to know when one is attempting to measure a system.

In order to find out where we are within a state, we must have means to measure the systems events we are interested in. There are a number of ways to measure these events, each with its own issues. We can use hardware monitoring, software monitoring, or hybrid monitoring. The decision about which of these techniques to use is dependent on many factors, such as accessibility, event frequency, monitor artifact, overhead of monitoring, and the flexibility of the technique used.

Hardware monitoring requires that the system analyst have the ability to add instrumentation to the measured system. For example, we may attach a logic analyzer to measure the signals within the system or insert a specially designed hardware card to extract some signals from a system. This mode of measurement will allow us to measure some subset of the total system's hardware elements. We can only measure what is exposed and available to be attached to for monitoring. If the item or action we wish to monitor is not easily accessible, we may not be able to get to it using a hardware monitor. We may need to use some other means to extract the information from the system.

Another form of hardware monitoring uses integral test hardware, which is designed into the system being monitored during systems design. A common form of this monitoring scheme is found in very large scale integration (VLSI) devices. Many VLSI devices are designed so that all data items of interest can be tested in the device itself, or, at a minimum, the test data points are brought outside of the chip so additional devices can be used to gather this information and compute the health of the device.

In all of these cases it is imperative that the hardware monitoring be designed as an integral component of the system, so that it will not interfere with the operational system. It is not desirable for the monitoring equipment to interfere with the system being monitored. If this is the case, the results from the monitoring are suspect and may lead to erroneous conclusions. The monitoring hardware must be selected and designed with the device being measured in mind. The determination of sampling sites and the frequency of measurements must be designed ahead of time, not after the monitor has been put in place. The monitoring method has to be set up ahead of time also. That is, we must determine if the monitor is to act synchronously or asynchronously with the measured system. We must determine and define all aspects of the monitor's existence in the measured system. Nothing can be left out if we are to get trustworthy data.

Software monitoring requires support from the system under study if it is to be successful. Software monitoring requires that there be a means for the monitor designers to get at systems hardware elements as well as low-level software elements-for example, systems clocks, programmable timers, interrupt registers, and systems status registers. The typical software monitor is designed for trace monitoring and sampling, not for synchronous monitoring. In trace monitoring, the analyst adds additional code to a code sequence so that the code's run time can be monitored. Typically we would be interested in how often a code segment is entered, how long the code segment runs, or how much of the total systems time the code segment utilizes.

Software monitoring, as with hardware monitoring, still requires that we know ahead of time where the sampling measures are to occur within the system and the frequency of this sampling if our measurements are to have meaning.

In software monitoring, where we are using sampled monitoring techniques, we need to have access to low-level operating systems calls. This type of access is required so that we may cause a system's interrupt and take control of the system. The interrupt control would allow for entrance into the system and collection of systems state information such as the contents of registers and status flags. One positive aspect of this form of software monitoring is that it may not lead to the alteration of any code, given that all required information can be collected using available information.

A more common form of measurement uses hybrid monitoring. This form of systems monitoring uses concepts and mechanisms from both hardware and software monitoring. To utilize hardware monitoring we must go through the same set of issues as was the case for hardware monitoring as well as for software monitoring. The setup may require the synchronization of multiple hardware and software setups. We must set up the control programs to determine when and how to monitor the system under test. We must determine what to capture with hardware devices and what to capture with software means. Upon execution of the monitoring subsystem, we must determine how and how often to retrieve collected information. In addition we must also determine how and when to synchronize the measuring and measured systems.

Hybrid monitoring comes with its own set of problems. As in hardware monitoring, we must have a means to extract signals of interest from our system. We must determine which elements we wish to test are best tested with hardware and which with software. We must understand and bound the impact the monitoring software has on the monitored systems, so that correct measurements are extracted. Finally, we must always keep in mind that the measurements are only as good as the available measurement points.

< Free Open Study >