11.3 Conducting experiments

---

11.3 Conducting experiments

Given that we have selected a tool, constructed our model, and validated the model, we must next develop our experiments to perform the initially intended function for the performance study. To develop our experiments we must have an idea as to what the performance metrics will be (performance variables) for the study. We saw previously, in Chapter 8, that to develop a set of performance variables we must begin by developing a list of services to be offered by the system under study. Given that we have done this selection and definition of services, we next must determine all possible outcomes for the service. For example, each service can have a request for service pending, be in service, be completing service, or rejecting a service request. The results of the service request are to accept the request for future service, perform the service either correctly or incorrectly, or simply reject the request as not being possible. For example, the lock manager for a database system can accept a request for a lock and either grant it, delay it, perform the request erroneously, or refuse the lock request altogether.

If the system performs the request correctly, the performance is measured as the time taken to perform the service, the rate the service is performed at, and the resources consumed while performing the requested service. These three metrics relate to the measures of responsiveness, productivity, and utilization-all important components of any computer system's performance study. These measures have also been altered to show speed, reliability, and availability. For example, the responsiveness of a transaction processing system is measured by its response time. This consists of the time between a transaction's request for service and the response to the transaction from the server. The transaction processing systems productivity is measured by its throughput. The throughput consists of the number of transactions performed completely during some prescribed unit of time (e.g., per minute or second). The third measure, utilization, provides a measure of a resource's business. In the transaction processing example, we could see what percentage of time the server is busy serving transactions versus the time it is idle during the same interval of time as the throughput measure. Using such information we can begin to isolate problems within our system. For example, the service that is the most highly utilized may represent the system's bottleneck.

If a service is done erroneously, we also wish to capture this information. Error detection and measurement are important in determining a service's resiliency and tolerance to errors. For example, in the transaction processing system, we may want to know what caused the errors and the failure of transactions being processed. It may be important to know that an error occurred due to a hardware failure or a software failure, or was caused by contention or a poor transaction design. Each will tell us something about the product we are evaluating. If a resource cannot perform the service function at all, it may imply it is down, or 100 percent utilized. Once again, knowing what state the resource is in will aid in the determination of its overall performance.

---