10.4 Testbed and model workloads

 < Free Open Study > 



10.4 Testbed and model workloads

The term workload defines the load placed on a real system (typically measured or observed on a computer system while it runs normal operations), while the term test or model workload denotes a computer system's load constructed and applied to a system for performance studies (typically synthesized using characteristics from a real workload). For most modeling projects the use of a synthetic workload makes more sense, since we can control the load applied to the experiments. By controlling the load applied to a computer system under analysis, we can possibly predict the outcome of the experiment or force the experiment to test specific components of the system. In addition to this reason, synthetic workloads do not possibly contain real information, which may be sensitive or valuable to the system under study, and its compromise or loss would be significant. Once a valid synthetic workload has been developed, it can be reused to study additional systems. An example is the Transaction Processing Consortium (TPC) workloads developed to study database systems. These TPC workloads have been used by vendors and customers to study various database systems and to determine which is better for different applications. Some of these workloads have been specialized for data mining or for distributed databases and other specialized applications.

To study computer architectures, a variety of instruction workloads have been developed. These are focused on low-level operations and consist of mixes of loads, stores, comparisons, branches, additions, subtractions, floating-point operations, multiplications, divisions, shift operations, logical operations, and register operations. These instruction mix workloads have become standardized for specific architectures such as PCs.

Other workloads do not focus on low-level operations but wish to examine more coarse-grained architectural concepts. These would be developed using high-order languages and would be designed to test things such as file transfer, task switching, memory management policies, and other operating systems components.

Some popular benchmarks include the TPC benchmarks described previously for examining database systems, the Sieve benchmark used to examine PCs and microprocessors, Ackerman's function for testing procedure call mechanisms in computer systems, Whetstone kernel developed to test low-level assembly-level operations, the Linpack package to test floatingpoint operations, the Drystone benchmark for testing low-level integer operations, and the Spec benchmark suite for measuring engineering-type applications (e.g., compilation, electronic design, VLSI circuit simulation, and complex mathematics manipulations such as matrix multiplications) on a computer system.

Given that all of these and other workloads exist, modelers must still determine which to use or which method to use in constructing their own workload for a given modeling project. There are four main considerations applicable when selecting a workload for a project. They are the computer systems services exercised by the workload, the level of detail to be applied, closeness to realistic load, and timeliness.

The most important component of the workload selection is to determine the services one wishes to examine. Making this list of services can be very daunting and time consuming but is time well spent. First, one must determine the system under test. This represents the complete set of components making up a system being studied. Often we may be focusing on some single component or some small set of components for comparison, called the components under study. For example, an operating system design team may be interested in different process scheduling algorithms on the total operating systems performance. The determination of the system and its components is a very important step in workload development and should not be trivialized.

An example will illustrate the service's concept. We are interested in this example: comparing an off-line backup paging storage system using disk drive arrays (e.g., such as one would find in a large database log subsystem). The system consists of several disk data systems, each containing multiple disk drives. The disk drives have separate read and write subsystems. Each subsystem uses fixed magnetic heads for these operations. If we specify the architecture from the highest level and work down to lower levels, the services, factors, metrics, and workloads are defined as follows:

  1. Backup system

    • Services: backup pages, backup changed pages, restore pages, list backed-up pages

    • Factors: page size, batch or background process, incremental or full backup

    • Metrics: backup time, restoration time

    • Workload: a database system with log pages to be backed up- vary frequency of logging

  2. Disk data system

    • Services: read/write to a disk

    • Factors: type of disk drive

    • Metrics: speed, reliability, time between failures

    • Workload: synthetic program generating transaction-like disk I/O requests

  3. Disk drives

    • Services: read record, write record, find record

    • Factors: disk drive capacity, number of tracks, number of cylinders, number of read/write heads

    • Metrics: time to find record, time to read record, time to write record, data lost rate, requests per unit time

    • Workload: synthetic program generating realistic operations requests to the disk drives

  4. Read/write subsystem

    • Services: read data, write data

    • Factors: data encoding technique, implementation technology

    • Metrics: I/O bandwidth, density of media

    • Workload: read/write data streams with varying patterns

  5. Read/write heads

    • Services: read signal and write signal

    • Factors: composition, head spacing, record gap size

    • Metrics: magnetic field strength, hysteresis

    • Workload: reads and writes of varying power strengths, disks moving at various rotational speeds

After we have completed the specification of the system and the components of interest, we need to determine the level of detail required in producing and recording requests for the defined services. A workload description can be as detailed as providing definitions for all events in the system or can simply be an aggregate or generalization of this load. Some possibilities for the detail may be average resource demand, most frequent request, frequency of request types (e.g., 25 percent reads and 75 percent writes), a timestamped sequence of specific requests, or some distribution of resource demands.

Typical modeling projects begin by using a variant of the concept of most frequently requested service. For example, in a transaction processing system we may use a simple debit-credit benchmark from the TPC benchmarks. Such a selection would be valid if a particular service is requested much more than others. A second alternative is to be more specific and construct a workload by selecting specific services, their characteristics, and frequency. The Linpack package is such a workload. It selects very specific computer operations in very prescribed patterns to test specific components of the system. The next alternative is to construct a time stamped record, where each record represents a specific request for a specific service along with details of the actual access (such a description could be constructed by taking a trace of all activities of an existing system). In most cases this type of workload may be too difficult to construct and to validate for use in all but the most complex modeling projects. The aggregate resource demand approach is similar to what we would expect to see in an analytical model. We look to characterize each request for services as averages or distributions. For example, each request may be characterized as requiring 50 units of one particular resource and 25 units of some other and making these requests every 1,000 units of time.

No matter which of these approaches we use, the modeler must determine if the selected load is representative of the real system load. Typically we will be interested in determining if the service request's load has similar arrival characteristics, resource demands, and resource utilization demands as the real load.

Finally, a developed workload should faithfully model the changes in use patterns in a timely manner. For example, the TPC benchmarks have continued to evolve to meet the needs of changing database systems design and use. The original TPC workloads were designed for the "bankers" database problem. That is, they simply were looking to provide transaction loads to well-structured, simple, flat relational database specifications. They were record oriented and had no dimensions beyond the simple relational model of the day. These have evolved now to include benchmarks for the new object relational databases and for data warehouses and data mining systems. Other important considerations in developing a workload include repeatability, external components impact, and load leveling. Repeatability looks at a workload's ability to be reproduced faithfully with little added overhead. External components impact looks to capture and characterize impacts on the system under study by nonessential components. Finally, load leveling may be of interest if our study wishes to examine a system under best-case or worst-case scenarios.



 < Free Open Study > 



Computer Systems Performance Evaluation and Prediction
Computer Systems Performance Evaluation and Prediction
ISBN: 1555582605
EAN: 2147483647
Year: 2002
Pages: 136

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net