10.5 Experimental design

 < Free Open Study > 



10.5 Experimental design

Once we have a testbed to study a computer system and a workload to load the testbed with, we need to design experiments that will help in discovering the performance limitations we as modelers are focused on. A correct experimental design will provide the maximum analysis information with the minimal number of experimental runs required. Some terminology must be introduced to make this discussion meaningful. A performance variable is a measured outcome for an experiment for a single component, process, or possibly an entire system. A factor is a variable that may have an impact on the performance variables and typically represents items that can be varied during an experiment. The steps are values a factor can take on during an experimental sequence of runs. For example, a CPU's memory may be adjusted from a minimum value to a maximum value in some distinct number of discrete steps. Each of these steps represents a value for the factor under study. Factors need not all be important. Typically, experiments on computer systems will have multiple factors, some very important (such as CPU speed) and others only peripherally important (such as terminal speed). Experiments may be repeated and are then referred to as replicants. An entire performance study for a particular system consists of a number of discrete experiments-when taken together this set of experiments constitutes the experimental design. Factors may have a correspondence to each other and must be defined as having a dependency.

Experimental design comes in a variety of ways. Three typical designs are the simple, fractional factorial, and full factorial designs. In a simple design, we start with a fixed configuration and vary one factor at a time to determine how this factor impacts performance. For example, when measuring the performance of a virtual memory management component, we may wish to study systems design by varying the size of the available primary memory. By running separate experiments, each with all conditions held stable except the memory size, we may be able to determine some useful information concerning the virtual memory management systems operations. The total number of experiments required for a simple design is simply the sum of the number of experiments for each factor. For example, if we wish to study the memory management system with three different memory sizes, using three separate CPUs and three different disk drives using three workloads, we would need:

(10.15) 

total experiments to be run. This form of experiment would give us some information but may not indicate to us how the various elements interact with each other. To determine how the factors interact we would need to run either fractional factorial experiments or full factorial experiments. In the fractional factorial experiments we may wish to examine only a few of our factor terms actively against each other. In this case we would require additional numbers of experiments. In our example, if we are interested in how the CPUs interact with the memory, we would be required to test all combinations of these against each other. This would require additional experiments to be run as multiples of each other. For our simple example we would now require:

(10.16) click to expand

experiments to be run versus the original 12 for the simple design method. For a full factorial experimental design we simple vary all of the factors against each other. In this case we would now require that we perform:

(10.17) 

specific experiments to look at how all of these factors affect each other over their entire range of values.

When doing experiments using factorial designs it is also important to determine how necessary the various factors are in relation to each other. This is typically determined using allocation of variation. In this method the importance of a factor is measured by the portion of the total variation in the performance variable explained by this factor. For example, if two factors explain 90 percent and 5 percent of the performance variation, respectively, then the second term can be considered to have little effect on the performance variable. The sample variance for a measure is found as:

(10.18) 

where f is the mean response time for all of the experiments combined for our measured performance variable. Many more such correlations between information must be examined and understood if we are to make sense of the performance information being returned by our models. More details on how to interpret such information can be found in the references.



 < Free Open Study > 



Computer Systems Performance Evaluation and Prediction
Computer Systems Performance Evaluation and Prediction
ISBN: 1555582605
EAN: 2147483647
Year: 2002
Pages: 136

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net