2.3 Benchmarking

2.3 Benchmarking

A benchmark measures the performance of a standard workload. By holding the workload fixed, you can vary the underlying system parameters to generate relative performance numbers .

In general, there are three types of benchmarks: component-specific benchmarks, like SPECint and SPECfp for microprocessors; whole-system benchmarks designed to emulate commercial environments, like the TPC series and SPECweb99, and user -developed benchmarks. In this section, I touch on these different approaches to benchmarking, providing examples of real-world benchmarks throughout.

This book is not about tuning systems to produce world-record benchmark results. In general, producing a world record requires an immense amount of work, strong support from the engineers responsible for the underlying subsystems, and an understanding that the end result is a world record (which may not necessarily be directly applicable to a customer environment).

2.3.1 MIPS and Megaflops

Before we get started in a discussion of benchmarks, we need to discuss a little bit of terminology and history: the MIPS and megaflops metrics.

2.3.1.1 MIPS

One of the first ways of estimating the performance of an application on a computer system was to look at how fast the system could execute instructions. However, this is a dismal metric. The execution rate of a processor (typically expressed in millions of instructions per second, or MIPS) is strongly correlated to the clock rate of the processor. It's probably safe to assume that a 5-MIPS CPU will outperform a 100-MIPS CPU, but a 50-MIPS processor might well actually outperform a 100-MIPS CPU. Using MIPS to compare the performance of different computers is flawed in a basic way: a million instructions on one microprocessor might not accomplish the same work as the same number of instructions on another. For example, let's say you have a processor that does all floating-point work in software, and takes 50 integer instructions to perform 1 floating-point operation; your 50-MIPS CPU now performs 1 million useful instructions per second. Coupled with this is the fact that different instructions can take different lengths of time to complete, especially on CISC processors. Were those million instructions "do-nothing" operations, or floating-point multiplies?

To make things worse , some vendors have redefined the meaning of "MIP" as their own internal metric. DEC was particularly famous for this; for a long time, the VAX 11/780 was considered as the standard "1 MIP" machine. DEC publicly stated that, for a reasonable job mix, the 11/780 was capable of executing about 470,000 instructions per second -- less than half a MIP. For this reason, you sometimes hear people refer to "VAX MIPS" or "VUPs" (VAX units of processing) to make the comparison clear.

The key point to take away from this is that MIPS is only a useful metric to compare processors from the same vendor that support the exact same instruction set with the same compilers. The fundamental weaknesses in MIPS as a performance metric have often led people to joke that it's really short for "Meaningless Indicator of Processor Speed."

2.3.1.2 Megaflops

Floating-point performance is often measured in terms of millions of floating-point instructions per second, often called megaflops or MFLOPS . A floating-point instruction is commonly regarded as one that executes on a microprocessor's specialized floating-point hardware; this includes floating-point additions, multiplications, comparisons, and format conversions. Less commonly included are square roots and divisions. Typically, though, megaflop rates are computed from some mixture of additions and multiplications. While it is considered misleading to present MIPS values, vendors regularly quote peak MFLOPS values. The peak MFLOPS for a processor is usually on the close order of an integer multiple (1, 2, or 4) of the clock rate. This is because optimizations in microprocessor hardware perform fast multiply-add operations (such as for accumulating a running sum of vector elements, like in a dot product of two matrices). As microprocessors become faster and faster, the peak MFLOPS value becomes less useful as a reasonable upper bound on application floating-point performance: memory bandwidth (how fast data can be moved) in and out of the processor becomes limiting.

2.3.2 Component-Specific Benchmarks

The first set of benchmarks I discuss are designed to stress specific areas of the system; typically, they are memory performance and/or microprocessor performance. I discuss two of the most common: the Linpack benchmark and the SPECcpu suite.

2.3.2.1 Linpack

One of the first commonly accepted benchmarks is the Linpack benchmark , written by Jack Dongarra, then of Argonne National Laboratory. Interestingly, the goal of Linpack was not to develop a benchmark; rather, it was intended to provide a high-performance library of routines for linear algebra problems. The Linpack benchmark derives megaflop rates by measuring the time and number of operations required by a few of these routines to solve a dense system of linear equations using Gaussian elimination . There are several versions of Linpack, differentiated by the size of the system of linear equations they worked with, the numerical precision required, and the ground rules for the benchmark. The most typically-quoted was given by the 100 x 100 matrix with double-precision results and strictly compiled FORTRAN (no hand optimizations were permitted).

In addition to the 100 x 100 benchmark, there is also a 1000 x 1000 benchmark, in which the system of equations can be solved via any method of the vendor's choice. The larger Linpack is an important benchmark because it tends to give a reasonable upper bound for the floating-point performance of very large, highly optimized scientific problems on a given system.

The heart of the 100 x 100 Linpack benchmark is a routine called daxpy , which scales a vector by a constant and adds it to another vector: ^[7]

^[7] Daxpy is actually written in FORTRAN; it is represented here in C.

for (i = 0; i <= N, i++) {

     dy[i] = dy[i] + da * dx[i];

}

Each iteration of this loop performs two floating-point operations (a multiplication and an addition) and three memory operations (two loads and a store). It turns out that the Linpack benchmark stressed both the floating-point and memory performance of computer systems up through the early 1990s. When microprocessor cache sizes increased to a size where they could hold the entire benchmark, the Linpack 100 x 100 ceased to be a useful benchmark -- the data structures for the entire benchmark are only 320 KB. The first system that was widely credited with "breaking" the Linpack 100 x 100 was IBM's RS/6000 series. Unfortunately, optimization techniques prevent the Linpack 100 x 100 benchmark from being scaled up to the point where they stress the memory subsystem again.

So, the Linpack 100 x 100 went from being the most prevalent single-number benchmark in history to being relegated to a historical footnote in about two years , as vendors scaled their caches to fit the entire benchmark.

2.3.2.2 SPECint and SPECfp

In the late 1980s, a group then called the Systems Performance Evaluation Cooperative, or SPEC, formed to establish, maintain, and endorse a standardized set of benchmarks. ^[8] The first SPEC microprocessor benchmark was called SPEC89, and was revised in 1992, 1995, and 2000. By releasing new versions, the SPEC organization keeps current with the latest trends in hardware and compiler design, preventing vendors from becoming too good at tuning their systems to the benchmarks. While the most commonly quoted SPEC benchmarks regard CPU performance, SPEC publishes many standard benchmarks. For more information on SPEC, consult http://www.spec.org.

^[8] They are now the Standard Performance Evaluation Corporation.

The first SPEC benchmark was SPEC89, which consisted of the geometric mean of the runtimes for 10 programs. It did not differentiate between floating-point and integer performance, which had the side effect of cloaking performance subtleties. For example, a system that did very poorly on integer performance but very well in floating-point would have the same SPECmark score as a system that performed in exactly the opposite way. Interestingly, when SPEC89 was released, it was not a particularly important benchmark; the different competing architectures of the time had very different performance characteristics, and no benchmark could effectively relate all of their performances . In the field of high-performance computing, the Linpack 100 x 100 code was completely dominant.

To improve the quality of the benchmark, SPEC revised it and released the successor as SPEC92. SPEC92 consisted of 20 test programs, broken up as 6 integer and 14 floating-point codes. The benchmark also reports two summary figures, SPECint92 and SPECfp92, which measure integer and floating-point performance, respectively. As the Linpack 100 x 100 benchmark started to fall, SPECfp92 became the battleground metric for floating-point performance. Around 1994, large caches and smart compilers broke the SPECfp92 benchmark. Thankfully, in 1995, SPEC again released a revision to the benchmark suite. It increased the size of the benchmarks so that they wouldn't fit into caches, greatly lengthened runtime, and became more like large user applications. A similar process occured in 2000, when SPEC again rescaled the benchmark.

The applications under test in the SPECcpu2000 benchmark are quite representative of typical user codes, and are summarized in Table 2-3.

Table 2-3. Codes under test in SPECcpu2000

Similar products