Chapter 12. Benchmarking Proxy Caches
| only for RuBoard - do not distribute or recompile |
Chapter 12. Benchmarking Proxy Caches
Benchmarking is the process of measuring the performance of a product or service. Benchmarks are used for many different computer systems and components , such as CPUs, disk drives , Ethernet switches, and databases. A good benchmark must be stressful, reproducible, and meaningful. A test or workload that doesn't stress the device under test is not very interesting. Reproducibility is important for tests that compare different products or slightly different configurations under the same conditions. A meaningful benchmark is one that accurately predicts the performance of the device or system under real-world conditions.
Benchmarks are used by engineers, marketers, and customers alike. Product engineers use benchmarks to evaluate design choices and overall system performance. A good benchmark identifies bottlenecks and tells them if changing a certain component improves or worsens the product's performance. Benchmarking results are often used in marketing literature, especially when one company claims to have a superior product. Even when a particular product does poorly, marketing departments find some way to put a positive spin on the results.
Customers use benchmarking results to make buying decisions. Good benchmarks enable customers to compare different products with each other, find the one that best suits their needs, and understand the price versus performance tradeoffs. Without published, audited benchmark results, customers have no way to verify performance claims made by individual vendors . If you are considering the purchase of a particular product, I highly recommend asking the vendor for recent published and audited benchmark results first.
In this chapter, I'll talk about benchmarking for caching products. First, I'll introduce you to the metrics that are important for web caches. Understanding the metrics and how they are related enables you to interpret published results quickly and correctly. I'll also tell you about some of the tools you can use to run your own benchmarks. Since executing a benchmark on a web cache is somewhat complicated, we'll also talk about the common pitfalls and difficulties others have encountered . The final section provides a basic recipe for how to benchmark a web cache.
Note that I am heavily involved in benchmarking caching products through my company, The Measurement Factory (http://www.measurement-factory.com). We organize the industry "Cache-Offs" and perform private tests for both vendors and customers. We have been fortunate enough to work with most, but not all, of the companies that sell caching products.
| only for RuBoard - do not distribute or recompile |
