| One way to avoid the expense and time involved in doing your own load testing is to use reports generated by independent testing organizations. Most of these tests, also known as benchmarks, usually focus on a single element, such as a server, a load balancer, a router, and so forth. These independent tests can be very helpful, but care must also be taken. Bear in mind that product suppliers are often the ones that commission such tests; no reports are published that show the test sponsor losing. There have been cases where the same testing organization has published different reports showing that the sponsor of each won hands down over the same competitors that beat them in a previous report. This happens for two reasons: The worst case is a testing facility that caters to whomever pays the bills. A more common reason is that each test uses slightly different conditions to favor the sponsor. As a case in point, many years ago, a network device vendor trumpeted its packet forwarding performance. It turned out that each interface card had storage for recently used IP addressessaving a routing database lookup and speeding up the forwarding process. Needless to say, their testing process always used a small number of IP addresses, so the local storage of recent IP addresses was leveraged to the hilt. The test generated impressive numbers, but had little relevance to a real environment. When a large set of IP addresses was used, the performance dropped significantly. This isn't surprising, and there is still a lot that can be learned, but you have to work a little harder. It's also an important lesson in attention to detail in designing internal tests; such effects can be inadvertently introduced into any test environment.
It is important to understand as much as you can about the testing process behind the data in the testing report. A good report should include the following: A description of the test bed, the elements, and their interconnection. A description of the testing profiles. What are the loading characteristics and workload properties? A full description of the Device Under Test (DUT) features used. Are they realistic in your world? Are features that lower performance used in the tests? Does the network equipment handle a mixed background workload, along with any possible access control lists and similar resource-using features, similar to your environment? A complete explanation of the results, along with graphs of performance versus applied load.
Gathering a set of load testing benchmark reports can be useful even if the results of the tests are not directly comparable to your situation. Studies sponsored by your vendor's competitors may reveal problems that your vendor will not mention to you. Benchmark reports may also point out common problem areas for which you should be looking. The value of independent element testing is that you don't pay for it, and it helps define realistic envelopes and their associated thresholds. |
