Performance Modeling Technology

Performance Modeling Technology

One goal of performance modeling is to be truly proactive in performance engineering to examine a proposed system in its entirety, from hardware and network resources to code optimization, before completely building any one component. In this section, we ll discuss the following:

• Scenarios in which performance modeling can replace other methods of performance assessment and engineering

• Different methods of modeling and when they are appropriate

• A brief look at currently available performance modeling tools

• A detailed look at the toolkit approach, represented by Microsoft s Indy project

Modeling Scenarios

How can performance modeling improve business practices and efficiency, as well as the overall performance of systems? Let s look at some of the common issues facing client-server systems.

Capacity Planning

Anticipating the need for more resources involves both business and engineering decisions. Once a company has determined an acceptable estimate for future growth, the process of improving available resources follows close behind. Simply adding more hardware, as discussed in our introductory example, can mean that the additional resources do not equal more capacity, and performance suffers, taking company profits along with it. Using TCA for capacity planning can be highly accurate, if the data in the TCA structure is accurate to begin with. However, it still requires verification and, possibly, unnecessary expenditures to verify the projected results. Through more detailed definition of the user load, hardware, and software components of a system, performance modeling can eliminate the verification step and suggest alternative ways to improve capacity before the need arises.

Bottleneck Analysis

Evaluating a system that suddenly reaches a plateau and does not respond to hardware upgrades can be a frustrating process. Using performance modeling to detail each transaction and the associated hardware requirements for those transactions can expose bottlenecks. In advance of an actual performance failure, it can also be used to test with higher potential system load and predict the occurrence of a bottleneck, simultaneously showing ways to alleviate the problem before it occurs.

Hardware Configuration

Detailed performance models can very accurately predict the behavior of proposed hardware upgrades, using performance counters and simulating transactions against the virtual hardware. The question of whether to spend money to upgrade from 2-CPU to 4-CPU servers can be answered in a performance model with a few clicks of a mouse, rather than through trial and error.

Architectural Assessment

One way to assess the performance quality of two radically different architectural structures is to build both and examine the actual results. This is hardly feasible in the real world of software engineering or multi-client services, however. Both cost and time would be prohibitive. TCA can be used to produce estimates regarding overall system performance based on hardware and network architecture, but it still requires verification. In this case, performance modeling can decrease both the time and money spent on architectural engineering by providing varying levels of detail, depending on the architecture questions to be answered.

User Scenarios

TCA is a simple and effective tool for examining what happens when the user load on a given system increases. It is more difficult, however, to use TCA to track the impact of changes in typical user behavior. Performance modeling adds the ability to test different user habits by relying on models of user actions instead of real or borrowed user data patterns.

Performance Modeling Methods

There are three main methods of modeling that can be used to predict, assess, and interpret performance during system construction. These methods are analytical, statistical, and simulation. Understanding the requirements and results of each method is critical for making decisions about which is best used for a given task.

Analytical Modeling

The process of analytical modeling involves the use of mathematical expressions to represent the interactions that take place on a system. In some cases mathematical notations such as queuing networks and Petri networks are employed to represent the architecture of a system.

Typically, each component in the model, whether it represents a hardware device, network transaction, section of software code, or user activity, is represented in the underlying system in mathematical terms. Complex equations, representing the relationships between these components, are developed and then solved to determine the projected value of a given variable in the model.

For example, if you have a Web server that only serves simple HTML pages to clients connecting over the Internet, you would need to gather information about the server s hardware configuration (CPU, disk speed, and memory usage), the content being served (including file sizes for HTML, image, and other elements of the Web pages), the network capacity at the server (the real bandwidth available to the server), and various scenarios for client activity (including low and high load situations, average number of page views, average amount of time between page requests, and various network connection speeds at the client level). Given this data, you can extrapolate a control equation that accurately relates all of the elements in terms of the time and resource utilization.

Once the control equation (which might be represented in a spreadsheet with embedded calculations) has been established, you can change the values of different elements in order to see how those changes impact other elements within the boundaries of the equation.

You will recognize this as the basic methodology behind TCA. Knowing the system s typical behavior, TCA can predict system performance based on possible changes in user behavior, or changes in hardware configuration (provided the data values are accurately modified to reflect the new hardware s capacity and response times).

Analytical approaches can provide an initial estimate of the performance of new systems, but typically can produce results for only a limited set of what-if scenarios. The use of a mathematical notation to describe the most challenging performance effects of a system, such as queuing delays and resource contention, requires substantial expertise on the part of the performance engineer.

Statistical Modeling

Statistical modeling in performance engineering relies on known performance metrics for existing systems. You can use this data as a basis to predict the behavior of a new, not yet built system. This is achieved by analyzing the measured data with techniques such as regression analysis. Then the resulting statistical models can be used to extrapolate the performance of the system in new configurations.

For example, a hosting company may have a number of existing e-commerce servers for its clients already in production. To recommend hardware and network capacity for a new client, aggregate statistical data about all of the production servers could be analyzed and a recommendation made, either to use similar hardware and bandwidth, or to use different elements in the new system in hopes of improving the current systems performance.

As with any statistical analysis, the accuracy of the resulting projections increases with the sample size of existing data. For this reason, the hosting company in the example would want to measure performance on several systems before arriving at an average model. One barrier to using statistical analysis in performance engineering is the lack of either quantity or complexity of existing data. Reviewing the published site statistics for a comparable, competing e-commerce vendor might yield valuable statistical data about number of visitors and those visitors traffic patterns, but probably would not include information about the site s hardware and network connection capacity, or the software used to provide the site s services. In this case, a combination of statistical and analytical models (again, similar to TCA when used to project behavior of a new service) would be required to accurately assess performance prior to launch.

Statistical models include assumptions about the way the performance extra polates. For example, the performance engineer might assume that the CPU utilization is linearly increasing with load. Although this might be true for the observed performance measurements, the CPU utilization behavior changes when it approaches saturation. Experience is the only way to avoid these pitfalls.

Simulation

Simulation may be done in two ways. The first involves building a proposed system and subjecting it to simulated load patterns to assess its performance. The accuracy of simulation modeling in this manner is rivaled only by real-world experience, because the hardware and software in place are the real tools used to provide the service being tested. Load generation can be done through the creation and automation of use cases, or by using commercially available traffic generation tools such as ACT. Stress-testing a real system not only exposes the performance problems, but demonstrates the aftermath of a performance failure as well.

For example, consider a server application that processes telephone calls for an IP telephony system. Its performance requirements run the gamut from lightning-fast CPU response time to 100-percent bandwidth availability on the network connection, because it must complete all of its requests to the telephone network within a 200-millisecond time window. Such a system might perform flawlessly under minimal user load, but an increase in the load that results in either queuing on the CPU or congestion on the network would result in telephone service outages for its users. By simulating call-processing load on the actual system before deployment, you could determine the real limits of the system and avoid overuse by setting strict limits on the number of clients who use that server for the handling of their calls.

The trade-off for accurate results in this type of simulation modeling is the fact that improvements must be made to a complete, or nearly complete, system. In the preceding example, the only way to avoid service failures with the existing call processing server is to limit the potential number of simultaneous users. Alternately, the system engineers can test alternate hardware or software configurations, but must purchase the resources or rewrite the code to do so.

The second method of simulation modeling involves writing software that accurately represents the behavior of hardware devices, and using another software tool to process existing code or simulated software behavior (such as UML) through the hardware device simulations. This eliminates the hardware costs associated with simulating only the load on a system, but can be extremely time-consuming, if a performance engineer has to develop the constructs and applications for processing the simulation from scratch.

Advanced Performance Modeling

Each method of modeling has advantages and disadvantages. In the real world, it is necessary to balance the potential quality of test results with the time and money required to run those tests. Advanced performance modeling tools, such as the Indy example later in this chapter, combine these methods to provide flexibility and scalability with lower cost and a broader learning curve.

Performance Modeling Tools

A few tools are available today that use models to evaluate and predict performance. Most of them are customized solutions, which can be purchased either as an entire system, or built to specification by the vendor. They typically feature very large libraries of existing hardware models, graphical drag-and-drop interfaces, and the services of consultants to assist engineering staff in applying the techniques provided by the software.

The power and features of these tools come at a high price often hundreds of thousands of dollars in licensing, and more in training for the engineer who will ultimately be responsible for the tool s use. More emphasis is typically placed on hardware than software.

The direction of most performance modeling tools on the market today is toward solving one or two specific problems, or addressing a single application s performance. For performance modeling to be more widely used, and accessible to a larger audience, it must be more modular and support a broader range of user skill levels.