Test Goals

First, before starting performance testing, complete a rigorous function and integration test of your web applications. This ensures that the applications function properly (at least in the single- user case) before you begin load testing. Remember: You cannot performance test broken applications. They behave erratically and may exhibit much better or much worse performance than a correct version. After completing function and integration testing, it's time to make sure that your application and supporting infrastructure supports the expected traffic volumes .

Start your performance test planning by setting performance goals. Without clearly defined goals, performance tests tend to continue indefinitely. Any web site contains an infinite number of bottlenecks. Solving one bottleneck merely exposes the next one. Without clear targets, your test team never knows if and when the web site reaches a sufficient level of performance to support its expected traffic. Setting goals lets you know when the web site is ready for production.

In Chapter 1, we discussed some of the basic performance concepts, such as peak load, throughput, response time, and scalability. In this chapter, we discuss how to develop performance goals for each of these concepts.

Peak Load

The performance test must simulate the web site's peak load. As we discussed in Chapter 1, if the web site does not perform well at peak loading, it does not really perform at all. We use the performance test in conjunction with the anticipated peak load to accomplish the following:

• Determine if the web site currently supports this goal.

• Tune an underperforming web site so that it supports peak loading.

• Determine the hardware capacity required by the web site at peak.

Surprisingly, many performance tests overlook the importance of peak loading. These tests frequently use average load numbers as their performance test targets. However, as Figure 6.1 demonstrates , average load often differs dramatically from the peak. Merely testing for average load sometimes means your web site cannot support key periods over the course of a year or even a day.

Figure 6.1. Peak versus average web site load

Also, use the numbers from your web site's peak usage period. As we discussed in Chapter 5, many e-Commerce web sites experience their busiest days during the Christmas shopping season. Take your peak daily load from the busiest day of your busiest season .

While determining the peak load sounds simple enough, in practice it often proves challenging. If the site currently exists in a production environment, use data from the existing site to gauge traffic patterns and peak usage. Having said this, we should say as well that we encounter many web masters and web site teams who have no idea how many users access the site per hour , per day, or even per year! If the web site team lacks this data, many tools exist to pull site usage statistics from HTTP access logs.

An existing thick client application also presents another avenue for predicting peak load. For example, reviewing the host database transactions generated by the thick client often provides insight into peak loading. (To generalize a bit, DBAs tend to keep much better records than web site teams anyway.) Of course, this technique works best when the web site under test is the planned replacement for the thick client application.

Even data from similar web sites frequently proves useful. If these sites serve the same target audience as your web site, or provide a similar class of functionality, their traffic patterns might help you set expectations for your site. The key word here is similar. Arbitrarily selecting a web site as a pattern for a completely different site rarely works, as we discussed in Chapter 5. For example, assume your company supports both an e-Commerce site selling retail clothing and a site supporting financial interactions. These two web sites receive very different patterns and peak load volume; you cannot use the data from either site to predict the peak load of the other. However, the traffic patterns from the "shoe store" portion of the e-Commerce site probably apply quite well to a new "handbag store" function on this same site.

If you performance test a completely new web site or function (or target the web site to a new user market), estimating peak load becomes more difficult. In these cases you rely heavily on projections based on any available market research. This includes any public data available on similar web sites operated by other companies. However, in the end, you end up with a best guess in these situations. We suggest generously buffering any peak load estimate you develop for a completely new web site.

As you gather these numbers, don't forget about growth data as well as current usage data. Predicting future usage is also tricky, but important. By the time you finish a performance test, today's peak load no longer applies. Analyze past growth, and discuss your early estimates with marketing and development to generate your growth targets. In particular, check with marketing for any plans that might dramatically increase load. Promotional mailings , television advertising, or new products all stand to dramatically impact web site loads. Again, generously buffer any load-growth estimates you develop.

Even though the performance teams we advise often begin their test planning without good peak loading estimates, these same teams usually develop reasonable and relevant estimates after consulting the available data. Teams without loading estimates often release web sites without sufficient traffic capacity. At the other extreme, we sometimes find teams building enormous web site infrastructures to support tiny user loads. Good loading estimates lead to a properly sized web site.

Peak loading also provides the basis for throughput estimates. In the next section, we discuss how to use rough load data to develop some estimates for both peak loading and throughput.

Throughput Estimates

As you begin your search for load and throughput estimates, expect an influx of ill-defined goals. For example, we often hear requirements such as, "The site needs to support 10,000 hits a day," or "We expect 10 million users per year," or (the most ill-defined) "We anticipate 100,000 users a day." You must turn this data into reasonable estimates for building a performance test. In this section, we demonstrate some of the formulas we use for turning rough guesses into the estimates you need. Appendix A also presents these calculations in spreadsheet format to help you better organize your data.

Important! After developing rough loading and throughput estimates, feed them back to the other teams involved with the web site (such as marketing or management) for validation. These numbers only work if they reflect reality. All of those involved with the web site must agree with the targets you develop for the performance test and, ultimately, for the production web site itself.

Hits per Day Estimates

So the marketing team tells you, "Expect 10,000 hits per day for the site." Before you take this estimate at face value, we suggest asking a few questions. As mentioned in Chapter 1, Internet terminology often varies between companies or even between departments in the same company. Let's use the marketing team's statement to explore how terminology usage impacts performance estimates.

What Is a Hit?

A "hit" potentially means a couple of different things. For an HTTP server specialist, a hit means any request sent to the HTTP server. Because HTML pages usually contain embedded elements such as gifs or jpegs, one HTML page might result in multiple HTTP "hits" as the browser retrieves all of the elements from the server to build a page. Figure 6.2 shows a web page constructed from multiple HTTP requests .

Figure 6.2. One HTML assembled from multiple HTTP requests

If your web master tracks site activity via the HTTP access logs, a hit might actually mean just an HTTP request rather than a full page. For example, if the average page on the site contains four embedded elements, our marketing team's "10,000 hits per day" number translates to become as few as 2,000 page requests per day:

 Requests to display a full page = 1 page request + 4 embedded element requests                                 = 5 requests 10,000 requests per day / 5 requests per page = 2,000 pages per day 

This assumes every request for a page also requires the browser to retrieve all the embedded elements. This isn't always the case, as most browsers cache static page elements so they do not have to request them repeatedly. If a web site's pages contain many of the same embedded static elements on every page (for example, a banner common to all pages), the user's browser may not retrieve this element again after the first page of the user's visit.

These considerations become important as we try to find out how much dynamic content our web site serves. We need these estimates to develop some key capacity planning ratios for our web site. We want to know how many HTTP servers we require as compared to application servers. This also allows us to predict the value of specialized components , such as caching proxy servers, within our web site.

In the example above, let's assume our page request is a dynamic page request requiring service by the application server. Let's also assume the static elements do not require interaction with the application server (they're handled by the HTTP server, not the application server). In this case, our dynamic request to static request ratio is 1:4. Establishing this ratio helps us to better tune our web site (for example, configuring the ratio between HTTP server listeners and application server threads). It also helps us to plan capacity for the application servers if we understand how much traffic they receive.

Returning to our example of 10,000 hits per day, we know this number resolves to 2,000 page hits (representing dynamic content) if the browser requests all the embedded static elements for each page. If the browser caches static elements, the 10,000 hits per day measurement contains a higher percentage of dynamic page requests versus static elements. Consider the impact of caching on your request ratios.

Regrettably, the rest of the world uses the term hit in very ambiguous ways. We've seen hit used to refer to an entire page, including embedded elements or even entire framesets. Also, many companies routinely use hit to mean an entire site visit by a given user. A site visit usually encompasses many pages, not to mention the embedded elements and frames included in those pages.

This leads us to an important point: Use performance terminology consistently. However you define a hit, define it explicitly, and stick with the definition uniformly throughout your planning and testing. Terminology misunderstandings make for unsuccessful performance tests.

What Is a Day?

Beyond the concept of a hit, what does the marketing team mean by the term day? In the world of web sites, even the concept of a day lies open for interpretation. A day usually defines the primary period of activity for a web site. Many, many web site teams wrongly assume their site will be fully populated by happy users 24 hours a day. With few exceptions (such as worldwide sporting event sites), the average web site does not receive a great deal of traffic on a round-the-clock basis. Instead, most users arrive during the period of a business day. Defining the day becomes very important for determining the peak load on the web site. As shown in Figure 6.3, overestimating the length of the day conversely underestimates the peak pressure on the site. Determining the period of time when most users arrive on the system becomes critical for estimating peak load.

Figure 6.3. The same daily traffic volume spread over both 8- and 24-hour days

Determining Peak Throughput from Hits per Day

Given 10,000 hits per day, how can we turn this number into a peak throughput estimate? The ideal estimate comes from the traffic patterns of an existing web site or application, as mentioned above. For example, if the web master tells us to expect 30% of the traffic during the peak hour, ^[1] expect 3,000 hits per hour during the peak period. This number gives us 0.8 hits/sec during the peak hour. Here's the math to get these numbers:

^[1] Your web site may track a smaller or larger peak time interval. Use the peak interval your team can describe with the most confidence.

 10,000 hits/day * 30% of traffic at peak hour = 3,000 hits during peak hour 3,000 hits per hour / 60 minutes per hour / 60 seconds per minute = 0.83 hits/second 

Again, keep in mind, depending on the definition of hit, this may resolve to more traffic. For example, if a hit really represents a whole page containing 30 embedded elements, this number resolves to as many as 24 HTTP requests per second. Understanding the meaning behind the number makes the difference in these estimates.

Frequently, however, you receive a daily or yearly traffic rate but no guidance on how much of the traffic arrives during the peak period. In these cases, applying some calculations to the available information yields a rough peak estimate. This algorithm first spreads the traffic evenly over the web site's workday . For this example, let's assume the web site receives most of its traffic during an eight-hour period:

 10,000 hits per day / 8 hours per day = 1,250 hits/hour 

Notice that this gives us an even distribution with no consideration for peak loading. To estimate the peak, we normally assume three to five times the weight of the daily average. For safety, in this case, let's assume the peak is five times the average.

 1,250 hits/hour * 5 = 6,250 hits/hour 

This gives us the following value for the per second hit rate at peak:

 6,250 hits per hour / 60 minutes per hour / 60 seconds per minute = 1.7 hits/second 

After arriving at this step, we proceed as follows .

• Convert the number to peak HTTP requests. For example, if our numbers represent page requests instead of HTTP requests, we do more work. If each page request translates to seven HTTP requests, the peak HTTP request rate becomes

   6,250 hits per hour * 7 HTTP requests per hit = 43,750 HTTP requests per hour 43,750 HTTP requests per hour  12.15 HTTP requests per second 

• After converting to HTTP requests, apply the dynamic to static ratio. From an earlier example, we know each dynamic page request generates four additional static requests. This means that, of our 12.15 requests/second:

   12.15 hits per second * 1/5 = 2.43 dynamic hits per second 12.15 hits per second * 4/5 = 9.72 static hits per second 

• Validate these calculations with others involved in the site, such as marketing or management. Always review the calculations in light of the real site.

Users per Day Estimates

Regrettably, the hit does not reign as the most ill-defined term in web site performance. The term user leads the pack as the most misunderstood performance metric. Often, the web site team receives guidelines for site performance such as "the site needs to support 100,000 users per day." Of course, this doesn't give us any information about what these 100,000 users will be doing on the web site.

So, once again, we must dig into the number to determine how the users interact with the site. Likewise, we need to develop some idea of how the site is constructed to understand the burden the typical user interaction places on the web site. Finally, we use all of this information to develop peak load estimates.

What Is a User?

User usually means one of three things in terms of web site usage:

1. One of potentially many " visits " the user will make to the web site over the course of a day

2. A complete set of activities accomplished by the user on the site for the whole day

3. A hit

Users as Hits

Let's start with the last definition: a hit. Many web masters and web site teams use hit and user interchangeably. So when they say "user," they really mean "user request." So, in this case, they expect 100,000 user requests per day. Usually, this translates best to pages, but, as with the discussion above, hit can mean a variety of things.

All Day Visitors

The second definition is often used in B2B, portal, and some other classes of web sites where the users log on to the site for the entire day. In these cases, a user represents a complete set of activities accomplished by the user for the whole day. These users may or may not be very active. If they infrequently make requests throughout the day, the primary web site burden may well be the "footprint" of the user on the system.

This user "footprint" includes any resources required to keep the user active on the system all day. For example, the system sometimes uses memory to keep HTTP session data about the user. Also, the web site may save the user's session to a persistent HTTP session database. The performance team must design a test for these systems simulating the burden of logged-on, but largely dormant , concurrent users.

User Visits

The term visit defines the most common usage of user. The web site receives many visitors over the course of the day. These visitors interact with the web site, performing one or more tasks that usually involve navigating through multiple pages. The user stays logged on to the site until shortly after his activity finishes (usually 15 or 30 minutes after the last request the user makes). After this timeout period, the web site removes the user's transient information (the HTTP session data) from the site. If the user wants to interact with the site later, he must log on to the site again.

User Scenarios

Regardless of the loading goal (users, hits, or transactions), eventually you must determine the load's distribution, or how the users interact with the web site. Usually, this requires developing several scenarios and determining the percentage of traffic executing each scenario at any given time.

For example, an e-Commerce site might receive users who want to browse, purchase, or check the status of an order. A financial site might receive visitors who want to check their portfolio value, trade stocks, or do research on a particular company. We then develop these usage scenarios as test scripts for each of these tasks, and estimate their relative execution frequency.

Scenarios come from several sources. Usability testing or analysis of site logs gives the most reliable data about usage patterns. For new web sites without the benefit of usability testing, we often look to use cases from the analysis/design phase of the web application to provide scenarios. However, the best scenarios involve real data on how the users interact with the web site. (See Chapter 7 for more details on developing test scripts.)

On the basis of these scenarios, we develop an idea of how many pages the users access for each scenario. However, we also need an estimate for the duration of the average visit. A good choice for this number is how long it takes us to run the scenario manually (see the caveats applying to this in Chapter 7). Even better, we might obtain some data from early usability testing. However, most sites do not perform rigorous usability testing, so a rough guess from walking through the scenarios is a good first start.

The frequency of each scenario is important as well. We must determine how many users execute each scenario over the course of the day. The scenario utilization becomes very important if some scenarios require "heavier" activities, such as database interactions, or a large number of page visits per scenario. Usually, we derive the scenario mix from the existing web site or receive this weighting from the marketing team. Scenarios tell what our traffic is doing. For "users per day" estimates, we also need the scenarios to tell us how much traffic our users generate during peak periods.

User Visit Projections

So what do user arrival numbers mean for web site load and throughput projections? Let's assume the following information from the marketing team:

• 100,000 user visits per day

• 30% of traffic arrives during peak hour

After some clever and dedicated investigation work, you uncover the following information about the web site:

• Average user visit: 10 minutes

• Session timeout: 15 minutes

• Average pages per visit: 5

During the peak hour, the web site has the following new user arrival rate:

 100,000 users * 30% peak users = 30,000 peak users 30,000 peak users/hour  8.33 new users/second 

Over a ten-minute visit, at a minimum, the web site contains the following concurrent users:

 8.33 new users/second * 10 minutes * 60 seconds/minute = 4,998 concurrent users (approx. 5,000) 

These users typically stay active for ten minutes and visit five pages during that time. This gives us the following page rate:

 5000 users * 5 pages per user / 10 minutes / 60 seconds per minute = 41.667 pages/second   (approx. 42 pages/sec) 

If each page contains an average of five embedded static elements per page, the overall throughput (in requests per second) is as follows:

 ((1 initial request) + (5 static requests)) * 42 pages/second = 252 requests/second 

Obviously, browser caching might reduce this request burden, but it serves as an upper bound for testing purposes.

User arrival rates and active log-on rates also help determine the HTTP session pressure of these users. If the Java web site uses HTTP sessions, each user may use memory on one or more application servers. This memory only frees after the user's HTTP session times out (or, in some cases, when the user logs out of the web site).

Let's assume each concurrent user requires about 2KB of memory. Since the average user visit is 10 minutes and, in this case, the HTTP session timeout is 15 minutes, the average session stays in memory a total of 25 minutes. Therefore, the server requires the following memory to hold these sessions:

 8.33 users/second * (25 minutes) * 60 seconds/minute = 12,495 users (let's say 12,500) 12,500 users * 2KB/user = 25MB of memory 

Of course, this equation does not take into account any users still logged on from previous arrivals outside the 15-minute window. (Remember, the 10-minute visit time is an average.)

Response Time Measurements

The web site requirements consist not only of load, but also responsiveness to the load. Response time requirements may be very general, such as "no more than five-second response time for any page at peak load." Some teams receive requirements broken out by pages. For example, a financial site might require two-second response time on stock quotes, but accept ten seconds for research pages.

Response time requirements change as the industry changes. The goal is to keep your response time consistent or better than the customer's experience at other web sites. Usually your QA or marketing teams provide response time targets based on current industry "standards."