Application-Level Metrics

Applications require instrumentation to make their behavior observable, and, as appropriate, controllable. Client-side collectors operating in passive or active modes provide some of the instrumentation because they measure the user experience from that location. Note that client-side collectors are usually not a part of the application instrumentation itself; they measure the application behavior for a specific virtual (synthetic) transaction.

Table 8-1 shows examples of application instrumentation for a web sales application. The instrumentation can be further divided into internal and external measurements:

• The internal measurements give insight into the behavior of the systems within the direct control of the IT group. There are categories of internal measurements, including those for workload, customer behavior, and business behavior.

• External measurements show the behavior as seen by an end user outside the scope of the IT organization, such as by an end user using the Internet to access the system and perform a transaction.

Table 8-1. Examples of Application Metrics for a Web Sales Application

Measurement	Insight
Workload
Number of transactions/second	Overall functioning; relationship of load versus performance
Peak transaction loads	Stress points; average:peak ratios
Number of concurrent connections	Overall functioning; relationship of load versus performance
Peak connection volume	Stress points; average:peak ratios
Web page volume or traffic volumes	Gauges of activity
Customer Behavior Measurement
Favorite content	Content optimization, choice of replication sites, and cache preloading
Ratio of visitors to sales	Effectiveness of content
Forward and reverse path analysis	Path analysis and navigation
Start and stop pages	Path analysis and navigation
Average pages per visit	Navigation
Average visit time	Stickiness
Business Measurement
Number of completed orders	Tracking site efficiency
Revenues generated	Tracking actual business generated
Abandoned carts	Tracking customer behavior
Promotion feedback	Tracking marketing effectiveness
Service Quality Measurement
Transaction response time and availability	Measure supporting services

Workload

Workload metrics track the capacity of the application. Capacity is determined by the quality of the implementation and the assigned computing, storage, and network resources. Some measurements of overall activity and capacity might include the number of transactions per second, the number of concurrent connections, or the actual server loading measurements.

Periodic measurements of workload can build activity baselines that profile the normal ranges over longer time intervals. Alerts can then be generated when the comparison of the actual workload against the baselines indicates a trend away from the normal ranges.

I recently visited a site that had melted down when a web page designer added two simple objects, and another 45 Kb of payload, to the home page. Testing in the lab showed no apparent bugs, so the new page was placed in production. The problems began appearing during times of heavy customer access. When there were over 10,000 active connections, which occurred during the hours of peak demand, the additional load was 450 MB being sent across the network for users downloading the home pages. As it turned out, this bump on the backbone was actually the final straw that convinced this organization to outsource their content delivery to a managed infrastructure provider.

Customer Behavior Measurement

Measuring customer behavior is essential for several reasons. Customer behavior offers great feedback and insight into application effectiveness. This is particularly important in web applications because they are the major customer-facing applications. Tuning a customer-facing application is more challenging than tuning individual elements or even an infrastructure.

The same technology tuning needs to occur for rapid page access and high transaction volumes. However, customer-facing applications must also meet business goals as well. These goals depend upon the characteristics of the web site itselforganization, structure, navigation, and layout. Content must be compelling and easy to navigate, and processes must be as simple as possible.

For example, sites that provide content may want a sticky environment that keeps customers at the site for extended periods of time. On the other hand, sites dependent on serving ads and promotions will use each new page as another selling opportunity.

In contrast, sites that provide information or products want to get their customers to the right content as quickly as possible, transact their business, and move on to the next customer. Clumsy navigation and excessive links discourage this type of consumer.

Measurements of the average number of pages a customer uses during a visit or the average length of a web site visit can track the overall effectiveness of the site. These metrics assess either type of site.

Other aspects of customer behavior are used to optimize application performance. For example, tracking the most heavily used content or the most frequent transactions gives valuable information for improving the effectiveness of the customer-facing application. For example, one site found that one of the most frequently accessed pages took five clicks to reach. This was a business site wanting short visits and quick navigation. The desired content was moved to the home page, resulting in improved customer satisfaction and increased revenue because fewer customers lost interest with more direct navigation to their desired destination.

The popularity of content also assists managers in making intelligent content placement, preloading caches, and determining the number of replication sites. The same value is provided by instrumentation that identifies the most frequently used transactions. Developers can focus their attention and optimize those transactions that will offer the highest payoff in improved performance.

Business Measurements

Business measurements are becoming increasingly important. They are directly important to business managers who want to understand how their online business is actually functioning in real time. These metrics are important to technology managers as well; being the source of critical business information establishes the value of better management investments.

Some examples of business metrics are completed orders, generated revenue, promotion feedback, and abandoned shopping carts.

A tally of completed orders indirectly measures the effectiveness of the web sitewhether it is keeping customer interest long enough to close sales, for example. This measures only bottom line effectiveness, not efficiency; however, it is a basic metric for many organizations at this time.

The completed orders metric can be broken into more details, such as the following:

• The ratio of customer orders to total customer visitors This is a measure of the percentage of visitors that actually buy. This is helpful when evaluating alternate page design strategies or navigation options.

• Active customers This is the identification of the best customers based on total sales, for example. Special promotions can be targeted to the best customers.

Generated revenue is calculated by measuring the cumulative revenue from completed orders. It gives business managers deeper insight into their current operation. They can compare the revenue generated against goals, or they can compare historical trends to gauge overall revenue growth. Other derived measurements could include building revenue baselinesplots of average run rates over the business day, for instance.

Promotion feedback can be invaluable when business managers and their marketing teams are always focusing on guiding users down a certain path to meet objectives such as strengthening the Internet brand, creating stronger differentiation with competitors, responding to market and competitor moves, and maintaining customer loyalty. They are under continuous pressure to capture a greater market share while simultaneously reducing customer acquisition costs.

Instrumentation can use special web pages, special buttons or links, or other ways of tracking responses to a variety of promotions. This information can be analyzed and organized to assess the effectiveness of different promotions and to understand acquisition costs.

For some reason, abandoned shopping carts always seem to get a business manager's attention. There have been some anecdotal reports that abandonment rates are often over 50 percent for some consumer sites. This should be distressing because these are potential buyers who have taken time to navigate the site and select products before they go to another site.

Business behavior metrics may be derived from other more basic measurements. For example, revenues are calculated after each order is completed. The basic revenues may be further segmented by the customer, the product, the time of day, a promotion, or other criteria. These metrics must be baselined, and thresholds should be established. Because business managers want to understand and respond to situations more quickly and because technology managers want to make adjustments to maintain compliance with Service Level Agreements (SLAs), an alarm can be sent to the appropriate business and technology managers when an application has a sudden drop in revenues or visitors.

Service Quality Measurement

For a web transaction, the principal service quality measurements are the external metrics of transaction response time and availability, both of which were first discussed in Chapter 2, "Service Level Management," in the section titled "High-Level Technical Metrics." They are further discussed in this chapter because of their importance.

Note that transaction time is a measure of how quickly a user can complete an end-user transaction on your system. If the user is a member of your own organization making an intranet transaction, poor performance will decrease productivity but might have no other bad effects. However, if the user is a web customer, and if the same function (for example, purchasing a book) can be accomplished more quickly, and with a greater chance of success, on a competitor's site, poor performance may cause transaction abandonment and loss of business. The faster speed of the competitor's site might be due to fewer pages being involved, or quicker downloads per page, or both; improved availability might be simply a side effect of shorter total times. To the end user on the Web, the key fact is that your competitor offers faster, more reliable service.

Each web page within the transaction should also be measured for download time because that can be a good indicator of user abandonment behavior. On legacy systems, users didn't see the computer screen directly; they spoke to call center operators. If the computer was slow, the operator would talk to the customer and save the sale. On the Web, the customers are directly exposed to slow web service, and they'll abandon a slow transaction. A two-minute transaction that consists of ten 12-second page downloads is considerably different from a two-minute transaction that consists of five 6-second page downloads and one 90-second download. Many users will abandon during that 90-second download. That's crucial information for the business groups and should be included in the SLA.