Section 3.1. Expectations About Data Collection | Optimizing Oracle Performance

3.1 Expectations About Data Collection

The whole point of data collection in a performance improvement project using Method R is to collect response time data for a correctly targeted user action . No more, no less. Unfortunately, many application designers have complicated the data collection job significantly by providing insufficient instrumentation in their applications.

Many companies, especially Oracle Corporation, are improving the response time instrumentation in newer application releases.

The data collection lessons you learn in this chapter will make data collection seem more difficult than you had probably expected. The benefit of doing it right is that you will reduce your overall project costs and durations by eliminating expensive and frustrating trial-and-error analysis/response iterations.

A Method R performance improvement project proceeds much differently than a project that uses the conventional trial-and-error approach introduced as Method C in Chapter 1. Figure 3-1 illustrates the difference. A project practitioner typically begins to feel like he is making real progress when the targeting and data collection phases are complete and he enters the analysis/response phase of a performance improvement project. The Method C practitioner typically reaches this milestone ( marked t ₁ in Figure 3-1) before the Method R practitioner working on the same problem would (marked t ₂ ). If you don't expect this, it can become a political sensitivity in a Method R project. The time between t ₁ and t ₂ is when your risk of losing commitment to the new method is at its greatest.

Figure 3-1. The targeting and diagnostic data collection phases of Method R consume more time than in conventional methods , but total project duration is typically much shorter

Finishing the data collection phase quickly is not the goal of a performance improvement project. The correct goal is to optimize the system with the smallest possible investment of resources. Method R is optimized for this goal. In fact, my colleagues and I created Method R specifically to help our customers fix performance improvement projects that had dragged on for weeks or even months without meaningful progress. In the overwhelming majority of Method R projects we've led, we've been able to demonstrate how to achieve the optimization goal within one hour of obtaining correctly scoped diagnostic data. Once you have collected the right performance diagnostic data, Method R will require only a single analysis/response phase before you'll make progress against your targeted user action.

Method C practitioners spend most of their time in trial-and-error mode trying to determine the cause-effect relationship between the hundreds of possible problem causes and the business symptom that is most in need of improvement. A huge inefficiency of Method C is the need to perform, on average, several iterations of analysis and response activities before you'll stumble upon a symptom's root cause. Each iteration of analysis and response tends to consume more time than the prior one, because analysts usually try the easiest responses they can think of first, saving the more time-consuming and expensive tuning activities for later in the project after the cheaper ideas are discarded.

The final blow to Method C is that there's really no quantitative way to determine when you're "finished tuning." In many projects, Method C users never positively identify an actual contributory cause of a performance problem. Even in "successful" projects, practitioners spend weeks, months, or even years without really knowing whether a targeted performance problem has been truly perfected ( optimized ) or merely partially improved ( tuned ). The problem of not knowing whether a user action could be further tuned leads to a condition that Gaja Vaidyanatha and Kirti Deshpande cleverly call Compulsive Tuning Disorder , or CTD [Vaidyanatha and Deshpande (2001) 8]. I joke that CTD is a debilitating condition caused by hope . More specifically, CTD is caused by an absence of complete information that would allow you to prove conclusively whether the performance of a given user action has any room for improvement. Method R fills this information gap, eliminating the possibility of CTD.

The first time you use Method R, collecting the diagnostic data will probably be the most difficult phase of your project. For some applications, diagnostic data collection is a cake walk. For other applications, proper diagnostic data collection can legitimately become quite a difficult challenge. Chapter 6 describes which kinds of applications are easy and which are hard, and it illustrates some of the techniques that my colleagues and I have used to overcome various challenges. The good news is that once you've figured out how to collect good diagnostic data for a targeted user action in your application, the process will be much easier and less time-consuming on your next performance improvement project. Method C, on the other hand, will always suffer from the problem of multiple analysis/response iterations, regardless of where you are on the experience curve.

I believe that in the future, most application software vendors will make it very easy for users and analysts alike to collect precisely the diagnostic data that Method R requires. Newer releases of Oracle's E-Business Suite are simplifying the diagnostic data collection process, and everything I hear indicates that the Oracle release 10 kernel and application server software are headed in the same direction. If the dominance of methods analogous to Method R in other industries is any indication, then success in simplifying diagnostic data collection should practically universalize the adoption of Method R for Oracle performance improvement projects.

Different Methods for Different Performance Problems?

Could it be that conventional methods are more effective for "simple" performance tuning problems, and that Method R is more effective for "complex" ones? The problem with that question is this: How do you know whether a performance tuning problem is "simple" or "complex" without engaging in some kind of diagnostic data collection?

One approach that we considered during the construction of Method R was to collect very easy-to-obtain diagnostic data to use in deciding whether the more difficult-to-obtain diagnostic data were even necessary to collect. We found this approach to be sub-optimal. The problem with it is that there's virtually no situation in which you can be certain about cause-effect relationships without the correct diagnostic data (and of course, sometimes the correct diagnostic data are difficult to obtain). The doubt and ambiguity that are admitted into a project by the analysis of easy-to-obtain diagnostic data rapidly deteriorate the efficiency of a performance improvement project. The thought-blocking fixations that I've seen caused by bad diagnostic data at many projects remind me of a wonderful quotation attributed to Cardinal Thomas Wolsey (1471-1530): "Be very, very careful what you put into that head, because you will never, ever get it out."

A dominant goal during the construction of Method R was that it must be deterministic . Determinism is a key attribute that determines how teachable (or automate-able) a method can be. We wanted to ensure that any two people executing Method R upon a given performance problem would perform the same sequence of tasks , without having to appeal to experience, intuition, or luck to determine which step to take next. Our method achieves this by creating a single point of entry, and a well-defined sequence of if-then-else instructions at every decision point thereafter.

Top