Monitoring Your ColdFusion MX 7 Server

Monitoring system performance involves two major approaches: historical analysis and active system monitoring.

You can incorporate many methods into your monitoring activities. Usually you need to implement a combination of monitoring activities into your infrastructure to comprehensively monitor the site. Let's first discuss analysis of past system performance, or historical analysis. Next we will look at active system monitoring of ColdFusion Web serverswhich may involve setting up server probes, utilizing performance monitors and third-party utilities, and other techniques.

Developing a Baseline

One of the first things you want to do before you start monitoring a application is have or create a 'baseline' for you system. A baseline is a set of metrics that define how your system and application should behave under normal conditions. Usually you do this by using a load and application testing system such as OpenSta to develop a performance baseline for your application. You should know exactly how much memory your application uses, how many requests a second it can handle, the level of CPU usage during normal operation, and so on (Chapter 1 describes various testing tools for gathering this information.

Once you have created your system baseline, you'll have a set of metrics against which to compare your system's operation, which will help you in locating problems. Every time you make a significant changes to the system, apply patches, upgrade ColdFusion, or apply new code, run your tests again and compare the system performance with your original baseline data. Do this as soon as you make the changes on your test environment, so you can see before deployment whether the alterations will have a negative impact on the application's overall performance, stability, and viability.

Historical Analysis

One of the most important methods for understanding your application, successfully troubleshooting it, and improving it, is historical analysis. Historical analysis is simply comparing data collected from your system over time to see what patterns, trends, or changes emerge. You can use historical analysis to detect security threats, bugs, problems with performance, bottlenecks, and much more.

ColdFusion offers several features for analyzing historical performance on your application server. By combining this data with other information stored on the server, you can create a clear picture of how your application is performing.

Analyzing ColdFusion Log Files

A consistently small system log file correlates to a healthy Web site. Regular monitoring of the ColdFusion log files is a key component to maintaining and improving your ColdFusion applications. Concentrating on reducing the number of errors that appear in the log will eventually produce a healthier, more responsive site. ColdFusion log files consist of several files representative of functions within the ColdFusion server, shown in Table 2.1.

Table 2.1. ColdFusion Log Files

LOG FILE NAME	DESCRIPTION
Application.log	Records every ColdFusion error on your site.
Exception.log	Records stack traces for exceptions that occur in the server.
Server.log	Records errors for the ColdFusion server.
Scheduler.log	Records scheduled events. Indicates initiated events and whether they succeeded.
Customtag.log	Records errors in custom tags.
Car.log	Records errors associated with site archive and restore.
Mail.log	Records errors generated when sending mail through a mail server.
Mailsent.log	Records email messages sent.
Jrun.log	When ColdFusion is connected to an external web server, this log stores Java runtime errors. It is stored in [cfusionmx]\ runtime\lib\wsconfig\1.
Flash.log	Records Flash Remoting errors.
Eventgateway.log	New in ColdFusion MX 7, this log records event gateway events.

The Application.log file records every ColdFusion error on your site. Two types of errors in particular clearly indicate a performance problem.

The first is a "Request timed out" message. This error comes up if a ColdFusion page takes longer to process than the TIMEOUT value you set in the ColdFusion Administrator. If your server is experiencing performance problems, some pages take so long to process that they trigger this error. If you have set your TIMEOUT value to 20 seconds, you have no way of knowing whether the pages that aren't timing out are taking 5 seconds or 15 seconds to process. If you're getting "Request timed out" errors for only a few specific ColdFusion pages, odds are those pages are at least one source of your performance problems. If these errors are spread evenly across most or all of the pages on your site, a single bottleneck may be affecting everything.

It's normal for ColdFusion to rely heavily on the processor and to grab memory as necessary to pull a large number of records from a database. Memory usage should climb, plateau, and then release. However, if you find that memory use on your Web server is increasing without ever releasing, look in the application log for database-related activity. If you find many errors (and especially if you see entire queries in the application log with associated errors), examine your database queries and see how you can tighten them up. Other issues can create what appear to be memory leaks, from threads never releasing, to synchronization issues. One of your best resources for solving these sorts of problems is the collection of log files described here.

The Exception.log is one of the more useful ColdFusion logs and probably the first place you should look if you are seeing ColdFusion application errors. Many ColdFusion developers are uncomfortable with the exception log because it shows Java stack traces, which are very detailed and full of seemingly undecipherable Java messages. When properly understood, however, the stack traces offer detailed information on exactly what was happening when the system experienced the problem that created the exception. A full discussion of Java stack traces is beyond the scope of this book, but you'll find an excellent introduction to this topic at http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/. Once you understand these exception messages, you can quickly determine hung threads, failed requests, memory problems, and much more. Even without a full grasp of the stack traces, you will be able to see which file threw the exception, at what time, and what part of ColdFusion experienced an issue.

Though not as useful at first glance as the exception log, the Server.log also provides information related to the stability of your Web servers that might further substantiate your application log findings. Search this log for "ColdFusion started," which indicates how often your Web server has been started and stopped.

Other Logs

ColdFusion MX 7 provides two places for tracking long-running requests. In Debugging Settings, you can set a benchmark (in milliseconds) and display any requests that take longer than the setting in the debug output. Additionally, you can log all pages that run longer than a given number of seconds to your Server.log. See Figure 2.1 for setting logging of long-running pages.

Figure 2.1. Log pages that are running too long.

Periodically reviewing all of your logs will create a clear picture of how your ColdFusion applications are functioning and provide information for resolving issues that may arise during the normal course of operation.

JRun Metrics

If you are using the standard version of ColdFusion, or ColdFusion for J2EE deployed on JRun, you can take advantage of yet another set of log files, as well as the enhanced logging capability in JRun. Look in the ColdFusion MX 7 install directory and go to \runtime\logs. You will see the logs listed in Table 2.2.

Table 2.2. JRun Logs for ColdFusion

LOG NAME	FUNCTION
coldfusion-err.log	ColdFusion stack traces.
coldfusion-event.log	ColdFusion event log.
coldfusion-out.log	ColdFusion services events, and data from garbage collection.

Each of these logs supplies additional information about ColdFusion, although the data is usually not particularly helpful in that much of it is duplicated in the standard ColdFusion logs.

Enhanced ColdFusion logging is another tool that can provide extremely detailed and specific information about a variety of ColdFusion metrics. These metrics are listed in Table 2.3.

Table 2.3. JRun Metrics

METRIC	DESCRIPTION
listenTh	Threads listening for a new connection.
idleTh	Threads waiting for a new request.
delayTh	Threads waiting to run.
busyTh	Threads currently running.
totalTh	Total worker thread count.
delayRq	Requests delayed due to high concurrency.
droppedRq	Requests dropped.
handledRq	Requests handled.
handledMs	Milliseconds spent servicing requests not including any delay time (delayMs).
delayMs	Milliseconds spent in delay state.
bytesIn	Bytes read from requests.
bytesOut	Bytes written to responses.
freeMemory	Kilobytes of free memory in the heap.
totalMemory	Total kilobytes in the heap (in use and free).
sessions	Current number of active J2EE sessions.
sessionsInMem	Number of J2EE sessions in memory.

As you can see, this is an extensive list of very detailed information! Using JRun metrics, you can see when threads are hanging, when specific requests are not responding, the amount of free memory, and more. The only disadvantage to using JRun metrics is that when they are turned on, they have an impact on system performance. In addition, this feature creates some very large log files, so it's best to enable it only when you are troubleshooting.

To enable JRun metrics, first find your JRun.XML file, usually found in the ColdFusion root. The path will look something like this:

 C:\CFusionMX7\runtime\servers\coldfusion\SERVER-INF\jrun.xml 

After you have found this file you should make a back up since we are going to change this file and if you make any mistakes ColdFusion server may not restart. Usually it is easy enough just make a copy of jrun.xml and call it jrun_original.xml or whatever is easy for you to remember. Now that you have made a back up copy open the jrun.xml , open it with any text editor. Look for this entry and uncomment it:

 <service  name="MetricsService"> <attribute name="bindToJNDI">true</attribute> </service> 

Then find this entry:

 <service  name="LoggerService">  

and edit its attributes. You want to change metricsEnabled to True, and meTRicsLogFrequncy (the interval, in seconds, to log metrics) to somewhere between 5 and 10 seconds. So you should have something that looks like this:

 <attribute name="metricsEnabled">true</attribute> <attribute name="metricsLogFrequency">5</attribute> 

The information that will be recorded to the logs will look something like this:

 10/14 12:11:23 metrics Web threads (busy/total): 0/2 Sessions: 2 Total Memory=7052 Free=3303 

Next, split the logged information into its own log file. Do this by editing this line:

 <attribute name="filename">{jrun.rootdir}/logs/{jrun.server.name}-event.log</attribute>  

to this:

[View full width]
 <attribute name="filename">{jrun.rootdir}/logs/{jrun.server.name}- {log.level}. log< /attribute> 

Once you have changed the jrun.xml file you will need to save the file and restart ColdFusion for these settings to take effect. Once you restart ColdFusion you will see that all the metrics data you record while troubleshooting will go to a separate log, coldfusion-metrics.log.

NOTE

Don't forget that turning on JRun metrics has a performance impact on your server and rapidly creates very large log files. Make sure that once you have resolved your issues or captured all the information you need, turn off the JRun metrics.

Analyzing Web Server Log Files

Sherlock Holmes often said "Eliminate all other factors, and the one which remains must be the truth." Trying to debug and troubleshoot your application can sometimes seem an impossible task when the problem is not apparent or obvious. In these cases you need to look methodically at everything that may be affecting your system. One of the major components in a ColdFusion application is the Web server, and careful monitoring and analysis of its logs are crucial to maintaining system performance. Additionally, the Web server logs often provide clues to other issues, such as gaps that might allow security breaches or attacks, as well as information on how real users are experiencing the application.

For all these reasons, good log-analysis software is essential in analyzing your Web server's log files. Since Web server log analysis isn't specific to ColdFusion we will not delve deeply into it. You should know, however, that without a good log-analysis tool, you'd be severely handicapped in all your other performance-analyzing ventures. If you don't have a log-analysis tool right now, you might like to look at Analog (http://www.analog.cx/), a free tool written in Perl. If you need more-powerful analytic tools, consider WebTrends's Enterprise Suite (http://www.webtrends.com/), one of the most popular commercial Web log-analysis tools. It offers ad hoc reporting and graphical representations of logged data.

Analyzing Web server logs will tell you about visits, users dropping off in the middle of a transaction, and general user activity. You can set up your Web server to store valuable statistics about your site. These can be very beneficial for tracking information about your site and then comparing the data to your load testing data. You can find how many users are visiting, peak loads, page-load times, and most-visited site sections, among other information. This analysis can also show where visitors are leaving your site, maybe due to problems such as errors or slow page-load times. Understanding how your users interact with your site can be very beneficial in creating a high-performance Web site.