Monitoring your Terminal Servers

Day-to-day monitoring can be one of the most important things an administrator does in his Terminal Server environment. We're not talking about performance monitoring per se, as was covered extensively back in Chapter 13, but rather, we're referring to two types of monitoring:

• Real time monitoring of servers.

• Historical data collection for reporting and trend analysis.

Real time monitoring software allows you to view the status of different components of your Terminal Servers. Quality real time monitoring software will also alert you to problems currently happening in the environment and possibly take "self-medicating" measures to correct them.

Collecting historical data from your servers allows you to go back in time and observe server performance over a set of points in the past. This technique is most useful when you are trying to determine either when a problem started or if an unusually high counter is really that unusual. Collecting historical data should help prevent the following conversation:

Your Pager: <beep beep beep>

You: Uh-oh. The helpdesk is saying that Server A is really slow.

Other Administrator: Really?

You: Yep. I see here that the context switches are really high—over 40,000 per second.

Other Administrator: Really?

You: Yeah, they've been that way for over two minutes now.

Other Administrator: Really?

You: Yeah, really.

Other Administrator: Is that bad?

You: ?

Other Administrator: ?

You: Well, it seems high. I think.

Other Administrator: So that's bad?

You: ?

The point of this dialogue is that unless you monitor your servers on a regular basis, you won't know what thresholds to set for alerts and whether a counter is high or low when it comes time to troubleshoot a problem. Considering the scenario above, the context switches might have been just fine. There may be no problem at all, or another issue altogether may be the problem, but the administrators have no idea where to start since they don't know how their servers normally run.

Most people collect detailed metrics when building their servers, but fail to collect metrics after they've been in use for several months. When a problem occurs, they have no idea which (if any) performance counters are out of line.

There are several tools you can use to monitor your servers. The old reliable method is the Performance Monitor MMC snap-in that comes built in to Windows. While this tool is useful, it's not as feature-rich as third party software monitoring tools like Citrix's Resource Manager, Lakeside Software's SysTrack, or NetIQ's AppManager. Of course, any third party tool also comes with a price tag, while the Performance MMC is free.

Let's take a look at the power of the Performance Monitor MMC snap-in before we turn to third-party tools.

Performance Monitor

The Performance Monitor MMC snap-in (still called "Perfmon" by many administrators) is an example of a simple monitoring tool that's built in and automatically installed onto every windows server. When used properly, Perfmon acts as a basic real-time monitoring tool and can be configured to send alerts when problems occur. It can also be used to automatically create log files of your server's performance.

Before we jump into the details of configuring Perfmon for sending alerts, we should first determine which counters you should monitor and what types of thresholds you should set for each counter. If you haven't yet done so, read Chapter 13 detailing how to conduct a performance load test of your Terminal Servers. That chapter also explains more about the Perfmon counters you'll be using here.

Objects and Counters

After baselining your servers, you should have some ideas about where your counters will read in production. You can choose to monitor any performance counter, but you'll usually end up configuring monitoring rules for different counters than you monitored when tuning the performance of your servers.

• Processor | % Processor Time | _Total An alert here can be used to notify you if some process has spun out of control or if you simply need to add faster processors. You don't always have to set this alert for 100%. If your Terminal Servers never show more than 60% processor utilization, then you should probably set this alert for something like 70 or 75%. That way, you'll know when something out of the ordinary takes place.

• Memory | Available Mbytes It's easy to run out of memory on a Terminal Server, and this counter can give you an advanced warning of that. Let all of these alerts, watch your server to establish a baseline, and then set your alert accordingly.

• Physical Disk | % Disk Time When configuring this alert, make a separate one for each physical disk.

• Terminal Services | Active Sessions This alert will let you know if one of your servers is getting overloaded. An overloaded server could indicate a load-balancing related problem.

No matter what alert you're configuring, know beforehand your server's true limitations. If you baselined your servers and determined that maximum load to be 50 users after which performance goes downhill, set your threshold just below that. That way, you'll be notified before the user load overwhelms your servers.

Configuring your Alerts

Now that you've decided which counters you want to monitor and at which levels you want to be notified, let's see what involved to actually configure the notification process.

1. Open Performance Monitor (Start | Administrative Tools | Performance) or (Run "Perfmon" from a command prompt).

2. Expand "Performance Logs and Alerts" and right-click the Alerts icon. Select "New Alert Settings."

3. Name the Alert Settings you are configuring.

4. In the Properties window of the new alert, add the counters you want to monitor. For example, for CPU monitoring, click the "Add" button and highlight the "Processor Object | % of total Processor" counter.

5. Click the "Add" button. It will seem like nothing happened, but really the counter was added to your alert. This lets you add many different counters at once. When you've added all the counters you need, click "Close."

6. For this counter (a counter that goes up as utilization increases) select the "Alert when value is over" option and set the limit to what you need. (A limit of 94 will alert you when CPU utilization reaches 95% or higher.)

7. Change the sample data interval to 30 seconds.

8. Just configuring an alert doesn't do any good until you give it an action via the "Action" tab.

   • Log an entry into the application event log: This logs a simple event into the event log that can be viewed using event viewer. This is nice tool since one of your daily tasks should be to check the event logs (see the routine maintenance tasks section later in this chapter).

   • Send a network message to: This allows you to send a message to a specific computer name or IP address on the network. In order to use this, the messenger service will have to be running on the computer slated to receive the message. This wasn't always an issue, although now that spammers have learned how to use the messenger service, most companies keep it turned off.

   • Start performance data log: This allows you to specify a saved performance counter log to start when the alert is kicked off. The log is useful when you are only monitoring a subset of counters all day but want more detailed information when a problem starts.

   • Start this program: This allows you to start a program when the alert is raised. The program is usually a script that performs some action such as sending an email.

9. Finally, the "Schedule" tab allows you to start or stop the alert counter manually or set up prescheduled times for the monitors to run, generally during working hours so that they monitor the servers when they're heavily utilized.

Performance Logs

Performance Monitor can be used to save historical performance data. Most people use such logs to generate simple comma-separated CSV files that can be imported into Excel in order to create charts and graphs in a readable format for future use and for your management staff. On the other hand, store the performance data in binary format only if you feel like viewing it in the Performance Monitor MMC.

Either way, CSV files enable you to collect the data that you'll eventually sort through, compare, and present.

Configuring the Performance Log

Set up for Performance logs is similar to that of alerts.

1. Launch the Performance MMC snap-in and expand the "Performance Logs and Alerts" icon.

2. Right-click on the "Counter Logs" icon and select "New log settings." Give it a descriptive name and click "OK."

3. Add your performance objects or counters. You have the option to look at all the counters in an object by simply adding the entire object. Alternately, you can select only specific counters to log.

4. Select the sampling interval for the Log. Since this log will run continuously, you probably should set it to 30 seconds or 1 minute intervals. Less time will yield too much data for long-term samples.

5. On the "Log Files" tab, configure your log file as a comma delimited file. You can also click the configure button and select its location, filename, and a size limit.

6. At this point you can configure your log to run on a schedule (as with the alerts) or you can choose to launch it manually.

CSV files can be opened directly into an Excel worksheet and used in data analysis. Open the file with Excel, highlight a column for one of the counters (their names are in the first row of each column), and select "Insert | Chart." It's helpful to highlight the counter you're interested in along with the active sessions (or total sessions) counters. That way you can compare your specific counter against the number of active users on the system and begin to see how the resource utilization goes up as the user load increases.

If you run a log like this once a week and save the file, you'll have data from which to create charts for processor usage (or any other counter) and compare the utilization from one week to the next or even one month to the next.

Third Party Monitoring Tools

Performance Monitor is free and useful for simple monitoring in small environments. Unfortunately, it soon becomes inadequate once your Terminal Server farm starts to grow. At that point, you'll need to implement a third party product.

Components

All third-party tools are made up of the same basic four components.

• Agents

• Data storage

• Analysis and monitoring tools

• Reporting tools

Agents

Most third party tools work by installing an agent on each of your Terminal Servers. These agents then collect performance information from the server and write it to a central storage location (usually a database). In addition to being able to collect information from the standard Windows performance counters, agents usually offer their own application-specific counters for certain environments (such as when Terminal Servers are used).

More recently, companies have begun to build agents that use WMI to gather information for performance monitoring. WMI capabilities are built into Windows 2003, and this is definitely the direction in which all monitoring is moving. When WMI is used, these reporting tools often work without requiring agent software.

Data Storage

Naturally, you need a place to store all this data flowing in from agents. Most software packages support multiple database platforms for storing server performance information. In most cases, the agents store the collected information locally on the server being monitored. Then, at predefined times, they push that data to a central database. The best of these tools allow you to create a tree structure with monitored servers pushing to upstream databases. This data can then be pushed even farther upstream to create an entire monitoring organization.

Analysis and Monitoring Tools

The third-party tools come with a console or monitoring tool for viewing your servers' performance. Most of these tools are available in web-based and Win32 applications. Either way, the tools allow you to configure the agents, monitor the servers, and send alerts. Some packages also come with scaled down versions of the same tools to be deployed to helpdesk technicians doing basic server monitoring.

Reporting Tools

Finally, the third party tools can be used to create reports based on server or application performance. Often theses packages have the ability to integrate with other report-creation tools such as Crystal Reports.

Popular Third Party Monitoring Tools

There are several great tools available, too many to mention here. However, here's a rundown of some of the more popular ones that work well in Terminal Server environments. An up-to-date list is always available at www.brianmadden.com.

NetIQ AppManager Suite

AppManager is an inclusive package that some people think is becoming the industry standard for monitoring applications and application servers in Windows. AppManager has specific connectors for Terminal Server and add-in products like Citrix MetaFrame.

AppManager allows you to monitor specific applications and application metrics on the servers. When using Terminal Services and network load balancing, the tool has a module that monitors windows load balancing for changes or problems, and even checks event logs for load balancing events. More information about AppManager can be found at www.netiq.com.

Lakeside Software SysTrack

Lakeside Software's SysTrack is one of the original monitoring tools used in Terminal Services environments. It has specific components for Terminal Servers but can also be used to monitor all of the servers in your environment. It supports all popular databases mentioned and is flexible and easy to configure. Many people use this tool to record usage of Terminal Servers and to automatically create invoices for users or departments. Details about SysTrack are available at www.lakesidesoftware.com.

Microsoft Operations Manager

Microsoft's Operations Manager (MOM) can be used to monitor Terminal Servers. Like many other products, it offers Terminal Server-specific "Management Packs." MOM also has the ability to aggregate event log information from your Terminal Servers. Visit www.microsoft.com/mom for more information.

Citrix Resource Manager

If you're using Citrix MetaFrame XP on your Terminal Server, discussion of other monitoring tools is probably not necessary. Resource Manager comes with MetaFrame XPe and is specifically made for monitoring Terminal Servers and their applications. Reports can be generated or customized with Crystal Reports and real time alerts can send messages via e-mail or SNMP. For more information on Resource Manager, visit www.citrix.com or www.brianmadden.com.