Choosing Terminal Server Hardware | Terminal Services for Microsoft Windows Server 2003: Advanced Technical Design Guide (Advanced Technical Design Guide series)

The hardware that powers Microsoft Terminal Servers is usually different from that of traditional server environments. Since multiple users simultaneously access Terminal Servers, the hardware tends to be more robust

There is much to consider when picking the hardware that will run your environment. The first rule is to pick "real" server hardware. Desktop computers turned on their side do not constitute "real" hardware. Even though many of your users might have computers that are faster than your servers, they will not perform well. A 2.0GHz Walmart PC with 512MB of memory will not perform nearly as well as an HP Proliant 2.0GHz with 512MB server. Server class hardware and system architecture are what make a server a server.

This is especially true in Terminal Server environments. Terminal Servers are usually pushed to their limits due to the aggregate user processing taking place on them. Low-end PCs typically do not have the internal bus speed or internal bandwidth to support many users, even if they have fast processors and a ton of memory. For testing purposes, low-end PCs are adequate "just to see if it works," but you will not be able to extrapolate any performance numbers from low-end test PCs to real servers.

Let's look at it one more way. You can't keep from upgrading twenty-five workstations by giving them a hosted desktop on a single new workstation. However, you can do this if you use a single server. Use common sense and build your Terminal Server like a server. You're going to have a lot of users on this server, so it's not worth cutting corners to save a few dollars.

Now, let's explore server hardware in these environments by looking at some initial strategies for sizing your servers. Detailed performance tuning and optimization techniques are discussed in Chapter 13.

Terminal Server hardware sizing can be accomplished by adhering to one premise: "Lots of processing power, gobs of memory, nothing else matters." Admittedly oversimplified, but if this is the only thing you remember from this section then you'll be doing all right.

Memory

In the real world, memory usage in Terminal Server environments is extremely complex. Rather than explain the details of memory usage here, this section outlines the basic concept. Chapter 13 provides the (somewhat excruciating) details about how to accurately estimate and test the amount of memory in your Windows 2003 Terminal Servers.

Microsoft recommends 256MB of memory for the Standard and Enterprise editions of Windows Server 2003. However, your Terminal Servers will need much more. Every user that runs an application on a Terminal Server will use the memory for that application just as if they were running it on a normal workstation. For example, a quick check of the task manager shows that Microsoft Word requires about 10MB of memory to run. Each user of Word will need 10MB, meaning that 20 simultaneous users will theoretically require 200MB of memory. This is on top of the overhead required for each user to run a session, which is about 4MB. Twenty users running Word require a collective 280MB of memory on the Terminal Server.

Even a small Terminal Server designed for only 20 Microsoft Office users could easily require 512MB. Large servers that support 100 users typically have 2 to 4 GB of memory.

Processors

When it comes to picking processors for your Terminal Servers, don't waste the time calculating how many megahertz or gigahertz you need to support your users. Processor speeds are dictated by Intel and the hardware vendors, and you're pretty much forced to take whatever they offer—save for some choices regarding cache (more is better for Terminal Server). The real decision when sizing processors is the number of processors. In most cases you need to figure out whether your Terminal Servers will be single, dual, quad or eight processor boxes.

In deciding how many processors you want in your servers, keep in mind that in Terminal Services environments, the multi-user kernel is based on a cooperative multitasking algorithm. Each user's session sends requests to the processor only as needed. If one user does not need much processor time, then more processing power is available to others. Terminal Services does not work the same as old timesharing servers with which each user was allocated his particular time whether he used it or not.

Due to cooperative multitasking and the fact that each user's session operates several threads, Terminal Servers tend to scale very well with multiple processors—even when more than two are used. (That whole "Windows doesn't use more than two processors" argument is old school and completely irrelevant in Windows 2003 Terminal Server environments.)

The downside to scaling above dual or quad systems is that you run a greater risk of creating a bottleneck in other areas. You might be able to put more physical memory and processors in the server but then you may have a disk or network bottleneck. (Chapter 13 provides a step-by-step approach to identifying and rectifying any bottlenecks you might find.)

In a perfect world, your processor utilization would constantly be 99%, indicating that you didn't waste money buying too many processors, but also that no users have to wait for processing bottlenecks.

Processors are very fast and very cheap. When sizing servers, many people buy single processor servers that are dual processor capable, allowing them to test with a single processor and then add a second if the single processor does not give the results they want.

Hard Drives

In most environments, your Terminal Servers need to only contain the operating system, the page file, your software application files, and enough free space to temporarily store user profiles and temp files. Permanent user and application data is usually stored on non-Terminal Servers or a storage area network (SAN).

Ideally, all your servers will be identical so that users can successfully load balance across them and you can easily replace individual servers without affecting the entire system.

Because of the way that hard drives are typically used in Terminal Server environments, you don't need very much storage space on individual servers. Most people buy two drives and mirror them for redundancy. It's hard to find drives smaller than 18GB anymore, and 18GB should be more than enough storage space for each server, especially if you follow the guidelines outlined in Chapter 6 to prevent users' roaming profiles from being permanently cached on your servers, and you prevent users from saving data on the servers.

Since Terminal Servers don't store much data locally, you should be able to build your servers with two drives configured to mirror each other for redundancy. Today, most Terminal Servers are the thin, 1U servers or blades with only two drives anyway. If your company standard calls for RAID 5 configurations, don't go overboard for your Terminal Servers. Buying five drives for a mirrored OS and a three-drive RAID array for data is useless if you're not storing data on the system.

Server Hardware Redundancy

Most name-brand servers now have options for redundant and hot-swappable components, including power supplies, fans, memory, network cards, and PCI cards. Many people think these types of redundant devices are needed on Terminal Servers since the Terminal Servers are so important to the business. While this is true, these devices are not always needed in Terminal Server environments.

Quite often, redundancy in Terminal Server environments is built in at the server level. Instead of spending extra money on fancy servers, many administrators spend money on an extra server that can be added into the load-balance cluster to support users if a server is lost. This has the additional benefit of increasing performance on a day-to-day basis since you have more servers than are required and will allow you to "pull" a server from the cluster for maintenance while still being able to host all of the users.

The only problem with this approach is that if a server fails, users' sessions will go down. Even if you build a well-designed load-balanced environment, all you can do is allow your users to seamlessly connect to another server. If you absolutely must do everything possible to prevent a user's session from going down, then you'll need invest in some expensive hardware redundancy for individual servers.

Server Capacity Testing

Once you finalize your strategy (or perhaps in order to help you solidify your strategy), you'll need to run performance tests on your servers to determine the number of users they can support. Though there are dozens of Internet sites and research studies that provide server-sizing benchmarks, it's impossible to get exact results given the variations between environments, including network speed, protocols, Windows profiles, and hardware and software revisions.

One mistake that many people make after performing benchmark tests is to assume that the system will scale linearly based on high-level information available in task manager. Suppose that a server with no users has 4% processor utilization and 150 MB memory utilization. With five active users, the processor utilization rises to 14% and the memory to 200 MB. A quick estimate reveals that the processor would max out at 48 users, with 630 MB memory utilization.

However, performance tests might indicate that 25 users can connect, but any more cause the system to run extremely slow, though plenty of memory and processing power is available. In these situations, the server bottleneck is not visible on the surface with task manager. The disks may be overused or the server's system bus may be full.

It is imperative that a detailed analysis be performed to determine the true system utilization. It should be no surprise that Microsoft recommends purchasing the fastest server possible. In most situations, however, this is not feasible, so you will have to test your server hardware to determine your maximum load.

There are several tools and techniques that you can use to determine the capacity and performance of your Terminal Servers. You'll need to simulate user load on your servers and record the performance of the system. From there, you'll be able to determine the system's bottlenecks and capacity limits.

It should be noted here that performance should not only be looked at from a server performance metric standpoint (like processor and memory utilization), but also from an end user response time perspective. User sessions may slow down before any server metrics seem maxed out. Some load testing tools will do this while watching server metrics and can inform you of a performance problem on either side of the test. If you are not using one of these products you should at the very least keep a connection open to the server that you can use to test responsiveness within the session.

Regardless of the exact tool or technique that you use, all capacity planning and testing follows the same basic methodology:

Choose the application or applications that you would like to use for the load testing.
Determine what tasks a user will do within that application.
Determine what performance speed or response time is required (how fast the user should be able to complete a task).
Determine how users use the application.
Create a script or automated process that can simulate a user using the application.
Prepare to monitor the server's performance during the test.
Perform the test by executing the scripts.
Analyze the results.
Ask your boss for more money to buy a bigger server.

Results from this method of server testing should tell you two things:

How many users the server can support.
How the server performs when it is highly loaded. (For example, does performance lag for current sessions, or does it stop accepting new sessions?)

Let's detail each of the testing steps.

Step 1. Choose your Test Application

When testing the performance of a server, the first thing to do is identify an application or applications to test. Ideally, you'll be able to test the applications that are most important to your business.

Step 2. Determine Test Tasks

Once you've determined an application that you want to use for your testing, you need to think about what users will be doing with that application. Is it a line-of-business application where users will be entering data into forms and running reports on that data, or is it a spreadsheet application where users will be performing calculations? Maybe it's a word processing application where users write documents?

Step 3. Determine Appropriate Response Times

Identifying the appropriate application response time will allow you to determine whether a server is too busy. If your test application is Microsoft Word, you might require that a letter appear on the screen within 0.2 seconds of the user pressing the key. This would be your threshold for acceptable performance. Later on in your testing process, you may find that your server can support 130 simultaneous users before crashing, although each user has to wait 0.5 seconds for the key response delay. In this case, you may find that you can only support 80 users with the 0.2 second response time.

As another example, your users might need to pull up reports in a line of business application. You need to determine what the appropriate wait time is for them. If you decide that a user should not have to wait more than 15 seconds for a report, then it is unacceptable to put 60 users on a server if they must each wait 20 seconds for their reports.

Step 4. Determine how Users Use the Application

Once the appropriate responsiveness of your applications is identified, you need to determine how active your users are. For example, some users enter data into a screen, then rummage around through papers at their desk, then enter more data. For these users, you might discover that they can enter the data in 10 seconds, but that they only do this once per minute. On the other hand, more active users might perform the same 10-second transaction six times per minute.

Knowing your mix of users is important, because a server that can support 75 "slow" users might only be able to support 40 active users. When testing you should try to simulate the activity of the user population as closely as possible.

Step 5. Create the Application Simulation Script

Now that you've thought about the application that you would like to test and the way that users will use the application, you can begin thinking about the testing process itself. The main technique you'll use in your capacity planning is to have multiple users access your application at the same time. By watching the performance of the system during this time, you can determine how the system will scale.

Instead of cornering a bunch of users and asking them to "use the system" while you observe the performance, most people create user simulation scripts. These scripts simulate users using the system. The nice thing about creating a script is that you (as one single person) can test hundreds of users accessing the system at the same time. The other advantage of using a script instead of test users is that you can get consistent, repeatable loads and thus you'll know if any changes you make to the server actually affect performance.

An application simulation script is essentially a batch file that automates the process of a user launching and using an application. There are dozens of tools on the market that can be used to script your user sessions. The most popular are detailed in Figure 5.4.

Product	URL	Cost
AutoIt	www.hiddensoft.com/AutoIt	Free
WinBatch	www.winbatch.com	US $100
WinTask	www.wintask.com	US $100

Figure 5.4: Popular Windows application usage scripting tools

These tools offer a "recording" mode that allows you to perform some functions (such as typing a document, browsing the web, using PeopleSoft, etc). Once a script is recorded, you can play it back to simulate a user using the system.

When your application simulation script is complete, you should end up with a file or files that you can launch from the command line (such as, "autoit.exe/word" or "myappscript.cmd"). The script should launch the application and then begin "playing back" the simulated user interaction.

Step 6. Prepare to Monitor the Performance

Before you begin testing, you need to configure your system to record the performance of your server during the testing. It's important that your testing data is logged and saved. While you're conducting the test, you will be focused on creating a good simulation. You don't want to worry about trying to view the results of the test as you're conducting it. It's much better to record the results and view them in detail at a later time.

There are two ways to measure performance in Terminal Server environments:

Use a third-party performance monitoring tool.
Use the Windows Performance MMC snap-in.

Most third-party tools utilize the same counters that are found within the Windows Performance MMC. These tools usually offer the advantage of logging results to a SQL database and offering more enhanced analysis capabilities.

Another advantage to using third-party tools is that some of them can track Terminal Server session response time. However, for many situations, Windows' built-in Performance tools will do just fine.

Think back to your Windows training. Do you remember how to use the Windows Performance tool to record performance counters to a log file so that you can view them later?

On the Windows 2003 Terminal Server that you would like to test, launch the Performance MMC (Start | Programs | Administrative Tools | Performance).
Expand the tree under "Performance Logs and Alerts" in the left pane. Right-click on "Counter Logs" and choose "New Log Settings…."
Type a name for your new log file. This should be a "friendly" name, such as "50 user test with MS Word."
Click the "Add…" button on the screen that pops up. This is where you choose the specific counters to record. For Windows 2003 Terminal Servers, you should monitor the following performance counters
- Process | Working Set | _Total
- Memory | Pages Input/Sec
- Memory | Pages Output/Sec
- Memory | Available Bytes
- Terminal Services | Active Sessions
- Paging File | % Usage | _Total
- Processor | % Processor Time | _Total
- System | Processor Queue Length
- Physical Disk | % Disk Time
- Physical Disk | Current Disk Queue Length
- Network Interface | Bytes Total/sec | Select your card
- Network Interface | Output Queue Length | Select you card
- Memory | Free System Page Table Entries
- Memory | Paged Pool Bytes
- Cache | Copy Read Hits %
If Terminal Server sizing and optimization is new to you, be sure to also read Chapter 13, which provides an in-depth description of how to read each of these counters and how they're relevant to Terminal Server sizing.
Highlight each counter and instance that you would like to record and click the "Add" button to add them to the log file. After you've selected all the counters you'd like, click the "Close" button to go back to the log file settings screen.
By default, the system is configured to record a sample of the data every 15 seconds. Depending on your test size and hard drive space, you might want to increase the frequency to every 5 seconds or so.
If you would like to change the path of the log file, you may do so by clicking the "Log Files" tab. (Note: if you wish to be able to import this data to Excel for creating charts, you can change the data type of the log file to a Comma Delimited format.)
You can also click on the "Schedule" tab to configure your log file to automatically start and stop at specific times, although most people don't do this when they're running specific tests.
Once your counter log is fully configured, it will appear in list in the Performance MMC. When it's time for your testing to begin, you can start the log by right-clicking it and selecting "Start" from the context menu. The icon next to the log will change from red to green.

Step 7. Conduct the Test

Now that you have your application simulation scripts created and are ready to monitor the performance of the server during the test, you can begin the actual testing process.

There are several ways to launch the test scripts. Most people simply launch several RDP sessions and then manually kick-off the test script from the command line. You could also automate this process by placing a shortcut to the test script in the Startup folder so that the script runs as soon as the RDP session is launched.

Microsoft provides some Terminal Server capacity planning tools in the Windows Server 2003 Deployment Kit and Windows Server 2003 Resource Kit. These tools, called Roboclient and Roboserver, help you automate the testing process. While these tools can be helpful, they have limitations. The Roboclient software only allows five concurrent RDP sessions from a single client device, even though a decently-sized client can easily support 10–15 sessions.

Regardless of the exact testing methodology that you use, you must also attend to the following:

Add your test users in groups. Never attempt to launch 20 connections at a time. Instead, try adding groups of 5 or 10 users spaced four or five minutes apart.
Start one session from a client device that does not run the automated test script. This session will allow you to understand how the load affects "real" user sessions throughout the testing process.
Keep in mind that the script interpreter (or whatever mechanism you're using to run your test scripts) will require some resources. This could potentially need to be subtracted from the actual performance numbers.
Once your server begins to near its full load, you should start adding users one at a time instead of five at a time.

Don't forget to stop your performance monitor recording log once your testing is done. You can then examine it to determine the results of your test.

Step 8. Analyze the Results

Once you've stopped your performance monitor log, view the results within the Performance MMC console. With the "System Monitor" object highlighted in the left-hand pane, click the button with the picture of the cylinder on it and browse to your log file, configuring the graph so that it pulls its data from the log file instead of from the current system activity.

Even after you configure the graph to get its data from the log file, you'll notice that the graph is still blank. You must manually add the performance counters that you want to view. Use the "+" button on the toolbar, just as you would with live data. The only difference is that when you're displaying data from a log file, the only performance counters listed will be the ones you recorded in the log file.

Chapter 13 details what you need to know to successfully analyze the results of your performance test.

As you analyze your results, keep in mind that you're looking for the bottlenecks in your system. Every system will max out at some point. If your system seems to be running out of memory then you can probably add more. Once you do that and run the tests again, you might find out that you can support 10 additional users but then your processors start to max out.

Another fact to keep in mind is the amount of resources that are used by the testing software. Some software packages can add 2–5 MB of memory per session that is tested. Also, capturing performance logs takes up system resources.

Based on the data from the log file of your test, you can probably figure out the point at which your server performance starts to fall drastically. Looking at the Active Sessions counter will tell you how many sessions there were when that performance drop occurred. Once you determine how many users your system can hold, you may want to run your tests again. For this second round of tests, ask live users to log on in addition to your test users. The live users will be able to tell you whether the system is usable or not.

You can import your performance data into Excel if you saved it as a comma delimited file. Browse to the log file and open it directly with Excel. You will notice the counter name is across the top row and the counters from the test form the columns. These columns can be highlighted and a chart can be inserted into the worksheet. When creating these charts, you will generally include a standard performance counter and the number of active sessions, allowing you to compare how a resource performed as the user load increased. Charts are an easy way to create detailed reports for management justifying why you need more hardware.

Server Sizing Tools

If your Terminal environment is important to your business, or if you work for a consulting company, there are some third-party server sizing and stress test tools that are easy to use and produce accurate results. (Of course, the downside is that they are expensive.)

The two most popular tools are StressTest by Scapa Technologies (www.scapatech.com) and LoadRunner by Mercury Interactive (www.mercuryinteractive.com). These tools are similar. In addition to being easy to use, they have the ability to test the performance of a Terminal Server from end-to-end, instead of only testing the impact of multiple user sessions and application execution as with Microsoft's RoboClient and Citrix's Citrix Server Test Kit.

The Scapa and Mercury tools can test the aggregate affects of server load, network bandwidth, compression, encryption, and virtual channel use. Also, because they work by using dedicated testing workstations to monitor session performance, the testing itself doesn't impact the results (unlike the Performance MMC which itself consumes server resources).

The process by which these third-party tools are used is the same as described earlier in the chapter. You still have to write application simulation scripts. However, the third party tools make it easier to run the tests and interpret the data.

Both the Scapa and Mercury tools are widely used, and it's up to you to determine which is more appropriate for your environment. Mercury Interactive has always positioned themselves at the top tier of testing products, and their pricing for their Terminal Server tools reflects that. (Their product is more than twice the cost of Scapa's product.) However, if you work for a consulting company, Scapa offers a "consulting" license that allows you to take their testing tools from customer to customer.

Advantages of Third-Party Testing Tools

They test the performance of the entire system, not just the server.
They employ outside "control" testing stations, so the act of testing does not skew the testing results.
They provide the most accurate, end-to-end Terminal Server sizing and stress testing.

Disadvantages of Third-Party Testing Tools

They are expensive.