4.2 Making Outlook a better network client for Exchange

---

Traditionally, Exchange has only supported MAPI RPCs over protocols such as TCP/IP over NetBIOS. This model assumes that clients connect to servers across perfect networks, or at least the consistent standard of service available in a corporate LAN. Many of us have experienced the frustration of watching Outlook report its chatty interchange with Exchange when we connect over slower or high-latency links such as an RAS dial-in. The RPC interchange is also fragile and prone to timeouts and failure. In fact, many things can interrupt the flow of data between client and server and lead to user frustration. While you might assume that RPCs are the root cause of the problem, in fact, Outlook has traditionally been a server-oriented application and as such generates a heavy communications load between client and server. Any interruption in communications due to temporary network glitches or other reasons caused Outlook problems. To address this issue and to accommodate new network devices such as the Tablet PC, the Outlook team focused on making Outlook 2003 far more tolerant to latency, reducing server communications, and not assuming that it is always possible to connect to a server. This design philosophy leads to features such as the local cache and on-the-wire compression. These factors, along with the constant need to remain competitive and add features in tandem with the rest of the Office suite, drive some of the improvements we see in Outlook 2003. Because Microsoft developed Exchange 2003 and Outlook 2003 in tandem, it is easy to understand why Outlook 2003 can take advantage of some unique features in Exchange 2003, but some of the features are backward compatible with previous versions of Exchange, so you can get some value through a client upgrade even if you want to remain with Exchange 2000. In passing, it is worth noting that Outlook 2003 only supports TCP/IP, so Microsoft has discarded some of its history in this release.

Figure 4.3 illustrates the Outlook 2003 interface.

Figure 4.3: Outlook 2003 interface.

4.2.1 Why RPC latency is important to Exchange

As with any other communications mechanism, RPCs are sensitive to network latency, something that can affect both Exchange clients and servers.

From a client perspective, problems with RPCs (most of which are due to excessive latency) become apparent when you see the pop-up window reporting that Outlook is waiting for Exchange to respond. Note that Outlook may communicate with several Exchange servers in a single session as users move between their mailboxes and public folders, and latency can affect any or all of these connections; so seeing pop-up windows appearing is not necessarily an indication that a network problem exists between the client and the mailbox server. The pop-up windows are only supported from Outlook 2002 onward, and they are welcome because you can decide to cancel a "hanging RPC" and terminate an operation, which is preferable to the previous situation where you had to kill the whole Outlook process if the client and server lock in a deadly embrace. Outlook is not always to blame here, since often Exchange simply fails to process the RPC and communicate back to the client in a timely manner.

You can use very low bandwidth networks to connect Outlook to Exchange (e.g., many people use 9.6-Kbps dial-up connections via cell phones) and transfer large items across the link, albeit slowly. All network protocols handle transfers over low-bandwidth links by splitting large items into multiple packets and then transmitting the packets in parallel. Unfortunately, applications may have to wait for the RPCs to complete and a response to arrive back, which is what you often see in Outlook to Exchange communications across high-latency links. You can now opt to have Outlook use the local cache to access information locally rather than fetch it from the server. While you work with local data, Outlook continues to fetch data from the server in the background to update the cache. The scheme works, but you sometimes see hiccups, so there is still some room for improvement.

Latency and slow connections can combine to generate interesting behavior for Outlook that can disconcert users. If you use a slow dial-up connection or perhaps a VPN link that disconnects and reconnects automatically, clients (prior to Outlook 2003) are sometimes slow in displaying new messages as they arrive in the Inbox. Users often notice this effect when they change folder view (by switching to another folder), only to notice that a number of messages suddenly appear in the Inbox, all seemingly delivered at different times. The underlying cause is that the connection between client and server is unreliable, and the synchronization necessary between Outlook and Exchange to process the RPCs sent by the server to the client to signal the arrival of new messages does not always work. It is uncommon to see this behavior over LAN connections when the link is reliable and neither latency nor bandwidth is a problem. However, it can happen if the server to client UDP notifications to trigger the update of the view fail.

Network gurus will cheerfully argue the factors that govern the point at which latency becomes a problem. Products such as Cloud (www.shunra.com) allow you to perform practical tests on a line to quickly measure where problems occur in your own environment. For example, you might set a limit of 300 ms for round-trip delays between Outlook and Exchange and test the line to determine whether you see pop-up windows during a variety of operations (create meetings, access public folders, send large messages, and so on). If problems occur, you can change the setting and test again. An approach such as this helps to quantify whether you need to upgrade your network and avoid user satisfaction issues, especially if you want to consolidate servers into central locations.

Note that it is important to distinguish between end-to-end latency and round-trip latency before conducting any tests, since otherwise you may confuse the issue. Obviously, a test result of 300 ms for round-trip response is equivalent to a 150-ms result for a trip between the server and client, but a quick glance at the headline figures may focus on one figure to the detriment of the other. In addition, any network analysis is very subjective and the measurements you take are highly sensitive to actual operating conditions, so the results attained in laboratory conditions may not be viable in the field. For that reason, it is wise to add some wiggle room to the figures to arrive at an acceptable standard.

4.2.2 Networking improvements in Outlook 2003

Among other improvements that it wanted to make in Outlook, Microsoft recognized the need to make the client smarter about the way it consumes network resources through its internal experience and customer feedback. The increasing demand for connectivity for mobile devices that connect using wireless networks, or even by using the latest cell phone networks, is another influence. In addition, devices such as the Tablet PC pose interesting challenges for applications that expect persistent high-quality connections to servers and get upset if they experience a network interruption. Previous versions of Outlook fall squarely into this category.

Microsoft had to do something to make Outlook a smarter network client, and the four major improvements in Outlook 2003 are:

• The introduction of Cached Exchange Mode in the form of locally cached mailboxes and the replication mechanism required to synchronize the local copy with the server mailbox: Outlook 2003 is able to fetch data from Exchange 5.5, Exchange 2000 and Exchange 2003 servers to maintain a local cache that the client then uses to access messages and attachments. Cached Exchange mode is now the default option for Outlook when you create new profiles for users. The intention here is that cached Exchange mode creates a condition where the user becomes unaware of the current network state and whether he or she has a connection to the server. Because they work with local data, performance is more consistent and acceptable than the previous model.

• Improved replication semantics, including header-mode downloads, compression, and "best body support": New code in Outlook 2003 optimizes the flow of RPCs exchanged with the server to reduce the amount of data sent over the wire during synchronization operations. The fact that Microsoft found opportunities to optimize Outlook's chatty interchanges with Exchange should come as no surprise to anyone who has monitored the flow of bytes across a dial-up connection as client and server patiently moved messages from one to the other. Better compression and smarter use of network resources mean that you see less dialog boxes popping up to report that Outlook is waiting for Exchange to respond.

• Outlook 2003 can use three types of replication behavior (download headers only, drizzle mode synchronization, always fetch full items) across any network connection: Some new code puts Outlook into "header only" mode when you use a slow connection, but you can override this option. Outlook detects changes in network state, including failover to new networks, transition between online and offline mode, and restore from standby and reconnect to the server. The client uses code different from that used across slow high-latency link to replicate data across fast LAN-type connections (note that high latency is not unique to slow connections).

• Using HTTP to transport RPCs: To do this, Outlook 2003 puts an HTTP or HTTPS wrapper around the MAPI RPCs sent to Exchange. You can connect Outlook 2003 to an Exchange 2003 server any time you are able to connect OWA to Exchange, with no requirement to remap RPC ports or establish a VPN. You must deploy Outlook 2003 on Windows XP SP1 (or above) on the desktop and connect to Exchange 2003 servers to enable secure authentication. In a front-end/back-end configuration, the front-end Exchange 2003 servers in the DMZ can proxy the RPCs to back-end Exchange 2003 servers. Because Microsoft made some changes to the network stack protocol to allow RPC requests to travel over HTTP, the Exchange 2003 servers must run on Windows 2003, as must the GCs and DCs used by Outlook. To make everything work, you also have to configure various ports (6001 to 6004) to allow proxies to work. While being able to connect Outlook back to Exchange over essentially any Internet link is a great feature, especially if you have to support a large community of traveling users, there are prerequisites and setup work that you must do before you can implement RPCs over HTTP. In some cases, it can be quicker, easier, and even cheaper for you to install a normal Virtual Private Network (VPN). Apart from anything else, a VPN allows users to access other resources on your internal network after they connect, whereas the RPC over HTTP feature only accommodates email.

If your PC runs Windows XP SP1^[1] or later, you can select to connect to the server over HTTP, specifying the URL to the RPC proxy server (such as http://server/). When Outlook connects to Exchange over HTTP, the standard UDP-based mechanism for new mail and calendar notifications changes to use polling instead. In a manner similar to OWA, Outlook polls the server regularly to detect whether it needs to notify the user of some new event rather than waiting for updates. The Outlook Custom Installation wizard and Custom Maintenance wizard also allow you to specify RPC over HTTP connections in client deployment kits. Figure 4.4 shows how you configure Outlook to use RPC over HTTP. Clicking the "Exchange Proxy Settings" button shows the screen on the right, where you enter the URL to connect to Exchange, the type of authentication to use, and whether Outlook should automatically revert to "normal" RPCs if it fails to connect over HTTP. You can force Outlook to use HTTP always and never revert by setting a new registry value, as shown in the following code segment. Set the value to 1 to disable use of "normal" RPCs.

Figure 4.4: Configuring Outlook 2003 to use RPC over HTTP.

Key: HKCU\Software\Microsoft\Office\11.0\Outlook\RPC Value: DisableRpcTcpFallback (DWORD) 

Outlook 2003 includes the ability to compress graphic attachments (Figure 4.5). This is particularly useful when you send screen shots or other images to illustrate a point. In the example shown, the original bitmap is over 2 MB. Before it sent the message to Exchange, Outlook compressed the file to an 89-KB attachment (using the medium compression option), which makes a huge difference in transmission and delivery times. Usually, Outlook achieves the highest compression ratios for bitmap (BMP) files, and the exact degree of compression varies across different graphical and other formats. This feature is client-side, so it works also with Exchange 5.5 and Exchange 2000 servers.

Figure 4.5: Outlook options to compress graphics.

"Best body support" means that Outlook can ask Exchange to provide native-mode content instead of forcing the server to convert message bodies into RTF. When an older Outlook client connects to Exchange, the server cannot be sure what format messages the client can support. For example, Outlook 97 and 98 do not support HTML format message bodies. If Exchange receives an HTML message, it could not take the chance that a client requesting synchronization supports HTML, so the server automatically converts the HTML into RTF and sends both copies down to the client. Every MAPI client and Exchange server support RTF, so it is effectively the lowest common denominator in any transfer operation. Sending two copies of every message body increases the amount of data that passes between server and client and makes OST files far larger than they need be. When Outlook 2003 connects to a mailbox on an Exchange 2003 server, it requests the server to send its best body part and the server can transmit just one copy without conversion. Not only does this change reduce network traffic, it also relieves a large processing load off the server, because messages do not have to be converted. Instead, if the client ever detects that it needs to convert the format of a message body, it can perform the conversion locally rather than going back to the server to request a new copy of the data.

It is difficult to quantify what benefit accrues to Exchange here, since everything depends on the percentage of HTML format messages received by the server. Outlook's default format has been HTML since Outlook 2002, so if you use recent clients you will have more HTML in the overall format mix. In addition, you will not realize full advantage of best body support until you upgrade every client to Outlook 2003.

Synchronization is the process of keeping one or more replicas in lock step with each other, normally based on a master copy. In Exchange, the server copy of a folder is always the master. The prospect of corrupt data always exists when computers want to send data to each other, and synchronization is no different. In previous versions of Outlook, when the client encounters a "bad item" in its replica of a mailbox or public folder, it places the item in the synchronization failures folder, logs an error in the synchronization log, and keeps on going. However, Exchange fails and halts synchronization if it hits a bad item when it attempts to synchronize down to the client. It is also possible to see loop conditions, where client and server merrily send data to each other with no effect except clocking up network traffic. Outlook 2003 incorporates a "bad item check" to validate that items are complete (formed with a body part and full set of properties) before it attempts to synchronize into the OST.

4.2.3 Cached Exchange mode

Previous versions of Outlook work in either online mode or offline mode. When online, the client connects to Exchange and you can work with any item in any mailbox folder, as well as public folders. All user actions generate server transactions and round trips between client and server. When offline, the client uses an offline store (OST), which you must first synchronize items to before Outlook can use it. All activity is local until you connect Outlook to Exchange to synchronize changes between server and local folders. Working offline is acceptable if you can do nothing else and cannot connect to Exchange, but it means that you have no access to the up-to-date GAL, cannot perform free and busy lookups for meetings, and do not have access to folders unless you marked them for synchronization and then performed synchronization. Organized road warriors prepare for trips by making sure that they download the OAB and synchronize their folders, but many people have found themselves off site with no connectivity and a set of unsynchronized folders. Before the advent of Outlook 2003, offline mode had some big traps to fall into if you did not prepare for every road trip by synchronizing all folders and downloading an up-to-date copy of the OAB. Things are a little easier now.

You can use the combination of Outlook 2003 and Exchange (5.5, 2000, or 2003) to implement "cached Exchange mode" for client/server communications, which is now the default mode for Outlook 2003 clients to connect to Exchange. Essentially, this is a combination of online and offline working, where clients continually fetch data from the server in a background thread while processing user actions against data held in a local cache. The default for new profiles is to use cached Exchange mode. If you already have a profile that you used with a previous version of Outlook, you have to set the properties of your Exchange server profile to use cached Exchange mode (a local copy of your mailbox), as shown in Figure 4.6. After selecting cached Exchange mode and logging out of Outlook, you log back on to Exchange in the normal way (to enable new mail notifications and so on), and begin using cached Exchange mode. Administrators can force existing users to move to cached Exchange mode through group policy settings, but this is not something to rush into until you understand the consequences for the server that must then handle the load generated by clients when they connect to build the cache.

Figure 4.6: Setting up cached Exchange mode for Outlook 2003.

After changing your profile to select cached Exchange mode, Outlook must build the local cache. If you already used an OST with a previous version of Outlook, it is likely that you did not synchronize every folder in your mailbox. Outlook therefore populates the content in those folders so that you end up with a complete local replica of your mailbox. If you did not use an OST, Outlook builds the local cache from scratch into a new OST file. In either case, building the local cache and making sure that it is completely up-to-date requires a substantial amount of processing and network activity, and you may find that your PC's performance is a little sluggish (largely due to disk I/O) until Outlook finishes downloading content from the server. This operation can take an hour or more to complete, depending on the configuration of your PC, other work that you are doing during the download, the speed of the network connection, the size of your mailbox, and the number of folders it holds.

In addition, the size of your OST file on your local hard disk inevitably grows bigger than your mailbox. OST storage is not as efficient as the Exchange database, so the usual result is a file that is approximately 20 percent to 30 percent larger. However, some conditions can force dramatic expansion. For example, my 686-MB mailbox exploded to become a 1.8-GB OST. Initially, I could not explain the vast increase in size unless it was due to the mixture of RTF, plain text, and HTML-format messages in my mailbox. Certainly, having a complete copy of all folders available for offline work gives users an excellent opportunity to clean out old material and reduce the size of the OST, but in reality, few will take this chance. Best body support, which is only available when you connect Outlook 2003 to an Exchange 2003 server, reduces OST bloat by only downloading a single copy of each message. Later, it became obvious that an early beta release of Outlook 2003 had synchronized full copies of every one of my public folder favorites, including some folders that held well over 10,000 items. After removing some old favorites, the file shrunk back to about 1.5 GB, and when I removed all public folder favorites, the OST reduced to 890 MB and settled down at this level, shrinking and increasing in line with the size of my mailbox. The shipping version of Outlook 2003 leaves public folder favorites alone, and you can control whether to synchronize the contents of public folder favorites into the cache through the advanced options for the mailbox. If you enable cached Exchange mode, you can expect to see OSTs that are 120 percent of the size of a mailbox, which is a reasonable overhead to pay for the facility of having a complete mailbox cached locally. You should try to keep an OST less than 1 GB, because past this point Outlook incurs a heavy I/O overhead to keep the OST synchronized.

4.2.4 Drizzle synchronization

By default, Outlook downloads full items into the local cache, just as it has always done when you synchronize items into an OST. Now, you can opt for Outlook to use two other download modes to better match current network conditions: header only and headers first followed by the full content of items. In the latter case, Outlook uses an algorithm called "drizzle synchronization." The idea behind drizzle-mode synchronization is to keep the local cache always up-to-date. Drizzle-mode replication is also more suited to the type of slow, high-latency roaming network links used by mobile devices (wireless and cell phones) and especially suits the "grab and go" nature of a wireless-enabled Tablet PC. The scenario here is that you can take a Tablet PC from its docking station that is equipped with a traditional Ethernet connection and immediately switch to a wireless link while Outlook stays connected to Exchange. With previous versions of Outlook, you need to shut down and restart Outlook to make a smooth switch between network connections, but Outlook 2003 takes network switches in its stride.

Windows XP plays a part here, too, because the operating system tells Outlook if the state of network connectivity changes, such as when you undock a tablet and move from a LAN connection to a wireless link. The situation is different with Windows 2000 Professional, which does not support a feature called Network Location Awareness (NLA) and does not update Outlook if a network change occurs. Instead, Outlook has to poll the network connection every 15 seconds to see if anything has changed. If Outlook detects a network change, it may decide to change its connection state to Exchange. For example, if a higher-speed connection is available, Outlook will use that. If no connections are available, Outlook changes to a disconnected state. Finally, if you use an RAS or VPN connection, Outlook gives that precedence over other connections. The logic here is that you probably established the RAS or VPN connection to reach an Exchange server outside the current LAN.

You can control how Outlook synchronizes your mailbox by using the "Cached Exchange Mode" option to instruct Outlook to download headers only (to restrict network activity), headers followed by content, or full items immediately. You can also ask Outlook to switch automatically to header-only mode if the client detects a slow network connection. However, if you connect Outlook 2003 to Exchange 2000 or 5.5 servers, the user interface forces you to download full items. This is because these servers do not support selective or header-only download.

My preference is to download headers followed by full items, because this means that I have a chance to review the contents of the Inbox immediately after Outlook fetches the headers and then take the decision to delete unwanted messages (such as spam). If you use a hard delete (press shift and delete together) to remove these items, Outlook will not move them through the Deleted Items folder and therefore will not download the full content when it updates that folder. Once I have reviewed new messages in the Inbox, I can begin working with messages and Outlook will download the full content behind the scenes. Without too much effort on my part, the mailbox is in a completely populated state any time I remove my PC from the network. Another small change in download behavior is that the most recent items appear first in the Inbox rather than the oldest items.

The advantage of cached Exchange mode is to deliver excellent client performance while retaining access to all of the server features. People will still want to work in traditional offline mode and configure suitable send/ receive synchronization groups to preserve network resources, especially across expensive dial-in links from hotel rooms or similar locations, but most users will find cached Exchange mode a benefit.

4.2.5 Download activity

On slow connections, you can opt for Outlook to download only headers. Outlook does not attempt to calculate throughput to decide whether a connection is fast or slow. Instead, it requests the network adapter speed from Windows and uses that as the base. You can assume that a modem connection is going to be slow and a LAN connection is fast, and the decision point is approximately 144 Kbps.

The top set of screen captures in Figure 4.7 illustrates how difficult it can be for software to predict the time necessary to perform any action. Outlook can ask Exchange how much data remains for the client to download and then estimate a time based on network speed and current throughput, but sometimes things just go wrong. Even the slowest connection can probably download 7.1 MB in less than four weeks! Fortunately, these screen captures come from beta versions of Outlook 2003 and the final shipping product is more accurate. The bottom part of Figure 4.7 shows the options available from the "Cached Exchange Mode" choice on the File menu, which allows you to determine how Outlook fetches information from the server. If you move to a folder that Outlook has not updated yet, it immediately begins to download the items into the folder so that you can work. Left to itself, Outlook will eventually download the headers for new items for all folders.

Figure 4.7: Outlook progress bar and Exchange connection types.

After you begin using a local cache, Exchange notifies the client to ask it to initiate synchronization any time new content arrives in your server mailbox. If you use an HTTP connection, Outlook polls for new items instead of waiting for notifications. The default interval to poll for new mail is 60 seconds, and Exchange provides this value to Outlook when the client connects. You can adjust this on the server by creating a new registry value at:

HKLM\System\CurrentControlSet\Services\MSExchangeIS\ ParametersSystem 

The new DWORD value is called Maximum Polling Frequency and its default value is 60,000 milliseconds, or 60 seconds.

Using the local cache introduces some additional synchronization overhead when Outlook starts, because it has to check every server folder in the mailbox to determine whether changes exist. Outlook also has to perform some housekeeping, such as registering for notifications on all folders, retrieving security settings, and so on. Outlook's initial connection to Exchange 2003 is faster than to previous versions of the server because Microsoft has optimized the housekeeping routines, but you should not see any evidence of extra processing except a small amount of network traffic.

After receiving a notification or detecting new items, Outlook downloads the message header (including the first 254 bytes of the message body) so that you know who sent the message, and the reading pane (previously called the preview pane) can function. The header also includes the data necessary to render views and to process rules together with an indication of whether any attachments are present. It does not download information about the number of attachments, their names, or sizes, but the message header does contain the overall message size, which is enough for a user to make a decision as to whether he or she wants to download the complete item. This data is stored in the local cache. If the user decides to read the message and view the attachments, Outlook fetches the information from the server, displays it, and stores it in the local cache. To keep everything synchronized, each time you connect to the server, Outlook checks each folder to make sure that its contents are up-to-date in the local cache and makes whatever adjustments are necessary.

Everything works without any obvious impact on mail delivery, and, apart from a brief pause while new messages are processed, users are not aware that Outlook is operating in a different manner. Indeed, you only realize that Outlook delays new message delivery slightly if you run two clients side by side and monitor delivery into the mailbox. Invariably, messages appear in the Inbox faster if Outlook connects to Exchange in a traditional noncached manner or if you use another client, such as OWA. This is expected, because the server notifies the client as soon as new mail arrives. When Outlook operates in cached Exchange mode, there is an interval between Exchange signaling Outlook that a new message has arrived in the Inbox (and seen by other clients) and Outlook downloading the message into the local cache and notifying the user with a pop-up display. At times, you may see a slightly extended delay in notifications, since Outlook attempts to bundle item synchronizations together if it receives notifications from Exchange that a number of new messages have arrived together.

If you decide to leave messages for later processing and Outlook connects to Exchange with a fast link, a background thread ultimately synchronizes the full message content, including any attachments into the local cache. Outlook uses up to five threads for synchronization: four dedicated to background synchronization and one reserved for immediate or foreground activity. This mechanism allows Exchange to distinguish between actions that it needs to respond to immediately and those that it can process with a little less speed. For example, if you spot a message from the list of downloaded message headers and want to read it immediately without waiting for Outlook to download full content, you can click on it to force the fifth thread to kick in and fetch the message. Exchange notes that the foreground thread is waiting for a response and puts the background synchronization threads on hold while it services the foreground thread and then resumes.

If a network interruption occurs during synchronization, Outlook can continue working with the local cache and wait for the network to come back before recommencing synchronization to update the local cache. Unlike previous versions of Outlook, where any network interruption severely disrupts client operations, Outlook 2003 is able to detect when the network is available again and reconnect to Exchange. This improvement is particularly noticeable across wireless connections, which often drop and reconnect automatically, or if you put a notebook computer into suspend mode and then restart it. When you work over a slow link, you can instruct Outlook to download headers or headers followed by full items. Downloading headers is often a good option, because it allows you to scan new items in the Inbox and delete some before you proceed to download email that you really need. You can still use the F9 command to perform folder synchronization as before, but this is now a feature used as an exception rather than a rule-for example, if you download headers for all folders and then decide to use F9 to invoke the default Send/Receive group to send and receive mail, or if you use Shift+F9 to synchronize a particular folder.

Once Outlook has downloaded messages to the local cache, the cache becomes its primary storage and Outlook never goes to the server unless absolutely required-for example, to fetch the content for a message whose header is in the cache.

Because Outlook focuses on data in the local cache, it can ignore temporary server outages or network interruptions. Clusters deal with hardware failures by transitioning resources such as storage groups from the node where the failure occurs to another node in the cluster. With previous versions of Outlook, the cluster state transition is noticeable, because the client "freezes" during this period, which can last from ten seconds to a couple of minutes or even longer. Outlook 2003 continues working during a cluster state transition and so can contribute to a higher service level for users, which may make the combination of Outlook 2003 and Exchange 2003 clusters more popular for system designers seeking the highest possible level of system availability.

The philosophy of using the local cache even extends to Outlook's address book. After all, if the client uses a local copy of the mailbox to avoid round trips to the server, should it not also use the local copy of the address book to avoid making LDAP queries to the GC? This is exactly what Outlook does when you work in cached Exchange mode after you have downloaded an OAB, as you can see by examining the GAL properties by opening the address book and then right-clicking on the "Show Names from the:" drop-down list (Figure 4.8).

Figure 4.8: Using the OAB as the preferred address book provider.

Of course, if you work offline and use the OAB to validate addresses, it is important that you update the OAB regularly by either including the OAB in a send/receive group or by downloading it explicitly on a weekly basis. To eliminate more reasons to consult the server, Outlook 2003 introduces a new unicode-format OAB that holds more information about recipients (such as multiple values for telephone numbers) than the previous OAB. You do not have to upgrade to the unicode-format OAB, since Outlook 2003 also supports the previous version. Whichever OAB you use, Outlook 2003 always attempts to fetch information from the OAB and only goes to the server if the user accesses data not held in the OAB. For example, if you look up a user in the GAL, Outlook first displays the General page. All of the data shown here comes from the OAB. If you now click on the Organization tab, Outlook has to go to the server to fetch organizational data (the person's manager and direct reports), because these links to other directory objects are not stored in the OAB. The user is not aware that Outlook switches between local and network data, because the client aggregates the results before presenting data to the user. Apart from the normal download process, there is no way to insert information on a selective basis about new GAL entries into the OAB or to remove entries for deleted users, groups, or contacts. If you want to access a new entry that does not exist in your copy of the OAB, you have to download an updated OAB or use the new object's SMTP address if you want to send a message to it. Fortunately, as explained later in section 4.6, Outlook only downloads the differences between the client OAB and the server version to minimize bandwidth consumption.

Outlook cannot possibly cache every piece of mailbox data, and server access is still required to work with some information-for example, if you want to browse free and busy information when you schedule meetings or when you use delegate access to another user's mailbox.

It is possible that the advent of cached Exchange mode will drive some change in user behavior around the use of folders. Up to now, you probably kept many messages in your Inbox and sent items folders simply because Outlook automatically synchronizes these folders for offline access. The result is often folders holding a huge collection of items relating to many different topics. The items are available offline, but it can be hard to find a specific item and you normally end up with a lot of junk stored alongside the valuable data. Now, if you have a complete copy of all server folders, you may be more inclined to refile messages out of the Inbox and send item folders to a more appropriate location. Of course, unless you help people understand how to use folders effectively by setting up a logical folder structure, they will not be able to take advantage of this capability. In addition, it takes hours to go through large folders to refile items, and it is hard to justify the return for such diligent filing.

4.2.6 Incremental synchronization

Outlook uses an algorithm called Incremental Change Synchronization (ICS) to set checkpoints to determine the data that it has replicated with Exchange during synchronization sessions. Up to Outlook 2003, Microsoft set large checkpoint windows with ICS, which was good for performance but meant that any interruption of connectivity between client and server forced Outlook to restart synchronization from a checkpoint that might be well back in the past. Outlook went back to the last full folder that the client had downloaded, something that could be a long way if you experienced a network interrupt when synchronizing a large Inbox. Microsoft has improved the algorithm (now called "advanced ICS") and now checkpoints after every successful network buffer download to ensure that Outlook has to recopy less data if an outage interrupts synchronization. The increases in performance elsewhere more than compensate for the small decrease caused by smaller checkpoint windows here.

4.2.7 Deploying cached Exchange mode

Microsoft argues that cached Exchange mode improves client performance, because Outlook works with local data while also reducing network traffic and server load by replacing the normal client/server traffic and server transactions with a constant trickle down to the client. At the same time, server performance improves, because client operations smooth out over time and do not encounter the same peaks in demand that you see with traditional client/server email systems. For example, the impact of several thousand people logging on at 9:00 A.M. each morning to read their email is significantly different if the clients download messages to the local cache to allow users to process the messages locally. You trade constant and unpredictable demands from clients as people go through their new messages (which might involve multiple downloads of specific messages and attachments) in favor of the one-time phased hit required to make single download and any processing required by the server to send new messages. If Microsoft's arguments are valid, it may be possible to support larger user communities on a particular server. However, consolidation to a smaller set of servers is impossible until you deploy Outlook 2003 clients and Exchange 2003 servers everywhere to implement cached Exchange mode and take advantage of automatic data compression, a feature only supported by the combination of Outlook 2003 and Exchange 2003.

There are instances where cached Exchange mode does not work well. For example, users who roam from PC to PC cannot really use cached Exchange mode, because Outlook would have to rebuild the local cache on each PC the user visits. Not only would this drain network bandwidth, it also increases server load. In addition, the user would not be very happy, because his or her mailbox would be in a state of constant refresh. It is best to advise roaming users to continue to connect to their mailboxes and work online. This is also inappropriate for someone who only uses a workstation infrequently, because every connection will provoke a massive download. Obviously, someone who uses Outlook through a Windows Terminal Services or Citrix Metaframe thin client connection cannot use cached Exchange mode either.

Another example is users with large mailboxes. There are people who have mailboxes over 2 GB-some over 5 GB-and it probably does not make sense to create a complete replica of such a large mailbox on a laptop, especially given the increase in the size of the OST, even with the size of disks in most modern laptops. If you want, you can now create an OST that uses unicode format and increase the OST to a virtually unlimited size. This is not a very good idea, because OST performance suffers due to excessive local disk traffic generated by Outlook to synchronize the local cache if you use a file larger than 1 GB. In particular, Outlook has to do a lot more work to insert new messages in the OST after it passes 1GB, so it is best to keep OST file sizes less than 1GB if possible, unless you store the OST on a high-performance disk. Most laptops are not equipped with disks in this category, but desktop systems may be.

There is no way to convert an existing OST to use the new unicode format. Instead, you have to delete your existing OST and then create a new MAPI profile. When you do this, the default option is to create a unicode-format OST (Figure 4.9). However, you then have to recreate the entire local cache into the new OST, so moving from old to new format takes time.

Figure 4.9: The OST is unicode.

Users who have very large mailboxes are often better off working with an OST in offline mode, because they can apply filters in their send and receive groups to restrict the amount of data that Outlook synchronizes with Exchange. However, you can advance the argument that the large disks now installed on most laptops make it easy to trade some disk space for the advantages accrued from a complete local cache, especially if your PC is fast enough to handle the increased overhead implied in keeping the local indexes in the cache updated and accurate. As in most situations, users will make up their own minds.

To check the impact of these changes, Outlook 2003 includes the ability to track and report RPC performance data. Outlook posts the data it captures periodically to the Exchange 2003 server, which collates the data reported by all of the connected clients. You can interrogate this data programmatically using WMI, or use the Performance Monitor utility to view it in real time. Table 4.2 lists the major counters that you can monitor, all of which belong to the MSExchangeIS (Information Store) object. Clients do not notice any difference in performance to collect data, and collation consumes some network resources when Outlook sends the data to Exchange.

Table 4.2: Performance Counters Updated by Outlook 2003

Performance Counter	Meaning
Connection count	Count of current client connections to the Store
Client: latency > 2 seconds RPCs	Total number of client RPCs with latency > 2 seconds
Client: latency > 5 seconds RPCs	Total number of client RPCs with latency > 5 seconds
Client: latency > 10 seconds RPCs	Total number of client RPCs with latency > 10 seconds
Client: RPCs attempted	Total number of client RPCs attempted since the Store initialized
Clients: RPCs attempted/second	Current rate of attempted RPCs per second against the Store
Client: RPCs failed	Total number of client RPCs that have failed since the Store initialized
Client: RPCs failed/second	Current failure rate per second of RPCs against the Store
Client: RPCs succeeded	Total number of client RPCs that have succeeded since the Store initialized
Client: RPCs succeeded/second	Current success rate per second of RPCs to the Store
Client: Total reported latency	Total count (in seconds) of latency of all client RPCs since the Store initialized
Client: RPCs failed: access denied	Total RPCs that have failed due to access denied
Client: RPC failed: call canceled	Total number of RPCs failed because client canceled the call
Client: RPC failed: server too busy	Total number of RPCs failed because the server was too busy to respond
Client: RPC failed: server unavailable	Total number of RPCs failed because the server did not respond
Client: RPC failed: all other errors	Total number of RPCs failed for some other reason

From a deployment perspective, you may have to be careful about how you introduce cached Exchange mode. Apart from testing third-party add-on products to ensure that they keep working when Outlook uses cached Exchange mode, you have to phase in the new client behavior to minimize any impact on system performance. For example, if you deploy Outlook 2003 to a large number of desktops overnight and create new profiles to force users into cached Exchange mode, you will see a colossal load suddenly generated on the servers the next morning as the clients connect and begin to populate their cache. Best practice suggests that you should limit the number of clients that turn on cached Exchange mode to less than 30 a day (per server). Of course, every company is different and the degree of load depends on several factors, including:

• The number of clients

• The average mailbox size-the larger the mailbox, the bigger the load required to build the local cache

• User filing behavior-if people refile messages from the Inbox to other folders and have not synchronized these folders to their OST in the past, then Outlook must fetch more data from the server to build the cache. On the other hand, if users keep most of their messages in the Inbox and Sent Items folders, then those folders already exist in the OST and less load occurs.

• The network connections that clients use to connect to the server- slow network links will extend the time that the additional load exists but may lessen the magnitude of the load.

• How quickly you deploy Outlook 2003-a phased rollout spreads the load

• The mix of Outlook and non-Outlook clients that you deploy

As with any other technology, it is wise to think about how it might influence your infrastructure before you rush to deploy.

4.2.8 Compression and buffers

MAPI clients request servers to provide data in chunks or buffers. To optimize communication between client and server, the size of the buffers varies depending on the client version and the type of network in use. For example, Outlook uses large 32-KB buffers on LAN-quality networks and reduces the buffer to 8 KB or even 4 KB as the connection slows. (See Table 4.3.)

Table 4.3: Outlook Network Activity Connected to Exchange 2000 or 2003

Version	Mode	Data Flow	Network	Client Buffer Size	Data Buffer Size	Size on Wire
Outlook 2002	Online	Download/Upload	LAN	32 KB	32 KB	32 KB
	Online	Download/Upload	WAN	4 KB or 8 KB	4 KB or 8 KB	4 KB or 8 KB
	Offline	Download/Upload	Any	32 KB	32 KB	32 KB
Outlook 2003	Online	Download	All	32 KB	32 KB	< 32 KB
	Online	Upload	All	32 KB	32 KB	< 32 KB
	Cached	Download	All	96 KB	> 96 KB	96 KB
	Cached	Upload	All	32 KB	32 KB	< 32 KB
	Offline	Download	All	32 KB	> 32 KB	32 KB
	Offline	Upload	All	32 KB	32 KB	<32 KB

Table 4.4: MAPI RPC Compression Parameters

Value	Description
RPC Compression Minimum Size (DWORD)	Defines the minimum size a packet is to compress (default is 1,024)
RPC Compression Enabled (DWORD)	Flag to allow or prevent compression. The default is 1, which enables compression. Set the value to 0 if you want to use a Network Monitor to track the actual data that passes between clients and server.
RPC Packing Enabled (DWORD)	Flag to enable or disable RPC packing. The default is 1, which enables packing.

In addition to the changes made for local caching, Outlook 2003 tweaks the buffer size and is now able to compress data. Exchange 2003 includes the same compression technology,^[2] so compression is full duplex rather than one way. However, compression only occurs when an Outlook 2003 client connects to Exchange 2003. The server only knows if a client can receive compressed data if it uses a special MAPI call to request data. Older clients do not use this call, so Exchange 2003 therefore reverts to noncom-pressed transfers.

Exchange incurs a performance penalty to perform compression and decompression, but Microsoft believes that the penalty is more than offset by the gain, because the server needs to process less network packets. You can influence how Exchange 2003 performs compression for RPCs to Outlook 2003 clients with the registry values described in Table 4.4, which you insert at the key:

HKLM\System\CurrentControlSet\Services\MSExchangeIS\ ParametersSystem\

Whenever possible, Exchange 2003 compresses complete messages, including attachments. In much the same way that ZIP utilities achieve different compression ratios for different file formats, the ratios produced by Exchange vary across different message types and attachments. For example, HTML and plain text body parts compress to between 60 percent and 80 percent of their original size. Word documents and Excel spreadsheets compress well, while PowerPoint presentations are less amenable to compression. Exchange already compresses RTF message bodies, gaining a further saving in bytes transmitted on the network.

Table 4.3 summarizes how Outlook 2002 and 2003 vary buffer sizes in different conditions. If you compare Outlook 2003 with Outlook 2002, you can see the wider range of buffer sizes in use. It is important to emphasize that the impact of this change on servers will vary with different user behavior and format mixes. For example, if you upgrade servers to Exchange 2003 and leave clients alone, you will not see any improvement. Upgrading each client to Outlook 2003 makes things a little better, but less so if everyone uses HTML as his or her message editor and sends large PowerPoint presentations to each other. You gain best results after everyone upgrades to the latest version of Outlook and sends a normal mix of HTML, plain text, and RTF messages with various attachments.

4.2.9 Conflict resolution

The nature of synchronization, where different versions of items can exist on client or server for good reasons, creates the potential for conflicts. A conflict occurs when some action modifies two or more replicas of the same item independently of each other, and Outlook then detects that different versions exist during the synchronization process. For example, you update an appointment using OWA and then update the same appointment with Outlook when you work offline, or you receive a reminder for a meeting on two devices (such as a Pocket PC and a desktop PC) and dismiss the reminder on both devices. When Outlook goes online again, it detects that the version of the calendar item on the server is different from the one in the client folder and a conflict occurs.

In previous versions of Outlook, the solution is to mark the item with the "crossed swords" icon to show the user that the conflict exists. Outlook keeps both versions and the user can decide which version is the definitive copy. Unfortunately, the process is manual and users may not be aware that any conflicts exist unless they attempt to open an item in conflict. In addition, applications such as ActiveSync do not handle items in conflict well, because they do not include code to resolve or otherwise deal with the problem.

Outlook 2003 includes a new conflict resolution engine designed to resolve conflicts automatically. The idea behind conflict resolution is to weed out spurious conflicts (caused when two versions of an item are identical) and improve the user experience when conflicts happen. Many conflicts occur because of predictable user behavior, such as dismissing reminders on two devices, and the conflict engine aims to resolve these conflicts without user intervention. Sometimes, as in the case of reminders, the conflict resolution engine resolves the problem by comparing the two versions, seeing that they are identical, and discarding one copy. In other situations, it is harder to determine why a conflict occurs, so the conflict resolution engine applies an algorithm to determine which version of the items in conflict "wins." In this context, when a version of an item "wins" in the conflict resolution process, it remains in its original folder. The "losers" end up in the "Conflicts" folder, a subfolder of "Sync Issues." You have to click on the "Folder List" shortcut to see the Sync Issues folder in the tree view.

The items in "Conflicts" (Figure 4.10) are alternate versions to items that exist in other folders such as the Inbox and Sent Items, and you can review them to decide whether you wish to keep a version of a specific item instead of the version selected by the conflict resolution engine. To do this, select the item and open it, then right-click on the conflict information band in the header. Outlook then displays the options you can take (Figure 4.11), including replacing the item in the other folder with this version or even replacing any line breaks that Outlook removed during formatting.

Figure 4.10: Reviewing an item in conflict.

Figure 4.11: Some items that Outlook had a problem with.

Outlook maintains two other folders under Sync Issues: Local Failures and Server Failures. Local Failures holds items that Outlook was unable to synchronize up to the server, while Server Failures holds items that Exchange was unable to push down to the client. Obviously, you should not see many items in these folders, and if you do, item corruption is the probable cause. In previous versions of Outlook, the Local Failures folder is "Synchronization Failures," so Outlook 2003 renames the folder to bring it in line with "Server Failures," introduced as part of Outlook's "skip bad item" feature that allows the server and client to complete synchronization even if they encounter corrupt data during the process. Ideally, you never see any items appear in these folders. If they do, then you need to understand why. One way to find out more information is to enable mail logging through Tools, Options, Other, Advanced (Figure 4.12), which instructs Outlook to generate synchronization logs in the Sync Issues folder for more operations. The information reported in the logs may help you understand the root cause of the conflicts, but it is more likely that you will pass this information on to PSS to help Microsoft debug the problem.

Figure 4.12: Enabling mail logging.

4.2.10 Is Outlook 2003 the best Exchange client?

In addition to its improved user interface, including features such as search folders, translucent pop-up notifications, and an improved reading pane, Microsoft has done a lot of work to make Outlook the best network client for Exchange. You can deploy Outlook 2003 only if you run Windows 2000 Professional (SP3 or later) or Windows XP, since these are the only operating systems supported by Office 2003. As shown in Table 4.5, you can exploit some of the features against Exchange 5.5 or Exchange 2000, but you can only realize the full benefits of the network improvements when you combine Outlook 2003 with Exchange 2003.

Table 4.5: Outlook 2003 Features Against Different Versions of Exchange

Feature	Exchange 5.5	Exchange 2000	Exchange 2003
Search folders	X	X	X
Cached Exchange mode (full items)	X	X	X
Cached Exchange mode (drizzle and header synchronization)			X
MAPI RPC compression			X
Buffer packing			X
Kerberos authentication			X
Best body support			X
Performance tracking			X
ICS check pointing			X
Smart change synchronization	X	X	X
Skip bad items		X	X
Presynchronization reporting and improved progress reports X			X
Integration with server-based antispam protection			X
RPC over HTTP			X
LIFO downloads-last in, first downloaded			X
Unicode OST and PST			X
Junk mail processing	X	X	X

All of these changes are very welcome, but many companies will take several years before they can leverage smarter network behavior. It is common to find two-or three-year desktop refresh cycles, so the need to deploy a new version of Office and possibly upgrade the desktop operating system is a more difficult hurdle to overcome than installing Windows 2003 and Exchange 2003 on their servers. However, if your company has many mobile users, it is well worthwhile to accelerate an upgrade program for the user who can benefit most-users of notebook computers who travel extensively.

^[1] . Even with SP1, you also need to install the hot fix described in Microsoft Knowledge Base article 331320. Later service packs likely include this fix. Outlook grays out the HTTP option if your PC is not running the right software versions.

^[2] . Microsoft calls the compression technology "XPRESS" and it is based on the Lempel-Ziv algorithm. Active Directory uses the same algorithm to compress data sent by RPCs between domain controllers.

---