11.9 The message tracking center

---

Exchange has always incorporated a facility called the message tracking center to track the progress of a message as it makes its way between servers in an organization. The feature depends on the availability of message tracking logs for interrogation by a requesting server that wishes to track a message.

The default properties set on Exchange servers allow message tracking. This means that a server maintains a set of message tracking logs, creating a new log daily, which is named according to the date-for example, 20030125.log (the log for January 25, 2003). Exchange stores the logs in a network share called server_name.log,^[3] which is usually located on the same drive as the Exchange binaries. Figure 11.44 illustrates a typical set of message tracking logs. Note the difference in the size of data generated on typical workdays (such as 20030711 or July 11, a Friday) and the weekend (July 12 and 13). Clearly, Monday and Tuesday (July 7 and 8) were easy work- days or the users of this server did not generate much email then, perhaps because some were on vacation. Despite being the current day, the log for July 15 is largest, which indicates some heavy traffic going through the server for some reason. Logs generated by an Exchange 2003 server are slightly larger than Exchange 2000, because Exchange 2003 captures additional data as messages progress through the postcategorization, prerouting, and routing stages within the transport engine.

Figure 11.44: Message tracking logs.

Exchange 2003 uses the same network share to store its tracking logs but makes a subtle change to the permissions on the share. On an Exchange 2000 server, the share is open and any user can map it to access tracking log data. You can argue that few users are interested in looking at tracking log data, but it is conceivable that someone could browse the data to look for message traffic for a specific user-for example, to determine to whom a senior manager sends email or who he or she receives email from. Exchange 2003 therefore restricts access to the share to administrators. Remote Exchange servers can continue to use the data, because they access the share through the local Exchange management service. Figure 11.45 shows the ACL on the tracking log share on an Exchange 2003 server (left) and Exchange 2000 server (right).

Figure 11.45: Security on Exchange 2000 and Exchange 2003 shares.

11.9.1 Tracking messages

Exchange generates a unique unifier for every message when a client submits the message to the server, and records the identifiers in the message tracking logs, thus making it possible to trace a message as it makes its way to its final destination. An example message identifier generated by a MAPI client is:

[e-mail] 

The only part of this identifier that makes any sense to people is the server name ([e-mail] in this case) at the end of the string. Everything else makes the computers happy when they track messages.

Note that this identifier is not the same as the MAPI message identifier that you can see by viewing the properties of a message. POP3 and IMAP4 clients also generate their own message identifiers in the form:

000001c19dbf$78c6bf90$[e-mail]

Aside from formatting, the major difference between MAPI and the message identifiers generated by non-MAPI clients is the omission of the originating server name. Client-specific message identifiers can be found in the Linked-MSGID field in the message tracking log. The messaging tracking center also displays the client-specific message identifier when you track a message and view its history.

You enable message tracking through three server properties (Figure 11.46). These properties can be set manually or across a set of servers (which may span multiple administrative groups) through a system policy (Figure 11.47). The properties are:

• Enable messaging tracking: This property instructs Exchange to begin creating and populating the message tracking logs. The transport engine logs every message that it processes and creates a new log every day. System messages, such as those that transport details of public folder hierarchies between servers are not logged.

• Enable subject logging and display: This property controls whether Exchange records information about message subjects in the message tracking logs and displays it when you track a message. Message subjects can contain confidential information, which is the reason why some installations would opt not to collect this data. However, users should not really be putting confidential data into message subjects (they can secure confidential information through encryption, if it really is confidential), and the subject field is a good way to isolate the message you want to track if the user who has sent it generates a lot of traffic. In most cases, there is no harm in recording this data.

• Remove log files after a specific number of days: The default value is seven days, and the Exchange System Attendant process cleans up old logs daily and logs this activity with event 5008 in the application event log. With the amount of disk space installed on most servers today, the space occupied by message tracking logs is not usually a problem and you can opt to keep the logs for a longer period if you like. However, if a user has not asked you to track a message that he or she suspects has not been delivered to the intended recipient within a week of sending it, the message is probably not very important and you can probably ignore the request.

Figure 11.46: Server properties.

Figure 11.47: A system policy for message tracking.

Obviously, you can only track messages if servers record details in the tracking logs. Within an organization, it does not make sense for some servers to implement logging and others not to, so it is a good idea to establish an enterprise-wide setting and apply it with a system policy. You should either enable logging on all servers, or not at all. Best practice is to enable logging, because it provides a useful facility at very little cost.

11.9.2 Changing tracking log location

Exchange creates its message tracking logs in a network share called \\server_name\server_name.log-for example, \\QEMEA-ES1\QEMEAES1.log. The default permissions for the network share allow anyone to read the log data, while administrators and the local computer account have full control. Some commentators advise that you consider removing the "Everyone" permission from the share to increase security and prevent people who are not administrators from attempting to access the data. However, unless you enable subject logging, the message tracking logs usually do not contain anything that is of much interest to people who might be tempted to poke around a system, so this may be a step to increase security a notch too far.

No user interface is available to relocate the logs in Exchange 2000, so you must use ADSIEDIT to update the server properties to move the logs to a new directory and then update the system registry to relocate the file share. To update the server properties:

• Take all Exchange services offline by stopping the Microsoft Exchange System Attendant service.

• Create a new directory for the transaction logs (e.g., "T:\exchsrvr\"). This will be the root of the new tracking log directory.

• Copy the current tracking log directory "D:\Exchsrvr\ServerName.log" to "T:\Exchsrvr\" (e.g., T:\Exchsrvr\SERVER1.log).

• Open the configuration naming context in ADSIEDIT and navigate to "Configuration, Services, Microsoft Exchange, Your Exchange Organization, Administrative Groups, your Admin Group, Servers, your Exchange Server."

• Select the Exchange server and click Properties.

• In the "Select a Property to View" box, choose "msExchDataPath," as shown in Figure 11.48.

  
  Figure 11.48: Changing the tracking log directory.

• Input the new directory into the "Edit Attribute" field.

• Click "Set," then "OK."

• Open the Registry Editor and update the following key to match the new tracking log folder (e.g,, "T:\Exchsrvr\Server1.log").

  Hkey_Local_Machine\System\CurrentControlSet\Services\ MSExchangeSA\Parameters\Log Directory.

• Restart the Exchange services and send a message. Verify that Exchange is writing details into the new tracking log as users send messages and that you can access the directory through the file share.

Fortunately, the situation is easier in Exchange 2003 and you can change the location of the tracking log directory through General Server properties, as seen in Figure 11.46.

11.9.3 Changes in Exchange 2003

The fundamental nature of message tracking does not change much in Exchange 2003. However, there are a few subtle details to understand. Exchange now records more information as a message makes its way through the Routing Engine. For example, notifications as messages pass through the postcategorization, prerouting, and routed stages. The net effect is that you see more information as you track a message. Tracking works between Exchange 2003 and Exchange 2000 servers, but, obviously, you see less information reported when a message's path crosses an Exchange 2000 server because some data is unavailable.

11.9.4 Starting to track messages

You track messages through the Message Tracking Center component of ESM. This is also available as a snap-in, which you can load into a customized console, but usually you access the Message Tracking Center via the "Tools" node in ESM. When you click on the Message Tracking Center, you can enter the criteria necessary to begin the search.

Clearly, the more information you have (who sent it, when it was sent, what was its subject) about a message, the better. Starting to search with some fuzzy details about a message that a user thought was sent four days ago to a group of people is much harder than looking for a message sent yesterday. Figure 11.49 shows the result of a search that has located two messages that I sent to a specific recipient. Note that in this instance, the server properties determine that message subjects are not included in the message tracking logs, so you will not see this information.

Figure 11.49: Results of a message tracking request.

The Browse buttons for the Sender: and Recipients: fields allow you to search the AD for users, contacts, or groups that sent or received the message. The Message Tracking Center will not allow you to begin a search until it verifies that you have provided valid sender and recipient data, using the logic that there is no point searching for anything based on invalid information. By default, you begin searching on the server that you connect to with ESM or the Message Tracking snap-in, but you can connect to another server to start searching if you need to. This is required if the person whose message you are tracking has his or her mailbox on another server. Even on large servers that handle heavy volumes of email, searching proceeds quite rapidly and you should have a result within a couple of minutes. Situations do exist when the Message Tracking Center appears to be unresponsive when it contacts other servers to retrieve data, especially if you search for a message addressed to a large distribution, but it will respond if left for processing to complete.

Apart from introducing a better user interface for the Message Tracking Center that resembles the "Find Message" functionality in Outlook, Exchange 2000 SP2 made an important architectural change to search processing. Prior to SP2, the server performing the search contacted each server in a message's path and retrieved all of the data from the remote servers before processing the search. From Exchange 2000 SP2 onward, the local server sends a search request to the Exchange Management Service running on each server involved in a message's progress. The Management Service processes the search request locally and then returns whatever relevant data it finds to the requesting server.

The number of messages found depends on the quality of the search criteria. In our example, we find only two messages, so it is likely that these include the message that we want. However, if you use some inexact criteria, it is possible that the MessageTracking Center will find ten or more messages, and it then becomes harder for you to decide which one is the message you really want. This is one reason why you should enable the capture of message subjects whenever possible; otherwise, it can be very difficult to isolate the exact message that you are interested in. You can view comprehensive details of the progress of a message through the Routing Engine by double-clicking on one of the found messages.

Figure 11.50 illustrates how Exchange tracks the path of a message as it interprets the tracking logs on each of the servers the message passes through. Note that we still cannot see the message subject, because server properties suppress this information. The tracking results show that after a client sends the message (the Store Driver reports that the Store submitted the message), the Advanced Queuing Engine and the Categorizer then process the message to determine on which queue to put the message. The result is a decision that the message is going to a remote SMTP recipient, so the Routing Engine transfers the message to a server that hosts an SMTP connector in order to use that connector to send the message to its final destination. In situations where messages move to other routing groups, the Message Tracking Center lists the destination bridgehead servers in the routing group in a tree view in the "Location" (left-hand) panel. As tracking proceeds, the bridgehead servers expand to show all of the servers within the routing group that receive a copy of the message, as shown in Figure 11.51. You can click on any server to follow the progress of the message on that server (such as the delivery to local mailboxes or onward transmission via a connector managed by the server). However, you will not see any data if an administrator has disabled or otherwise blocked access to tracking logs on a server or your server has not completed the search process.

Figure 11.50: Results of tracking a message.

Figure 11.51: Tracking through multiple servers.

You may see that the Routing Engine transfers messages to the MTA for further processing. This implies that these messages are destined for an X.400 recipient, a mailbox on an Exchange 5.5 server, or for transmission via a legacy connector (such as IBM PROFS or Lotus Notes) for which the MTA has a responsibility to service. Messages to recipients in other routing groups go via SMTP to a bridgehead server in the same routing group for onward transfer or directly to another server in another routing group if your server hosts the routing group connector.

The Message Tracking Center may also list deliveries to mail-enabled public folders. Organizations often use public folders as archives for messages sent to distribution groups and, in terms of delivery, Exchange processes these messages in exactly the same way as those sent to a local mailbox. The only evidence that the recipient is a public folder is its name, and you would have to know that this is the name of a public folder rather than a real person.

Tracking continues as long as the Message Tracking Center can follow the progress of a message. All of this functionality depends on administrators enabling message tracking on servers, preferably in a consistent manner across all servers in the organization. Creating a server policy to enable message tracking and then applying the policy to all servers in the organization, with or without the capture of the message subject, is the best way to achieve this goal. Obviously, if you do not enable message tracking, then Exchange will not create tracking logs and you will never be able to find out where a message goes.

11.9.5 Tracking log format

The message tracking log format for both Exchange 2000 and 2003 uses a set of tab-delimited fields (see Microsoft Knowledge Base article 246965 for more information), including:

• The date and time when the originating client sent the message. This value is set when the message is first submitted to the server, not when it first arrives on the server that hosts a message tracking log.

• The client IP address and client network name that generated the message

• The partner or messaging service that the message is handed to for processing-for example, the Store (for local delivery)

• The server IP address and server name that generated the message

• The recipient address: Upon submission, Exchange notes the address in its internal X.500 format. The address is shown in SMTP format after it is processed by the Categorizer (event 1023).

• An event identifier to indicate the specific processing that has occurred

• A unique message identifier that never changes, no matter how many servers the message travels through

• The priority of the message: -1 means low, 0 is normal, and 1 is high.

• The total size of the message in bytes

• A flag to indicate whether the message is encrypted. The flag is set to 1 if the message is signed, 2 means encrypted, while 0 means that the message is clear text.

• The version of the Routing Engine running on the server or the version of the SMTP server on a remote server

• Optionally, the subject of the message is recorded if permitted by server properties (see Figure 11.46). The subject is truncated to 256 bytes, if necessary.

• The number of recipients

• The originator's email address: This is the primary address of the originating mailbox, if known. It could be in X.400, SMTP, or a distinguished name format, depending on the transport that introduced the message.

11.9.6 Understanding message tracking log data

The easiest way to understand the data held in the tracking log is to take a copy of a log from a test server, load it into Excel (Figure 11.52), and then try to understand the different events as the Routing Engine processes a message from submission through to final dispatch. Of course, you can opt to review the data in a log from a production system, but typically these logs are very large (>50 MB daily), and it is harder to find a particular message and then track its progress, especially if the message goes to a large distribution list. The size of message tracking logs is not a particular concern for administrators, because it is unlikely that a collection of log files will ever cause a disk to fill up; the logs for ten days should be comfortably less than 750 MB on the largest server. However, large logs do slow down tracking, especially if you attempt to track a message across a number of different servers within an organization. Servers that host active public folders can generate heavy replication traffic, all of which Exchange captures in the message tracking logs. You can identify replication messages by the sender name, which is one of the following:

• The Exchange internal X.500 address for the Public Folder Replication Agent (e.g., EX:/O=Acme/OU=Central/CN=Configuration/ CN=Servers/CN=Server1/ CN=Microsoft Public MDB)

• The SMTP address for the Public Folder Replication Agent (e.g., [e-mail])

Figure 11.52: Examining data from a message tracking log loaded into Excel.

For replication messages, the message tracking logs store the name of the destination server (the replication partner) in the "Recipient-Address" field. The message tracking logs also store data for incoming messages, including nondelivery notifications and delivery and read receipts, since administrators on other servers may want to issue a tracking request on your server.

Figure 11.52 shows the raw tracking log data used by the Message Tracking Center in Figure 11.50, as displayed through Excel. This is a relatively simple example, because the message originates from a MAPI client and goes to two external SMTP recipients. The logical processing performed by the Routing Engine is therefore very straightforward. The following steps are required: Accept the message, check the SMTP addresses, and route the message to an SMTP connector either on the same server (if present) or to the server that hosts an SMTP connector with the lowest possible routing cost. We can track the events in the message tracking log as follows, noting that entries exist for each address on the message:

1027: A client submits the message to the Store Driver.

1019: The Store Driver submits the message to Advanced Queuing to be processed.

1025: The message is processed by Advanced Queuing.

1024: The Categorizer begins to process the message.

1020: The Routing Engine places the message on a destination queue and is prepared for transfer from the server.

1031: The Routing Engine successfully transfers the message and processing completes.

In comparison, the sequence of events for a message delivered to a mailbox on the same server is as follows. The same events occur for deliveries to a mailbox in another storage group.

1027: The message is submitted to the Store Driver.

1019: The Store Driver submits the message to Advanced Queuing.

1025: The message is processed.

1024: The message is handed to the Categorizer.

1023: The Routing Engine places the message on the local delivery queue.

1028: The Store delivers the message to the mailbox.

Messages submitted from OWA clients log the same sequence of entries. Messages from POP3 and IMAP4 clients are submitted directly via SMTP to the Routing Engine and do not go through the Store Driver, so the sequence begins at event 1019. Other common entries include 1023 (successful delivery to a local mailbox) and 1029 (transfer to the MTA for onward delivery). Unlike the logs generated by Exchange 5.5, there is no special entry to mark the expansion of a distribution group. Instead, you will see 1020 (Exchange 2000) or 1033 and 1036 (Exchange 2003) entries written into the log for every member of the group after expansion. This applies to normal distribution groups and query-based distribution groups.

Complex messages addressed to a mixture of single recipients and distribution groups that resolve to local and remote addresses pose a particular challenge for email servers, especially if some of the recipients are included in multiple distribution groups. The fact that the properties of a group influence the number of messages that Exchange eventually generates for dispatch further complicates processing. For example, the properties of the group shown in Figure 11.53 reveal that any delivery reports for messages sent to this distribution group go back to the originator. In other words, if the group contains an invalid address (perhaps a contact whose email address is no longer valid or a mailbox with an exceeded quota), the originator will receive a nondelivery notification. You can set the properties of the group to suppress delivery reports, a situation that cuts down email traffic while creating a virtual blackhole where users are never quite sure if their messages get through. The Routing Engine performs the same per-message processing for out of office notifications, which you can also suppress or enable through the properties of a distribution group. Collectively, a number of message generation permutations exist due to the different values of these properties.

Figure 11.53: Exchange advanced properties of a distribution group.

Because Exchange processes delivery reports and out of office notifications on a per-message basis, if you send a message addressed to two groups, one that permits delivery reports and one that suppresses reports, the Routing Engine must generate two sets of messages. Thus, the recipients in one group will receive a copy of the message with its properties set to allow delivery reports, while recipients in the other group will receive a copy with properties set to suppress delivery reports.

You do not want users to receive multiple copies of a message, and it is easy to end up with this situation if the code does not handle all possible circumstances when the Routing Engine builds the recipient list for a message. To optimize the recipient list, the Routing Engine must expand all distribution groups, check the delivery reports and out of office notification properties of the distribution groups, and then generate the appropriate number of messages to meet the requirements as determined by the properties of each group. The result of the complex nature of recipient lists means that you may see multiple 1020 and 1028 entries for each address on a message when you look through tracking log data.

By comparison, the MTA expands distribution lists (or groups) for first- generation Exchange servers using a process called "fan-out," where the MTA builds the address list and then figures out how to send a single copy of the message to every addressee using the best possible route. First-generation Exchange servers do not support the same set of properties to control out of office messages and delivery reports, so building a recipient list is easier. In either case, if duplicate messages get through, the Store on the receiving server can identify their presence by reference to the message identifier and can then suppress the duplicates. This serves as a backstop to ensure that Exchange never delivers duplicate messages.

11.9.7 Analyzing Message Tracking Data

Details of the event identifiers used in message tracking logs are listed in Appendix B. It is easy to load a complete message tracking log into a program such as Access, where you can analyze the data to your heart's desire. However, if you're interested in using message tracking log data as the basis of any analysis, you are probably better off writing some code to reduce the data to its essential figures (such as number of messages sent during a period, the number of addressees, size of messages, and so on). Expect to process a large amount of data, since a single message sent to a large distribution group generates hundreds of entries in the tracking log. For example, a message sent to two common distribution groups that include 1,350 users (some duplicates) at HP generates over 4,000 entries. In addition, you have to separate entries that you are interested in (typically those belonging to user-generated messages) from those that you probably want to discard (e.g., system-generated messages such as public folder replication).

All versions from Exchange 2000 SP2 onward support the Message Tracking Logs WMI provider, a standard interface that allows program- matic read-only access to data held in message tracking logs. You can use this provider with programming languages that support COM scriptable objects (including WSH) to analyze the log data to your own requirements. Judging by comments made at conferences and Internet mailing lists, Perl is a particular favorite of those who write homegrown code to analyze message tracking log data.

Alternatively, you can buy a commercial product to do the job, such as Promodag Reports (www.promodag.com). MessageStats from Quest Software (http://www.quest.com/messagestats/index.asp) does a more sophisticated job of analyzing the contents of message logs to report on many different aspects, such as the most popular distribution list, the originator of most messages, and so on. Figure 11.54 is a good example of a common report of message traffic that you can generate with a product such as MessageStats, purely by analyzing the contents of the message tracking logs. These utilities usually publish reports in Web format, so they are easy to access and analyze. Be careful to insist that whatever product you choose is capable of capturing data for all connector types, as well as messages sent to nested distribution lists, where the useful data results from the list expansion rather than the fact that a message went to a specific list.

Figure 11.54: MessageStats analysis of tracking log data.

While most administrators are happy to use message tracking data to report and understand traffic volumes, the data is sometimes used for other purposes. For example, if you have a suspicion that users are sending confidential information outside the organization, perhaps to a competitor, you can quickly check where they are sending messages by looking through message tracking logs for all their messages. While this will not tell you what the messages contain, it will tell you where the messages went, and, if you find enough evidence to warrant further investigation, you can then move the investigation to another level by turning on message archival or deploying other tools to capture copies of message traffic. Your management and human resources department must support these activities, since casual checking of people's message patterns is unlikely to be an approved system management activity in most jurisdictions.

^[3] . First-generation Exchange servers place their message tracking logs into a share called tracking.log.

---