9.5 Categorization and routing

---

All inbound messages that arrive on an Exchange server are eventually processed by the Advanced Queuing component. They then move to the Pre- Categorizer queue, where a Categorizer thread picks them up for further processing. A number of operations are then performed on the message:

• The address of the message sender is extracted from the envelope. A search is then conducted against addresses held in the AD (using the proxyAddresses attribute) to determine whether it can be resolved.

• Each entry in the recipient list held in the envelope is examined and a lookup performed against the AD to determine whether the recipient can be resolved. If the recipient is a distribution group, the group is expanded to include the members of the group in the recipient list, but only if expansion is allowed on the server. If a recipient cannot be resolved, it is marked "unknown."

• A check ensures that the resolved recipients can accept messages, according to any restrictions that might currently apply for the recipients, such as quota exceeded on a mailbox.

• Any per-sender and per-recipient limits are checked.

• Internet message format settings (such as whether the target domain can only accept plain text messages) are checked and properties set to ensure that recipients receive messages in a format they can deal with.

• Occasionally multiple copies of a message may need to be created, because different recipients must see different versions of the message. A good example of where this happens is if the Routing Engine can find one recipient in the AD that has a mailbox on an Exchange server and another recipient that does not. In this instance, Exchange creates one version of the message for delivery inside the organization (which might, for example, send TNEF content as a binary body- part) and another for delivery outside the organization (which might have the same bodypart encoded in Base64).

• Each recipient is then marked as either "gateway" or "local." Gateway recipients are reached via the MTA or a connector, while "local" recipients have mailboxes on an Exchange server in this organization.

After categorization, messages are placed in a prerouting queue, and the routing event is called to create a "next-hop identifier" for the message. The Routing Engine writes information about categorization processing into the XEXCH50 properties of messages so that subsequent Exchange servers along the path do not repeat work. The next-hop identifier tells the queuing engine which per-domain queue will handle the message.

9.5.1 Handling messages for protected groups

Exchange 2003 allows you to protect mail-enabled objects, including query-based distribution groups, so that they will only accept messages from authenticated users. This feature requires changes in the SMTP service, Routing Engine, and categorization to ensure that Exchange stamps incoming messages with authentication data and then checks to validate that the message does in fact come from an authenticated user.

Exchange 2003 processes incoming SMTP messages as before, but now appends some authentication data^[6] in the "Mail From:" field as defined by RFC 2554. This data is retained as long as the message passes between Exchange 2003 servers (as in front- and back-end servers), but drops if the message goes to an Exchange 2000 or earlier server. Exchange uses the same mechanism to transmit data for the "Sent by" field when a user sends a message on behalf of another user.

After the SMTP service accepts the message, it passes it to the transport engine, which detects that authentication data is present and then performs a security check to validate the data. If the check succeeds, the transport engine adds a flag to indicate that this is a trusted sender. Note that the Store sets the same flag on messages that come from authenticated users, which means that every message that originates on an Exchange 2003 server has the trusted flag set. After the security check, the categorization engine then takes responsibility to evaluate the message and decide whether to deliver it to its destination (the group). The following process occurs:

• If we have an authenticated user, and delivery to the group requires authentication, the categorization engine proceeds as normal.

• If we have an authenticated user, and the group is open to anyone, delivery proceeds as normal.

• If the user is unauthenticated, and the group is restricted, the categorization engine logs event 9007 (the restricted group name is logged, but not the sender that failed) and dumps the message.

• If the user does not authenticate, and the group is open to accept messages from anyone, delivery proceeds as normal.

9.5.2 Moving messages off queues

The Routing Engine identifies recipients on other Exchange servers by examining the HomeMDB attribute in their AD entries. If this attribute holds the distinguished name of a Mailbox Store, Exchange knows that the recipient belongs to the same organization and it can deliver the message by routing the message to the server that hosts that Mailbox Store. The Routing Engine can examine the routing table to determine how to route the message and write this information into the next-hop identifier. If the Mailbox Store is located on the same server, Exchange can make a local delivery by placing the message in the local delivery queue.

Figure 9.9: Messages waiting to go.

When you view a queue, you are looking at data structures that the SMTP service keeps in memory. The actual messages that form the contents of the queues remain in the Store or an NTFS directory (\exchsrvr\mailroot\ queue-see Figure 9.9) until they are transmitted. You get best performance when the queue is located on a fast drive, preferably away from its standard location, which is to go on the same drive as the Exchange binaries. In Exchange 2000, you have to edit the configuration of the SMTP virtual server with ADSIEDIT and insert the path to the new folder into the value for the msExchSmtpQueueDirectory attribute. Even worse, you have to make the same edit for every virtual server on every server that you want to tune. With Exchange 2003, you simply select the SMTP virtual server through ESM, choose the "Messages" property page, and then browse for the directory where you want to relocate the queue, as shown in Figure 9.7. Changing the queue directory is not something you need to worry about for small servers, but it can deliver a real benefit for any server that handles significant SMTP traffic, such as bridgehead servers for large routing groups, or any server that acts as an SMTP entry point for the organization.

Exchange uses SMTP to deliver messages to other Exchange servers in the same routing group or to servers connected by a routing group connector, as well as messages going to external SMTP addresses. In these instances, Exchange places the messages on the relevant queues, where they remain until the connectors retrieve them for onward dispatch.

Figure 9.10 shows how Exchange 2000 lists messages waiting on queues, while Figure 9.11 shows the streamlined and much improved queue viewer interface in Exchange 2003. The improvements made in Exchange 2003 include:

• All queues are viewed together rather than as per virtual server (or MTA).

• There are new queues for "Deferred" messages (awaiting delivery at a future time), "Retry" (delivery failed because of a problem that might be transient), and "DSN" (messages checked against DSN but still not routed). These queues give a more complete view of the current routing situation.

• You can disable outbound mail at one point.

• You can set a queue refresh interval (default is two minutes) through the "Settings" command button.

• You can select a queue (obviously one that has messages on it) and then search based on criteria such as sender, recipient, and message state (e.g., "frozen"). (See Figure 9.12.)

Figure 9.10: Viewing queues-Exchange 2000.

Figure 9.11: The Exchange 2003 queue viewer.

Figure 9.12: Search message queues.

Collectively, these changes make the Exchange 2003 queue viewer easier to work with and more functional. Table 9.4 lists the major queues that you can expect to see in the Exchange 2003 queue viewer. Obviously, any queue with more than a couple of messages waiting deserves some attention to determine why the messages are on the queue.

The queue viewer reports a state for each queue. These states are:

• Ready:   The queue is ready to accept messages.

• Disabled:   The link between this server and the next-hop server is not available.

• Active:   There is an active connection between this server and the next-hop server.

• Retry:   Previous connection attempts have failed and the server is waiting for another attempt. You can use the Force Connection command to change a queue in the Retry state to Active, which causes the Routing Engine to begin processing messages on the queue again. If the problem that caused the original retry state still exists, you will see the queue revert from Active to Retry very quickly.

• Scheduled:   The queue is waiting for the next scheduled connection attempt.

• Remote:   The queue is waiting for a remote server to execute a dequeue command (TURN/ETRN) to fetch the queued messages.

• Frozen:   An administrator has frozen the queue. The Routing Engine can add new messages to the queue, but it will not attempt to process them onward. Freezing an Active queue immediately terminates the transport sessions for that queue.

All the queues shown in the viewer are per-link queues, so they reflect the next step in the routing path for messages. The names used in the queues are the connectors that will transfer the message the next step along its journey. You might expect that the queue viewer would list domain queues, so that you could see all of the messages going to a particular final destination, but the link queue is actually more interesting; it shows queued messages waiting to depart from a specific server, and you know where the messages are destined to go next.

Table 9.4: Exchange 2003 Queues

Queues	Protocol	Description
Local domain name (Local Delivery)	SMTP	Contains messages queued for delivery to a mailbox on the local server. Queues are named for "Local Delivery" or the name of the local server.
Messages awaiting directory lookup	SMTP	Messages waiting for address validation against AD-for example, expanding the contents of a distribution group.
Messages waiting to be routed	SMTP and X.400	Messages waiting for the Routing Engine to determine the next link queue.
Messages with an unreachable destination	SMTP	Messages that the Routing Engine cannot currently find a route to, possibly because a connector to a specific foreign email system is unavailable.
Failed message retry queue	SMTP	Queued messages that failed in the initial submission
DSN messages pending submission	SMTP	DSN (service) messages waiting to be submitted to the categorizer
Presubmission	SMTP	Messages that the Routing Engine has accepted but has not yet begun processing
PendingRerouteQ	X.400	This queue is empty unless there are messages pending reroute after a temporary connection problem.
READY-IN	Foreign	Messages that have arrived from a foreign email system (e.g., cc:Mail) via an MTA connector. The connector has converted the message format (e.g., content conversion, attribute mapping, etc.), but the recipient addresses have not yet been resolved.
MTS-IN	Foreign	Messages that have arrived from a foreign email system and the routing engine has validated their addresses
MTS-OUT	Foreign	Messages going from Exchange to a foreign email system via an MTA connector and still awaiting address resolution
READY-OUT	Foreign	Messages going from Exchange to a foreign email system with resolved addresses (but message format transformation may still be outstanding)
BADMAIL	Foreign	Messages that an MTA connector cannot process for some reason. The messages remain on the queue until an administrator removes them.

There is a situation when you might see two queues for the same link. This occurs when routing information has changed-you may have added a new connector, the properties of an existing connector were adjusted, or Exchange has updated the link state table because of a network problem. In this situation, the Routing Engine invalidates the old queue and reroutes messages from the old queue to other queues, one of which it might have created for the same link. Exchange then cleans up the old queue and removes it.

Normally, if all network links are available and messages are flowing freely, very few messages are queued. The SMTP transport processes messages very quickly, and there should be little delay between a message entering the queue and its final transmission. Because they do not handle directory replication traffic, current Exchange queues are usually much smaller than Exchange 5.5 queues. Some Exchange 5.5 servers, normally those in hub sites, process hundreds of thousands of directory replication messages daily; all of the replication message traffic is now under the control of the AD. (See Figure 9.13.)

Figure 9.13: Properties of a queue.

Exchange ensures that routing information is as accurate as possible by replicating configuration data along with other link state information. If you change the configuration of a connector, the changes replicate along with other link state updates, so they will get through if the servers can contact each other via SMTP, irrespective of whether AD replication works properly. See section 9.7 for more information on the link state routing mechanism.

Potential still exists to swamp message queues with large quantities of public folder replication traffic, especially if you replicate the contents of large Internet newsgroups around an organization. Keep an eye on the queues, and take action to restrict public folder replication schedules if queues build up and seem to include a lot of public folder traffic. If you see messages building on a queue, you can use the "enumerate" option (Exchange 2000) or the "Find Messages" option (Exchange 2003) to list the messages and reveal more details, as shown in Figure 9.14. Again, this is the Exchange 2003 user interface.

Figure 9.14: Examining a queue.

You can then select an individual message from the set and examine its properties (Figure 9.15). At this point, you can also decide to delete the message from the queue and opt to send an NDR back to the originator. You can also choose to delete the message invisibly (no NDR).

Figure 9.15: Properties of a message on a queue.

Of course, there are reasons why messages build up. The most common cause is that a server is unavailable because it is down or the network link is saturated and Exchange cannot make a connection to the remote server. Other, less common, reasons include the possibility that a server might be underpowered and unable to provide the raw processing power to deal with messages sent to it, or the sheer number of messages has exhausted the capacity of the SMTP transport to process them quickly enough to reduce the queues. The messages might be very large and require a substantial amount of time to transfer to the target server, or the target server might be slow to process incoming traffic. The most common problem is that a network link or server is unavailable when Exchange attempts to make a connection.

In Figure 9.10, we see that the queue for "qemea-dc1.qemea.qtest.cpqcorp.net" is currently in a retry state, meaning that the SMTP service has failed in an attempt to transfer messages to this destination and is now waiting to make a retry. Viewing the properties of the queue, as seen in Figure 9.13, reveals more detail about the likely cause of the problem. In this case, we can see that Exchange believes that the remote server does not exist! While it is possible that someone has removed the server from the network, a more likely reason for this error is that Exchange (and, more specifically, the SMTP service) is not running on the server, so all attempts to contact the server to pass messages have failed.

9.5.3 Unreachable domains

Any time a domain is unreachable, the SMTP service registers a warning in the system event log. You can use these events to determine what domains are proving troublesome and take steps to try to establish better connectivity. Sometimes problems are purely transitive and result from servers going offline or network errors. Other problems may point to an issue in your infrastructure, such as DNS updates that are not properly applied or propagated, or perhaps a server that is not routing messages to the correct smart host.

In the case shown in Figure 9.16, we can see that the warning was issued because the SMTP service tried to send a message to a recipient with a Compaq.com address, but the message could not get through because the domain was unreachable. Sending a simple "PING" command to a troublesome domain will usually determine whether the problem still exists, or you can use the NSLOOKUP utility to validate whether Exchange is using the correct name servers to resolve the domain. Remember that a successful PING only indicates that the computer is available on the network; it does not mean that you can send data to the computer. Something else might be causing SMTP to fail to connect: the Exchange services might not be running on the target computer or a disk might even be full. During virus outbreaks, system administrators often disable network connectivity to servers while they clean the system, so this is another thing to check if you see queues building up for a specific server or domain.

Figure 9.16: SMTP error in the system event log.

Exchange servers do not need MX records to enable SMTP message interchange across routing group connectors, because they use routing information held in the configuration naming context in the AD rather than DNS MX records. Exchange servers do need MX records if they communicate with each other across an SMTP connector, unless you route everything through a smart host. See the discussions about creating routing group connectors in section 9.8.1 and creating SMTP connectors in section 9.9. Note that Exchange does use DNS "A" records to resolve server names to IP addresses.

We have already said that the queue viewer displays link queues, but surely you can regard a domain such as hotmail.com as a final destination? The answer is: yes and no. It is true that an address such as hotmail.com is the final destination for a message, but it is also the next hop or link along the message's path. Let us assume that you configure an SMTP connector with an address space of *, meaning that the SMTP connector is able to route messages to any Internet domain. The message sent to hotmail.com is placed on the queue for the SMTP connector (its final destination), but its next hop is to hotmail.com, which is the reason why this queue is shown. Using the same logic, you can see link queues for messages going to a server in the same routing group. Again, this is a final destination, but it is also the next hop in the message's route.

Note that link queues are transient and only appear when they are in use. If you go back to refresh the queue viewer after Exchange has successfully transferred the message, the queue for hotmail.com may not be visible. Queue cleanup happens each time a message goes through the system, and unused or unwanted queues are removed at this point.

When you look at queues, you will see two special queues-the "Mes- sages awaiting directory lookup" queue and the "Messages waiting to be routed" queue. The first holds messages that the Categorizer has not yet processed. The second queue holds messages that Exchange has not yet analyzed to determine on which link queue to put the messages. If messages show up on this queue, it is an indication that something is not working as planned, so it is a good idea to check the application event log and review any errors shown there.

Messages to other routing groups connected by an X.400 connector transfer via the MTA. Sometimes, an interim routing group is along the route to the eventual X.400 connection, and, if so, the Routing Engine dispatches the message to that routing group, where the message is then reassessed and routed to X.400. Once the message arrives at the server that hosts the X.400 connector, it is placed on the local delivery queue and processed by the transport core Store driver, which assesses the recipient list again and makes any local deliveries for mailboxes on this server. The message is then placed into the MTA's MTS Out folder in the Store, which is the final stop before it is dispatched out via the X.400 connector.

The same type of routing occurs for foreign connectors. The Routing Engine assesses the route to the connector and either dispatches the message directly to the connector, if it is available locally, or routes it via another routing group to reach the connector.

All mail-enabled users, including those who have their mailboxes on Exchange 5.5, also have a value in their HomeMBD attribute, which points to the distinguished name of the Mailbox Store for their mailboxes. The Routing Engine can use this information to consult the routing table to determine whether the server exists in the same routing group; a mixed- mode site; or in another Exchange 5.5 site, which must be routed through a connector hosted by an Exchange 5.5 server. Delivery to an Exchange 5.5 server in the same site is routed via the MTA and executed with RPCs.

Mail-enabled contacts do not have a HomeMDB attribute, so the Routing Engine consults the TargetAddress attribute, which should contain a routable address in the form "ADDRESS-TYPE: ADDRESS"-for example: "SMTP: [e-mail]." The categorizer encapsulates the found address inside an SMTP address, with the right-hand side of the address containing the FQDN of the server that has the most suitable gateway to process the address. Of course, if the address is an SMTP address, no rewrite is necessary and Exchange can route the message directly to the lowest-cost SMTP connector that is able to handle the address space specified in the address.

9.5.4 Exchange and MX records

In a classic SMTP environment, the MX records held in DNS indicate the preferred servers to handle mail traffic for a domain. MX records are not strictly required for mail to flow into a domain, and if the MX records are not present, external servers will attempt to connect to all the servers listed with "A" DNS records for the domain. In effect, therefore, the MX records enable us to channel messages to a set of servers that we can configure to handle mail effectively.

Exchange must accommodate two cases when it wants to send messages: connection to a server within the same Exchange organization or connection to a server outside the organization. The server outside the organization could be another Exchange server, but it could also be a UNIX server running sendmail or some other server that supports SMTP, such as Lotus Notes.

Exchange consults the configuration data in the AD to decide whether a target server is inside the organization. If it is, Exchange fetches the GUID of the server or connector from the AD and uses it to determine the next hop. The Routing Engine has a special protocol sink that can act as a DNS resolver, which returns the IP addresses of SMTP servers to Exchange. During this process, Exchange uses the GUID to reference the AD to retrieve the host name. The resolver responds with the host name of the server to connect to or one of the servers that can act as a bridgehead for a connector.

The resolver then passes the name of the selected server back to the Routing Engine, which can then check DNS to get the IP address of the server and proceed to make a connection to port 25 to transfer the message. While the Exchange 2000 version of the DNS resolver works, experience proved its performance was not always efficient, so Exchange 2003 adds logic to improve resolution by using multiple DNS servers and retries if a server becomes unresponsive.

If a target server or domain is outside the organization, Exchange performs a standard DNS lookup to locate an MX record for the name. Three possibilities exist:

• An MX record is found. In this case, the Routing Engine uses the MX record priorities to locate the name of the server to send the message to, and then resolves the name to find the IP address to which an SMTP connection should be made.

• DNS responds with "authoritative host not found" for the record, which indicates that DNS was able to contact the root of the DNS tree and verify that not even an "A" record existed for the target domain. The most common cause is that the sender mistyped the address, turning, for instance, compaq.com into compak.com. The message is returned to the originator with a nondelivery notification.

• No MX record is found, but either the DNS root cannot be reached to make an authoritative determination or an "A" record is found for the domain. In this case, Exchange attempts to use a gethostbyname() call, which performs both a DNS lookup for the "A" record and also attempts a WINS/NetBIOS name resolution.

In all cases, the result should be that the Routing Engine knows where to send the message. In summary, Exchange servers do not need MX records to send messages to other servers either inside or outside the organization, but MX records are used if they exist and are defined in DNS. The Routing Engine is also able to retrieve information about servers and connectors from the AD and resolve this data into the basic IP address to transfer messages.

^[6] . The format is AUTH=user@namespace as in AUTH=[e-mail].

---