Lesson 5: Troubleshooting Technologies That Support Exchange Server 2003

In this lesson, you learn about Windows Server 2003 technologies that operate in the background, supporting the Exchange Server 2003 organization. Although these technologies are not directly concerned with e-mail, public stores, or newsgroups, Exchange Server 2003 needs them in order to work. If Exchange Server 2003 servers cannot resolve the hostnames of clients and other servers to IP addresses, and in turn resolve these IP addresses to Media Access Control (MAC) addresses, then Exchange will not operate. If access to Active Directory fails, then Exchange will fail.

In Lesson 1, you learned how the netdiag and dcdiag utilities can be used to diagnose connectivity problems within an Exchange organization. You also need to know how to check connectivity to other networks, including the Internet. A number of command-line utilities exist for testing connectivity, and you need to become familiar with their use.

After this lesson, you will be able to

• Explain how the Address Resolution Protocol (ARP) works and use the arp command-line utility to manage the ARP cache

• Use the nslookup and ipconfig command-line utilities to diagnose and debug common DNS problems

• Troubleshoot Active Directory problems

• Use command-line utilities such as ping, tracert, ipconfig, and pathping to diagnose network connectivity problems

Estimated lesson time: 90 minutes

Troubleshooting Host Resolution

Hosts on a network identify each other using a MAC address, which is a unique 48-bit number programmed into to every network interface card (NIC). When a host needs to locate another host by hostname, the hostname is first resolved into an IP address (typically by DNS) and then ARP resolves the IP address into a MAC address.

How ARP Works

Typically, ARP operation is invisible to the user. If anything does go wrong, however, you need to examine the ARP cache or use Network Monitor to look at the content of ARP frames. To make sense of the information that these tools provide, you need to know how ARP works.

ARP resolves IP addresses used by TCP/IP-based software to MAC addresses used by network hardware, such as Ethernet. As each outgoing IP datagram is encapsulated in a frame, source and destination MAC addresses must be added. ARP determines the destination MAC address for each frame.

When ARP receives a request to resolve an IP address, it first checks to ascertain whether it has recently resolved that address or whether it has a permanent record of the MAC address that corresponds to the IP address requested. This information is held in the ARP cache. If it cannot resolve the IP address from cache, ARP broadcasts a request that contains the source IP and MAC addresses and the target IP address. When the ARP request is answered, the responding PC and the original ARP requester record each other's IP address and MAC address in their ARP caches.

Resolving a Local Address ARP operation is best illustrated by considering examples of local and remote address resolution. In the first example, Host A, Host B, and Host C are on the same subnet. A ping command is issued on Host A, specifying the IP address of Host C. ICMP instructs ARP to resolve this IP address.

ARP checks the cache on Host A. If the IP address cannot be resolved from cached information, then an ARP request is broadcast to all the hosts on the subnet. The ARP broadcast supplies the source IP and MAC addresses and requests a MAC address that corresponds with the IP address specified. Because the ARP frame is a broadcast, all hosts on the subnet will process it. However, hosts that do not have the corresponding IP address (such as Host B) reject the broadcast frame. Host C recognizes the IP address as its own and stores the IP address/MAC address pair for Host A in its cache. This process is illustrated in Figure 14-16. The target address shown in this figure is the Ethernet address for a broadcast frame (FFFFFFFFFFFF). The MAC address of the target host is not known and is assigned the value 000000000000.

Figure 14-16: The ARP request

Host C sends an ARP reply message that contains its MAC address directly back to Host A. When Host A receives this message, it updates its ARP cache with Host C's address pair. Host A can now send the ICMP ping datagram (or any IP datagram) directly to Host C. This process is illustrated in Figure 14-17.

Figure 14-17: The ARP reply

Resolving a Remote Address When the target address of an IP datagram is on a remote subnet, ARP will resolve the IP address to the MAC address of the NIC in the router gateway that is on the source host's local interface. In this example, Host A and Host B are on different subnets. A ping command issued on Host A specifies the IP address of Host B.

As in the previous example, ARP first checks its cache on the source host (Host A). If the destination IP address cannot be resolved from cache, an ARP request is broadcast. ARP does not know that the target host is remote because routing is an IP function, not an ARP function. The ARP request to resolve a remote IP address is therefore exactly the same as the ARP request to resolve a local address.

All the ordinary hosts on the local subnet reject the request because none of them has a matching IP address. The router, however, checks its routing table and determines that it can access the subnet for the remote host. It then caches the IP address/MAC address pair for Host A and sends back an ARP reply that specifies the MAC address of its gateway NIC. On Host A, ARP caches that MAC address with the IP address it is resolving. As far as ARP on Host A is concerned, it has done its job. Thus, Host A resolves a remote IP address to the MAC address of its default gateway.

At this stage, ARP on the router takes over the task of IP address resolution. First, it checks its cache for the target host's interface. If it cannot resolve the target host's IP from cache, it broadcasts an ARP request to the target host's subnet, supplying the IP address and MAC address of the gateway NIC that accesses the target host's interface.

In the example illustrated in Figure 14-18, Host B recognizes its own IP address, caches the IP address and MAC address of its default gateway, and returns its MAC address in an ARP reply frame directed to that gateway. On the gateway, ARP caches Host B's MAC address along with the IP address it is resolving, and the process is complete. The address pairs in the ARP caches shown in Figure 14-18 are the result of a successful resolution.

Figure 14-18: Resolving a remote IP address

Troubleshooting DNS

Several methods are available for resolving a hostname to an IP address. If the same hostname was resolved recently, the information will normally be available in the host's DNS cache. Cache resolution is quick and efficient and is always the first resolution method that is attempted. Static host files can resolve hostnames, but these require a lot of administrative effort because you need to put them on every computer. NetBIOS methods such as the Windows Internet Name System (WINS) are useful in mixed-mode domains. However, in Windows Server 2003 (and Windows 2000 Server), dynamic DNS (DDNS) is available and is the resolution method of choice. In the remainder of this section, when we consider hosts registering their DNS records dynamically, functionality assumes that DDNS is used.

DNS is discussed in several chapters of this book. In particular, Chapter 10, "SMTP Protocol Configuration and Management," describes the creation of MX records. DNS needs to be available for Active Directory and hence for Exchange Server 2003 server installation. Therefore you can assume that DNS was available and correctly configured on installation, and you need to identify what could cause DNS to fail during Exchange Server 2003 server operation.

Failure of a DNS Server

It is unusual for DNS to fail completely in an Active Directory domain. Typically, Active Directory–integrated DNS is available on more than one domain controller to provide failover support. If Active Directory DNS is not used, then a secondary DNS server is used to back up the primary DNS server. A primary DNS server that is not Active Directory–integrated is a single source of failure. If it goes down, you cannot add new entries to the DNS zone file. However, the secondary will continue to provide a name resolution service, usually for a length of time sufficient to bring the primary DNS server back on line.

However, the failure of a DNS server can cause problems if a host is not configured with the IP address of at least one alternative DNS server. If a host is configured with only one DNS server's IP address and that server goes down, then the host is unable to resolve hostnames, even though the DNS service is available on the other server. Typically, client machines are configured through the Dynamic Host Configuration Protocol (DHCP) and receive a list of all the available DNS servers. However, servers such as Exchange Server 2003 servers are usually configured manually. It is easy to forget to add alternative DNS servers, and everything will work perfectly unless the DNS server fails.

The dcdiag utility described in Lesson 1 of this chapter is mainly used to troubleshoot Active Directory problems, but it can also check DNS operation. The netdiag utility, described in the same lesson, also runs a DNS test. The nslookup utility described in Chapter 10 obtains DNS statistics and lists available DNS servers. You can test connectivity to a DNS server by pinging its IP address. However, possibly the simplest and most useful test is provided by the ipconfig /all utility, which lists the primary and alternative DNS servers available to any host. It is wise to use ipconfig /all to test all your Exchange Server 2003 servers and ensure that they are configured with a list of the IP addresses of all available DNS servers.

A Server Does Not Register in DDNS

When a new server comes online, it takes some time (sometimes as long as 15 minutes) for it to register dynamically in DDNS. If the services provided by that server are required immediately, then you can force registration by opening the Command console and entering the following commands in succession:

• ipconfig /registerdns

• net stop netlogon

• net start netlogon

You need to check Event Viewer for errors if registration fails to occur. However, unless there are other errors, you should see the server's A (host) record appear in DNS almost immediately.

Negative Caching

If DNS resolves a hostname to an IP address, the hostname/IP address pair is held in cache on the host that originated the request (the resolver). However, if resolution is unsuccessful, that information is also cached. This is to stop the waste of resources when a user types in a hostname incorrectly. Suppose, however, that the hostname is correct but because of some fault, the resolution does not take place. This negative information is cached. Suppose that the fault is then fixed. Now every client can resolve the hostname except for the client that tried to do so earlier. It attempts to resolve the hostname from cache, obtains the negative information, and returns an error. You can solve the problem by opening the Command console on that client and entering ipconfig /flushdns.

Real World: The No-Effort Solution

There is an alternative method of solving a negative caching problem that is useful if the client machine is too far away for you to get to easily, and if the user is not sufficiently skilled to access the command prompt and type in the necessary command. Tell the user to take a lunch or coffee break, and the problem will be solved when he or she gets back. Cache entries (including negative entries) time out, typically in an hour or less.

DHCP Problems

Sometimes a client machine cannot access any servers on a network or resolve any hostnames, when all the other clients are having no problems. In this case, check the configuration of the client using the ipconfig utility. There is a good chance that the client's IP address will be in the 169.254.x.x range. What has happened is that the DHCP service has stopped for some reason, or has run out of leases, and the host has been configured through automatic private IP addressing (APIPA). If you fix the DHCP problem, then the client will obtain a DHCP lease in approximately five minutes and the problem is solved. If you need an immediate solution, then open the Command console on the client and enter the following commands:

• ipconfig /release

• ipconfig /renew

  Note

  It is sufficient to enter only ipconfig /renew when converting an APIPA address to a DHCP lease. However, it is good practice always to release an IP configuration before you renew it.

Troubleshooting Active Directory Issues

As with DNS, Active Directory must be available in order to install Exchange Server 2003 and create an Exchange Server 2003 organization. Active Directory is robust because the Active Directory database is replicated between domain controllers. Unless you have only one domain controller (not recommended), there is no single point of failure for the entire Active Directory.

However, Active Directory uses operations masters, and the failure of an operations master affects the functionality of the Active Directory directory service. Typically, the following problems are associated with operations masters:

• You cannot create security principals Assuming that you have sufficient permissions to create a security principal, then typically this problem occurs when the Relative Identity (RID) master is not available or has failed to replicate. This may be caused by a network connectivity problem or may be due to the failure of the computer holding the RID master role. This fault can also occur when the Access This Computer From The Network user right is not assigned to the appropriate groups on the RID master.

• You cannot change group membership Assuming that you have the necessary administrative credentials to manage group membership, this problem typically occurs when the infrastructure master is not available. This may be caused by a network connectivity problem. It may also be due to a failure of the computer holding the infrastructure master role.

• Users cannot authenticate This can be a problem in mixed mode domains in which some clients are not Active Directory–aware. Typically, it happens when the user's password has expired and the primary domain controller (PDC) emulator master is not available. This may be caused by a network connectivity problem. It may also be due to a failure of the computer holding the PDC emulator master role.

In all of these cases, you can identify the computer holding the RID master role, the infrastructure master role, or the PDC emulator role by issuing the netdom query fsmo command from the Command console of any host in the domain, as shown in Figure 14-19. You can then repair or replace the computer holding the appropriate operations master role. You may need to seize the operations master role. Alternatively, you may need to resolve the network connectivity problem.

Figure 14-19: Identifying operations masters

The dcdiag utility, described in Lesson 1 of this chapter, is a powerful tool for checking the general health of Active Directory. As previously stated, Active Directory is robust. Provided that it was configured correctly when Exchange Server 2003 was installed, very few Active Directory problems are likely to occur.

Important

If you have any Windows 2000 domain controllers in your domain, ensure that SP3 or later is installed on them. Otherwise, Exchange Server 2003 cannot access them.

Troubleshooting Network Connectivity

Lesson 1 of this chapter discussed the netdiag utility and Chapter 10 discussed the telnet and nslookup utilities. In this lesson, we saw how the ipconfig utility is used to display a host's IP configuration, to register a host in DDNS, to clear the DNS cache, and to obtain a DHCP lease. In this section, we discuss various other command-line utilities that can be used to troubleshoot network connectivity.

Ping

The ping utility uses ICMP echo commands to test IP connectivity. Some firewalls and routers filter out ICMP packages, and you cannot ping across them. In spite of this limitation, ping remains one of the most useful and widely used troubleshooting tools. You can test that TCP/IP is correctly installed on a host by pinging the loopback address 127.0.0.1. You can ping all the interfaces on your local computer. You can ping another host on your subnet by both IP address and hostname to test connectivity and name resolution. You can ping all the servers that you need to connect to. Finally, you can ping a host on a distant subnet to test internetwork connectivity.

Ping lets you set a number of parameters to specify, for example, the size of the ping packets, how many packets to send, whether to record the route used, what time-to-live (TTL) value to use, and whether to set the "do not fragment" flag. If you open the Command console and enter ping /? you can obtain details about these options.

For example, ping -n 6 -l 2000 -w 10000 10.0.10.100 pings the host with IP address 10.0.10.100 six times using a ping packet 2,000 bytes in size. By default, ping waits for up to 4,000 milliseconds for each response to be returned before it displays the "Request Timed Out" message. If you are pinging a remote system across a slow link, then you can use the –w (wait) option to specify a longer timeout. In the example given, the timeout is 10 seconds.

Arp

You can use the arp command-line utility to manage the ARP cache, which is a table that stores IP address/MAC address pairs. Whenever a source computer resolves a target IP address using an ARP request broadcast, the address pair for the target computer is stored in the source computer's cache. Similarly, when a target computer responds to an ARP request with an ARP reply, the address pair of the source computer is stored in the target computer's cache. Cache entries generated automatically by ARP resolution are called dynamic entries. They remain in the cache for a specified TTL (2 minutes by default) and, if not accessed during that time, are then discarded. If an entry is referenced again before it is removed, its TTL is increased by another two minutes. Thus, a frequently referenced entry can increase its TTL up to a maximum of 10 minutes.

Address pairs for frequently accessed targets, such as default gateways or member servers, can be entered manually. Manually entered address pairs are called static entries; they persist in cache until the host is rebooted or until they are manually deleted.

Tip

Static arp cache entries are deleted on reboot. If you want them to be persistent, create a startup script that re-enters them.

Nbtstat

Network basic input/output system (NetBIOS) over TCP/IP (NetBT) resolves NetBIOS names to IP addresses. TCP/IP provides many options for NetBIOS name resolution, including local cache lookup, WINS server query, broadcast, DNS server query, and lmhosts and hosts file lookup. In theory, DDNS is the main name resolution method in native mode Windows 2003 and Windows 2000 Active Directory domains, and NetBT can be disabled. In practice, many services and BackOffice products (for example, Systems Management Server) use NetBT. You need to investigate very carefully if you want to remove it.

You can use the nbtstat command-line utility to troubleshoot NetBIOS name resolution problems. The available options are as follows:

• nbtstat -n Displays the names that were registered locally on the system by programs such as the server and redirector.

• nbtstat -c Shows the NetBIOS name cache, which contains name-to-address mappings for other computers.

• nbtstat -R Purges the name cache and reloads it from the lmhosts file.

• nbtstat -RR Releases NetBIOS names registered with a WINS server and then renews their registration.

• nbtstat -a name Performs a NetBIOS adapter status command against the computer specified by name. The adapter status command returns the local NetBIOS name table for that computer plus the MAC address of the adapter.

• nbtstat -S Lists the current NetBIOS sessions and the status of each, including statistics.

  Note

  The nbtstat utility was developed for UNIX and is case-sensitive.

Netstat

You can use the netstat command-line utility to display protocol statistics and current TCP/IP connections. The available options are as follows:

• netstat -a Displays all connections.

• netstat -r Displays the route table plus active connections.

• netstat -o Displays process identities so that you can view the port owner for each connection.

• netstat -e Displays Ethernet statistics.

• netstat -s Displays per-protocol statistics.

• netstat -n If you use this option, addresses and port numbers are not converted to names.

Tracert

You can use the tracert command-line utility to determine the path that an IP datagram takes to reach a destination. The utility uses the IP TTL field and ICMP error messages to determine the route from one host to another through a network. Because it uses ICMP, tracert will not work across firewalls and routers that block ICMP frames. You can try tracert www.microsoft.com. This may or may not work depending on the route the IP datagram takes to its destination. You can also use tracert to trace the path of a datagram through your intranet. The utility is useful for troubleshooting large networks where several paths can be taken to arrive at the same point. The tracert command has the following syntax:

tracert [–d] [–h maximum_hops] [–j host-list] [–w timeout] target_name

The options are described in Table 14-2.

Table 14-2: Tracert Command-Line Options

Option	Description
–d	Specifies that IP addresses are not resolved to host names.
–h maximum_hops	Specifies the number of hops to allow in tracing a route to the host named in target_name.
–j host-list	Specifies the list of router interfaces in the path taken by the tracert utility packets.
–w timeout	Waits the number of milliseconds specified by timeout for each reply.
target_name	Name or IP address of the target host.

Pathping

The pathping utility is a route tracing tool that combines the features of ping and tracert and gives additional information that neither of those tools provides. The utility sends packets to each router on the way to a final destination over a period of time and then computes results based on the packets returned from each hop. Because the command shows the degree of packet loss at any given router or link, you can determine which routers or links might be causing network problems. A number of options are available, as shown in Table 14-3.

Table 14-3: Pathping Options

Option	Name	Function
–n	Hostnames	Does not resolve addresses to host names.
–h	Maximum hops	Maximum number of hops to search for target.
–g	Host-list	Loose source route along the host list.
–p	Period	Number of milliseconds to wait between pings.
–q	Num_queries	Number of queries per hop.
–w	Time-out	Pathping waits this many milliseconds for each reply.
–i	address	Instructs pathping to use the specified source address.
–4	IPv4	Forces pathping to use IP version 4.
–6	IPv6	Forces pathping to use IP version 6.

The default number of hops is 30, and the default wait time before a timeout is three seconds. The default period is 250 milliseconds, and the default number of queries to each router along the path is 100.

When you run pathping, you first see the results for the route as it is tested for problems. This is the same path that the tracert command shows. The pathping command then displays a busy message typically for the next 125 seconds (this time varies depending upon the hop count). During this time, pathping gathers information from all the routers previously listed and from the links between them. At the end of this period, it displays the test results.

Practice: Managing the ARP Cache and Analyzing an ARP Packet

In this practice, you manage the ARP cache and use Network Monitor to capture and analyze an ARP packet. As did the previous practice, this practice assumes that Network Monitor has been installed and that this is not the first time it has been used. The instructions for installing Network Monitor are given in Chapter 13. If this is the first use of Network Monitor, you need to instruct it to monitor Local Area Network when prompted.

Exercise 1: Manage the ARP Cache

To manage the ARP cache on Server01, perform the following steps:

1. On Server01, open the Command console.

2. Enter arp. When entered with no arguments, the utility lists the command syntax (as does arp /?).

3. Enter arp -a. This displays the current ARP cache, as shown in Figure 14-20.

   
   Figure 14-20: The ARP cache

4. Because Server01 is multihomed, you can use the interface addresses to display the ARP cache for each interface. Enter arp -a -N ip_address, where ip_address is the IP address of Local Area Connection on Server01, as shown in Figure 14-21.

   
   Figure 14-21: Displaying the ARP cache for a specified interface

5. Enter arp -d without arguments to delete all cache entries.

   Note

   The command arp -d ip_address will delete an individual cache entry.

6. Enter ping server02.

7. Enter arp-a to display the IP address/MAC address pair that the ping operation places in the arp cache, as shown in Figure 14-22.

   
   Figure 14-22: The ping command adds an entry to the ARP cache

8. Frequently accessed machines on your subnet, such as the default gateway, should be placed in the ARP cache as static entries. Enter arp -s 10.0.10.1 00-d0-b7-4c-56-a8 to add a static entry. Both the IP address and the MAC address in this step are examples and their values are not significant.

9. Enter arp-a to view the ARP cache. Both the static and dynamic entries should be present.

10. Wait for approximately 10 minutes, and then enter arp -a to list the ARP cache entries. The dynamic entry is removed because its TTL has expired. The static entry will remain until Server01 is rebooted.

11. Static entries can, however, be removed using the arp -d command. Enter this command to clear the ARP cache.

Exercise 2: Use Network Monitor to Display the Contents of an ARP Broadcast Frame

Before you start this exercise, ensure that the ARP cache is clear. If it holds entries for Server02, then the broadcast frames that you want to analyze will not be sent.

To use Network Monitor to display the contents of an ARP broadcast frame, perform the following steps:

1. On Server01, open Network Monitor. On the Capture menu, click Start.

2. Do not close Network Monitor. On Server01, open the Command prompt and enter ping server02.

3. In Network Monitor, on the Capture menu, click Stop And View.

4. There should be two ARP-RARP frames at or near the top of the list in the Summary pane. (Note that Network Monitor calls ARP frames ARP-RARP, where RARP stands for Reverse ARP.) Click on the Request frame (the first one), expand the list in the detail pane, and read the source IP and MAC addresses, as shown in Figure 14-23.

   
   Figure 14-23: Analyzing the ARP-RARP Request frame

5. Analyze the Reply ARP-RARP frame in the same way.

Lesson Review

The following questions are intended to reinforce key information presented in this lesson. If you are unable to answer a question, review the lesson materials and then try the question again. You can find answers to the questions in the "Questions and Answers" section at the end of this chapter.

1. You suspect that a router somewhere in your intranet is causing transmission delays and may be dropping frames. Which tool would give you the most information about the problem?

   1. pathping

   2. ping

   3. tracert

   4. netstat

2. You successfully ping a host on a remote network. You examine the ARP cache on your computer and find that it contains one IP address/MAC address pair. What are these addresses?

   1. The IP address and MAC address of your default gateway

   2. The IP address and MAC address of the remote host

   3. The IP address of the remote host and the MAC address of your default gateway

   4. The IP address of your default gateway and the MAC address of the remote host

3. You ask all users to work offline while you fix a connectivity problem. When the users come back online, one of them is unable to access a server by hostname. All other users can access the server without any problems. The user admits that he was slow coming offline and tried to access the same server unsuccessfully before doing so. What command will fix the problem?

   1. ipconfig /release

   2. ipconfig /renew

   3. ipconfig /registerdns

   4. ipconfig /flushdns

Lesson Summary

• ARP resolves IP addresses to MAC addresses. You can use the arp utility to display and manage the ARP cache.

• You can use the nslookup and ipconfig command-line utilities to diagnose and debug common DNS problems. Also, you can use ipconfig to diagnose incorrect TCP/IP configuration.

• The dcdiag utility can be used to diagnose Active Directory problems. Sometimes you may need to transfer or seize an operations master role.

• You can use command-line utilities such as ping, tracert, ipconfig, and pathping to diagnose network connectivity problems. Nbtstat and netstat return NetBIOS and TCP/IP statistics, respectively.