22.5. Troubleshooting Networking Problems

Poor network server or workstation performance is often associated with physical network problems. In an overly congested or poorly designed network, the performance level of a server or workstation can do little to improve overall network performance. In a resource-sharing environment, the network communications subsystem, because of its heavy use in this environment, is one of the most likely places for network performance problems. The resolution can involve adding more network interface cards (NICs) and rebalancing the network load or adding more network bandwidth.

! Important

When making modifications to any network, document all modifications to ensure that accurate records are available if a similar network problem occurs at a later date.

The network subsystem is the entry point for all server data. NIC performance problems limit the amount of data that can be handled by the server. Monitoring this subsystem involves the following:

• Eliminating NIC queuing

• Identifying bandwidth deficiencies

• Controlling processor and I/O bus overhead attributed to the NIC

• Optimizing NIC-related configuration parameters

• Properly configuring the network infrastructure

In Microsoft Windows 2003, the following features affect network performance:

• Domain Name System (DNS)

• Domain controllers

• Environment

• Global catalog servers

• Operation masters

• Kerberos authentication and public key infrastructure

Often performance problems can be traced to the networking components in the server, or to the network infrastructure that is external to the server.

When evaluating the network subsystem, start by making an inventory of the existing network and identifying any existing bottlenecks. Performance Monitor can help you isolate performance bottlenecks. Using the Performance Monitor and its logging facility will help you identify problem areas.

The main concern when searching for bottlenecks is how the network subsystem interacts with the memory, processor, and disk subsystems. Performance data for the network subsystem is collected from the Network Interface Performance object.

22.5.1 Network Interface Counters

The Network Interface Performance object allows independent observation of each NIC. You must install the Simple Network Management Protocol (SNMP) service and TCP/IP to monitor the network interface object.

The following table describes the most useful network interface counters.

Counter	Description
Output Queue Length	The length of the output packet queue (in packets). If this is longer than two, delays are being experienced and the bottleneck should be found and eliminated if possible. Because the requests are queued by the network driver interface specification (NDIS) in this implementation, this will always be zero.
Current Bandwidth	An estimate of the current bandwidth of the interface in bits per second. For interfaces that do not vary in bandwidth or for those where no accurate estimation can be made, this value is the nominal bandwidth.
Bytes Total/s	The rate at which bytes are sent and received on the interface, including framing characters.

22.5.2 Network Segment Counters

The Microsoft Network Monitor agent is required to monitor the Network Segment object. With the Network Monitor agent installed from the ProLiant Support Paq (PSP), you can gather performance data using Performance Monitor on the Network Segment object. Data is sourced through the agent and passed to the System Monitor.

The following table describes the Network Segment counters.

Counter	Description
Network Segment: % Network Utilization	Displays a percentage of the network bandwidth being used. It confirms data provided by the queuing counter and measures the degree of the bottleneck. Segmenting and upgrading the network infrastructure yields measurable improvements in latency and bandwidth when sustained usage is greater than 60%.
Network Segment: Frames/s	Counts the number of frames transmitted on the network per second.
Network Segment: Bytes/s	Displays the number of bytes transmitted on the network per second. This counter indicates segment saturation. Note: To convert bytes/s to Mb/s, use the following formula: Mb/s = [(bytes/s * 8) / 1024] / 1024 If the Mb/s calculation approaches 60% of the network capacity, the segment has surpassed maximum efficiency and might need to be segmented.
Network Segment: Broadcasts/s	Displays the number of broadcast frames on the network.
Network Segment: Multicasts/s	Displays the number of multicast frames on the network.

Note

The Network Segment object can negatively impact system performance. The data collection for this performance object places the NIC in a promiscuous mode. When a network card is in a promiscuous mode, it reads all the network packets.

! Important

Additional NICs do not improve performance if the current NIC is not close to saturation.

22.5.3 Running the HP ProLiant Network Configuration Utility 7

The HP ProLiant Network Configuration Utility 7 enables you to configure and monitor HP NICs running under Windows 2003.

There are two ways to access the ProLiant Network Configuration Utility 7:

1. From the Windows Control Panel, double-click the HP Network icon.

2. Double-click the HP icon in the system tray.

The ProLiant Network Configuration Utility 7 displays NICs recognized by Windows Server 2003 in its list. The icons used by the ProLiant Network Configuration Utility 7 are described in the following table.

10Mb/s and 100Mb/s Icons	Gigabit Icons	Description
		Active OK NIC is operating properly. The driver is installed in the Registry and the driver is loaded. If the NIC is a member of a team, this is the active NIC.
		Installed inactive NIC is installed but not active.
		Cable fault NIC is installed in the Registry and the driver is loaded. The broken cable indicator means the cable might be unplugged, loose, or broken; or the switch/hub might be defective.
		Inactive cable fault A cable fault occurred when the NIC was inactive.
		Hardware failure on the NIC NIC is installed in the Registry and the driver is loaded. The driver is reporting a hardware problem with the NIC. This indicates a serious problem. Contact your service provider.
		Unknown (teamed NICs only) NIC is unable to communicate with the driver. The NIC is installed in the Registry, but the driver is not loaded. If you just installed the NIC, it will be in this state until the machine is restarted. If the NIC is in this state after the system is restarted, the driver might have not loaded or the Advanced Network Control Utility is unable to communicate with the driver. Unknown (in systems that support PCI Hot Plug) A teamed NIC is displayed as unknown if you have a system that supports PCI Hot Plug. Remove the teamed NIC. Unknown NICs On Windows 2000 systems, when you physically remove a NIC from the system, the NIC Registry entries are not removed until you uninstall the NIC using the Device Manager. Therefore, some removed NICs might display as unknown until they are removed through the Device Manager.
		Disabled NIC has been disabled through Device Manager or the network control panel tool.

If you see the cable fault icon next to the NIC, recheck the network connections and ensure that the switch or hub is working properly. After the connection is restored, you should no longer see this icon.

Note

Only teamed NICs display as unknown. If you power-off a teamed NIC, it becomes unknown. When you restart the NIC, its appropriate state is restored (active or nonactive).

22.5.4 Tuning the Network Subsystem

Numerous options are available when optimizing a network subsystem and network segments. These techniques are listed in the suggested order of priority.

22.5.4.1 ELIMINATING UNNECESSARY PROTOCOLS

Never use more protocols than needed. Unnecessary protocols place an additional load on the processor and memory subsystems. Using unnecessary protocols also puts an extra load on the network because browsing is performed separately for each protocol. Most networks require only the TCP/IP protocol.

22.5.4.2 USING PCI BUS MASTERING DEVICES

PCI devices provide the highest possible bandwidth, using direct memory access (DMA) burst modes. The PCI interface provides the highest possible throughput efficiency compared to processor utilization. This reduces processor involvement, saving processor bandwidth for applications.

22.5.4.3 SEGMENTING THE LAN USING SWITCHED NETWORK TECHNOLOGY

Network switches reduce the collision rate and increase bandwidth by allowing several nodes to communicate simultaneously through the switch. The traffic is directed to the ports only where the addressed servers are connected.

• Switched LAN Uses a switch between the server and workstations. This is the simplest, quickest, and most cost-effective method of network segmentation. It requires no server reconfiguration because the topology remains intact. However, broadcast traffic could be transferred to all ports.

• Router-segmented LAN Gives somewhat better bandwidth usage and reduces the impact of broadcast storms, which are error conditions resulting in a high broadcast rate that consumes a major part of the bandwidth. Router-segmented LANs can be reconfigured from an SNMP console using switches with virtual local area network (VLAN) capabilities.

• Emerging LAN technologies New features such as channel aggregation, fiber technology, wireless LAN, switching routers, and more.

22.5.4.4 ADJUSTING NIC SETTINGS

NIC drivers can yield slow or unreliable performance in a Windows environment if they are not adjusted properly. You can use the ProLiant Network Configuration Utility 7 to optimize these parameters for your server.

22.5.4.5 MULTIPLE NICS

Multiple NICs enable you to share the network load between separate networks. This can increase performance by splitting the network traffic between different segments. However, adding NICs does not improve performance if the current NIC is not close to saturation.

Note

If NetBIOS name support is enabled, the NICs must be connected to separate LANs to avoid name conflict.

22.5.4.6 DISABLING AUTODETECT FEATURES

Parameters that are set by the autodetect feature of the NIC card can be unreliable in a Windows 2000 environment. These issues are seen especially in a Windows 2000 Terminal Services environment. HP recommends that you use the ProLiant Network Configuration Utility 7 to hard code the settings for the following:

• Media type Unshielded twisted-pair (UTP) and shielded twisted-pair (STP) are twin copper wires twisted together and typically used for telephone or Ethernet networks. The difference between the types is the shielding used in the STP cable functions as a ground to limit electromagnetic interference. UTP cable is cheaper, but the shielding makes STP cable better suited for the demands of faster networks.

• Duplex setting Full-duplex Ethernet data is transferred in both directions simultaneously, doubling the available bandwidth over half-duplex Ethernet. If your network equipment supports full-duplex Ethernet, migrating your servers to full-duplex Ethernet can dramatically increase performance and reduce user response time.

• Speed Ethernet, Fast Ethernet, and Gigabit Ethernet move data at 10, 100, and 1000Mb/s respectively. Faster is better, but only if your network infrastructure can support it.

22.5.4.7 USING ADAPTIVE PERFORMANCE TUNING

The NIC sends interrupt signals to notify the server processor that a packet or group of packets is ready to be processed and to request processor cycles for that process. A low number of interrupts per second is not necessarily a good performance indicator nor does it automatically equate to higher processor utilization.

Too few interrupts leads to latencies that reduce throughput, and too many interrupts leads to high processor utilization. If the NIC sends too few interrupts, packets will form bottlenecks in the receive buffer and the system idles. The result is that latencies are introduced into the operation of the NIC, which in turn negatively affects server performance.

Tuning the hardware and software for optimal throughput is the key. Logically, the rate of interrupt signals should strike a balance where it does not hinder the processor or cause too many packets to overload the NIC.

Use the ProLiant Network Configuration Utility 7 to fine-tune the number of packets received before an interrupt is triggered. The Adaptive Performance Tuning option sets the number of frames the adapter receives before triggering an interrupt. Under normal operation, the adapter generates an interrupt each time a frame is received.

Receive Interrupts

Setting a low value for the receive interrupts parameter causes a high rate of receive interrupts. (That is, a setting of the lowest value will result in an interrupt for every frame received.) This increases NIC bandwidth, but might reduce processor efficiency, slowing your computer.

Setting a high value for this parameter reduces the interrupt rate (that is, increases the number of frames the adapter receives before generating an interrupt). This improves processor efficiency, but might reduce NIC bandwidth.

The default setting is 1536. The range is 0 to 4096 (in increments of 32).

After monitoring processor utilization, this parameter can be adjusted to decrease processor activity. The system should be thoroughly baselined before and after adjusting this setting to determine the effectiveness.

Receive Buffers

This parameter in the ProLiant Network Configuration Utility 7 specifies the number of buffers used by the driver when copying data to the protocol memory. In high network load situations, increasing receive buffers can increase performance. Increasing buffers, however, also increases the amount of system memory used by the driver. If too few receive buffers are used, performance suffers. If too many receive buffers are used, the driver unnecessarily consumes memory resources.

The default is 750. The range is 4 to 4000.

Note

The receive buffers parameter is available on Netelligent NICs only.

Receive Descriptors

The number of descriptors used by the driver when copying data to the protocol memory can be adjusted. In high network load situations, increasing receive descriptors can increase performance. The trade-off is that this also increases the amount of system memory used by the driver. If too few receive descriptors are used, performance suffers. If too many receive descriptors are used, the driver unnecessarily consumes memory resources.

The default is 48. The range is 8 to 1024 (in increments of 8).

22.5.4.8 TEAMING NICS

The design goal of HP NIC teaming is to provide fault tolerance and load balancing across a group of two or more NICs. The term team refers to the concept of multiple NICs working together as a single NIC, commonly referred to as a virtual network adapter. Multiple NICs are required to make a team, but like a single NIC, a team connects only to one network.

Before you configure teaming NICs, consider the following:

• In a load-balancing configuration, the NICs must run at the same speed.

• The NICs must use proper protocol configuration.

• Do not depend on a Dynamic Host Configuration Protocol (DHCP) server to assign an IP address.

• Do not depend on a static NIC configuration to ensure proper network function.

• Ensure that protocols for the Virtual Miniport, which represents the team, have been configured.

Note

To configure protocols for the Virtual Miniport, highlight the Virtual Miniport, select Properties, and edit the properties as needed.

Install the NIC teaming driver from the HP PSP for Windows 2003 by following the instructions provided with the PSP.

The most common reasons that NICs failover are physical link failures and heartbeat failures.

A physical link failure is anything that causes the link light on the back of the network adapter to go out, such as a pulled cable, switch power loss, or other such condition.

Note

A failed link LED could indicate a physical link failure when none actually exists.

ProLiant NICs also failover when they detect heartbeat failures, such as when heartbeat packets are not successfully transmitted and received from one network adapter to another via the network infrastructure. They are only transmitted to increment the receive counter on a NIC that has been idle to verify that the NIC still has network connectivity.

All ProLiant Ethernet NICs support the following three types of teaming:

• Network fault tolerance

• Transmit load balancing

• Switch-assisted load balancing

Teaming Capability	Network Fault Tolerance	Transmit Load Balancing	Switch-Assisted Load Balancing
Number of adapters supported per team	2 8	2 8	2 8
Fault tolerance	Yes	Yes	Yes
Transmit load balancing	No	Yes	Yes
Receive load balancing	No	No	Yes
Requires 802.3ad-compliant switch	No	No	Yes
Can connect a single team to more than one switch for switch redundancy (must be same broadcast domain)	Yes	Yes	Switch dependent
Uses heartbeat for network integrity checks	Yes	Yes	No
Can team NICs that do not support a common speed	Yes	No	No
Can team NICs operating at different speeds as long as the NICs support a common speed	Yes	Yes	Yes
Maximum theoretical transmit/receive throughput in Mb/s with maximum number of 100Mb/s NICs	100/100	800/100	800/800
Maximum theoretical transmit/receive throughput in Mb/s with maximum number of 1000Mb/s NICs	1000/1000	8000/1000	8000/8000
Load balances TCP/IP	No	Yes	Yes
Load balances protocols other than TCP/IP	No	No	Yes

Note

Switch-assisted load balancing was formerly referred to as Fast EtherChannel / Gigabit EtherChannel (FEC/GEC) teaming.

The following table lists teaming options that are supported by Windows 2000.

Teaming Options	Windows 2000 Protocols
Network fault tolerance	IP, NetBEUI, IPX (NCP), IPX (NetBIOS)
Transmit load balance	IP
Switch-assisted load balancing	IP, NetBEUI, IPX (NCP), IPX (NetBIOS)

22.5.4.9 NETWORK FAULT TOLERANCE

Network fault tolerance provides simple redundancy with two to eight NICs in a fault-tolerant team. Each server can support up to eight teams where one NIC per team is defined as the primary NIC. All other NICs are secondary. Network fault-tolerance teaming functions at any speed, on any media. It is switch-independent and can be split across OSI Layer 2 switches, but must be in the same Layer 2 domain.

Network fault tolerance is a simple, effective, and fail-safe approach to increase the reliability of server connections. It enables you to set up link recovery to the server NIC in case of a cable, port, or NIC failure. By creating a team, the network fault-tolerance approach enables you to maintain uninterrupted network connectivity.

During normal operation, the backup NIC has transmit disabled. If the link to the primary NIC fails, the link to the secondary (backup) NIC automatically takes over.

22.5.4.10 TRANSMIT LOAD BALANCING

Transmit load balancing, formerly known as adaptive load balancing, incorporates all the features of network fault tolerance and takes teaming one step further by adding load balancing of all outgoing server traffic. It is also switch-independent and can be split across Layer 2 switches; however, all transmit load balancing members must be in the same Layer 2 domain. Transmit load balancing increases a server's transmission throughput.

Example

A transmit load-balancing team containing four HP Fast Ethernet NICs, configured for full-duplex operation, can provide an aggregate maximum transmit rate of 400Mb/s and a 100Mb/s receive rate. In this example, total bandwidth is 500Mb/s.

With transmit load balancing, traffic received by the server is not load balanced. One NIC is primary, and as many as seven other NICs are secondary. The primary NIC transmits but is also responsible for receiving all traffic destined for the server. The secondary NICs only transmit data. The transmit load-balancing agent adjusts the data flow evenly between the multiple NICs. An algorithm that uses the last 1 or 2 bits of the source and destination MAC or IP addresses determines which port is used for a particular server-to-client communication flow.

Under transmit load balancing, all NICs in the team operate under the same IP address, but different Memory Access Controller (MAC) addresses. If one of the secondary connections fails, the server driver redirects the information flow from the failed connection to the remaining NIC team members. If the primary NIC in the team fails, one of the secondary NICs assumes the MAC address and the duties of the primary NIC.

22.5.4.11 SWITCH-ASSISTED LOAD BALANCING

Cisco Systems developed FEC technology based on the Institute of Electrical and Electronics Engineers (IEEE) 802.3 full-duplex East Ethernet specification. Switch-assisted load balancing incorporates all the features of network fault tolerance and transmit load balancing. If any of the NICs fail, the remaining NICs share the data load.

The operating system sees the multiple NICs as the same IP and MAC address and therefore as one NIC. The HP Intel-based Fast Ethernet NICs that support switch-assisted load balancing also support PCI Hot Plug technology, so that NIC servicing is possible during normal work hours without interrupting network services.

The differences between transmit load balancing and switch-assisted load balancing center around three design issues:

• All network connections transmit and receive data simultaneously, enabling a maximum transfer rate of up to 1600Mb/s.

• Switch-assisted load balancing is supported on IEEE 802.3ad-capable intelligent switches only.

• All ports must be connected to the same switch.

For more information on the IEEE 802.x standards, refer to http://www.ieee.org.

22.5.5 Gigabit Ethernet

Networks continue to require increasingly higher throughput to support the bandwidth requirements brought on by the demands of the following:

• The Internet

• Multimedia applications

• Large files

• Increased use of e-mail

• Faster server processors

Customers are also expecting increased reliability, availability, and scalability from their servers without comprising on manageability or flexibility.

Gigabit Ethernet is an evolving technology that supports ultra-high-speed connections along the backbone of Internet and intranet networks. It provides a raw data bandwidth of 1000Mb/s and includes both full- and half-duplexing modes. It can also play an important role in connecting certain types of high-demand servers to the network as a whole.

22.5.5.1 ADVANTAGES AND DISADVANTAGES OF GIGABIT ETHERNET

The biggest advantage of Gigabit Ethernet is the boost it gives to data transmission over existing Category 5 cabling. Performance increases up to ten times compared to Fast Ethernet (100Mb/s) solutions, which is especially critical for servers, workstations, and Pentium 4 desktop computers that are bottlenecked by aging 10/100 Ethernet networks. This congestion is the result of an increased use of network-oriented applications such as automatic file backup, e-mail, messaging, streaming business video, and network attached storage (NAS) applications.

In addition, Gigabit Ethernet is a natural outgrowth of the existing Ethernet base and can be considered an incremental technology rather than a complete replacement for the existing infrastructure.

Gigabit Ethernet can greatly improve the network subsystems, but it will also bottleneck other network and server devices if they are not upgraded as well. Gigabit Ethernet can deliver one and a half million small packets per second with an interpacket latency in nanoseconds. This stream of information will overwhelm slow routers that are connected to the network. Before deploying Gigabit Ethernet, you should understand and upgrade the Layer 3 packet forwarding devices on the network.

In the same manner that it has the potential to flood routers on the downstream side, Gigabit Ethernet can saturate the server on the upstream side. The data transfer from a Gigabit Ethernet NIC will generate so many interrupts that most servers will be crippled.

HP offers several Gigabit Ethernet products, including switches and NICs. All Gigabit Ethernet NICs offered by HP support a technology called interrupt coalescence that is designed to take the gigabit load off the server. NICs that support interrupt coalescence have the intelligence on-board to hold back interrupts during periods of high packet offloading. Essentially, the NIC generates one interrupt for a group of packets rather than creating an interrupt for each. When the network is not heavily loaded, the NIC resumes one-for-one, packet-to-interrupt processing.