One of the most common issues on Cisco Catalyst switches is device connectivity. Device connectivity issues can be easy to discover (e.g., My PC can't connect to the network!); however, more commonly connectivity issues are less obvious, where connectivity is established between devices and switches with performance being impaired. These issues can be difficult to discover as they often rely on user perception. A power user might notice performance problems, while a normal user might not even notice performance problems because they rarely attempt to take advantage of the full speed of their LAN connection. Some users notice problems but just don't complain. Relying solely on user perception is simply not enough information to understand any device connectivity issues that might be present.
A good tactic to discover issues relating to performance problems is to go to the lunch room and pretend you are reading a magazine. Listen out for comments like "Gee, the network is slow today" or "I really need a new PC." Seriously, implementing some form of monitoring procedure (could be implementing a network management system or could just be the regular execution of a few show commands) for your LAN switches help you to discover issues proactively, reducing the impact to users of any issues you discover. In this scenario you learn how to look for errors on switch ports that indicate possible performance problems.
This scenario is designed to give you tips on how to monitor and troubleshoot basic device connectivity. This scenario is based on the topology of the previous scenario (Scenario 1-4) and assumes the configuration at the end of Scenario 1-4 is in place on each switch.
Successfully resolving device connectivity issues involves identifying a connectivity issue, troubleshooting the issue, and then resolving the issue. Identifying a device connectivity issue exists can happen in a number of ways. It might be that you've placed your switch on the network and configured an IP address, but you can't seem to get access to the rest of the network. It could be that you've connected a new server to your switch; however, your switch can't access rest of the network. Or it could be that a network management system has identified excessive errors on a particular port and generated an alarm.
Once you have identified a potential issue, the troubleshooting process begins. Whenever you are troubleshooting device connectivity issues, it is important to take a step-by-step layered approach, verifying connectivity exists at each layer of the network. Taking such an approach ensures you do not waste time troubleshooting an issue at a higher layer when there is a fundamental underlying issue at lower layers.
This section now shows you how to troubleshoot device connectivity issues by performing the following layered approach:
Verifying Layer 1 Connectivity
Any form of network connectivity relies on a physical (Layer 1) connection being established. A working physical connection relies on an electrical or optical circuit being established between two devices, with some form of clocking established (assuming the connection is synchronous). Once a physical connection is established, devices can then establish Layer 2 connectivity.
On Cisco Catalyst switches, a working Layer 1 connection is verified as follows:
Checking Network Cabling
Many physical layer connectivity problems are caused by faulty network cabling. All cabling between your switch and the rest of the network should be thoroughly tested prior to implementation using a professional cable tester. Ensure that you always use compatible cables for the protocol you are using.
A common source of cabling issue is the incorrect use of straight-thru and crossover RJ-45 cables. Any device-to-switch connection should use a straight-thru RJ-45 cable, while any device-to-device or switch-to-switch connection requires the use of a crossover RJ-45 cable. Figure 1-22 shows the cabling pinouts for a device-to-switch and switch-to-switch connection.
Figure 1-22. RJ-45 Cabling Pinouts
In Figure 1-22, for any Ethernet connection to work, the following wiring scheme must be used:
As you can see in Figure 1-22, for a device-to-switch connection, the switch port pinout is such that using a straight-thru cable ensures the wiring scheme above. For a switch-to-switch connection, however, if a straight-thru cable is used, Transmit + is incorrectly wired to Transmit + and so on, causing the connection to fail. A crossover cable ensures that when connecting like ports (i.e., switch port to switch port) the preceding wiring scheme is maintained. Of course, if a crossover cable is used for a device-to-switch connection, then Transmit + is incorrectly wired to Transmit + and so on, causing the connection to fail.
You can quickly distinguish a straight-thru cable from a crossover cable by lining up both ends of the cable and comparing the colors of the wires attached to the pin outs. If the wire positions are identical, then the cable is straight-thru; otherwise, the cable is a crossover cable.
All cabling should be well clear of possible sources of interference, such as power cords and air conditioning units. Always check that your cabling is properly connected. RJ-45 has a snap-in lever that ensures a proper physical, and you should avoid using RJ-45 connector where the snap-in lever has broken off.
Many cabling issues also arise when fiber cabling is being used. Make sure you understand the distance limitations of your fiber, and ensure that you are using the correct type of fiber cabling. Table 1-15 lists the distance limitations associated with the various fiber Gigabit Ethernet Interface Converters (GBICs) available for Cisco Catalyst switches.
An optical time domain reflector (OTDR) is a tool used by many cabling professionals to measure fiber distance and attenuation, locate fiber breaks, and measure any losses associated with splices or connectors. A good practice is to take a baseline measurement of attenuation and splice losses when fiber cabling is installed, which allows for future comparisons should a potential issue with the fiber cabling occur.
A common error with fiber cabling is to connect the transmit fiber on one end to the transmit fiber connector on the other end.
If you are certain that your cabling is installed and attached correctly, the next step is to move onto your devices and check physical layer status. With Ethernet, the physical layer status and Layer 2 status are tied together, because Ethernet is a connectionless protocol. For this reason, you'll examine the process of verifying both Layer 1 and 2 connectivity in the next section.
Checking Port LEDs
A simple way to check the current status of an Ethernet port is to physically look at the switch and check port LED state. Table 1-16 describes the various LED states for ports on Cisco Catalyst switches.
As you can see in Table 1-16, the LED state provides a quick indication as to the nature of a fault. If the LED is off, either the port is disabled (e.g., shutdown interface configuration command on Cisco IOS or set port disable command on CatOS), or if the port is enabled, no electrical/optical connectivity is present, meaning a physical cabling issue exists.
If the port is a solid amber, this means the switch operating system has disabled the port for some reason. Typical reasons include:
If the port is a flashing amber, this means the switch operating system has detected a hardware fault on the port (i.e., the port failed a hardware test during POST after bootup), or on the Catalyst 29xx/35xx, a flashing amber green LED indicates a link fault. A link fault refers to excessive errors on a port (e.g., excessive collisions, CRC errors, alignment errors, and jabbers).
Verifying Layer 2 Connectivity
Verifying Layer 2 connectivity involves checking switch ports and each connected device at an Ethernet level, ensuring Layer 2 connectivity has been established and also checking errors for a connection. Although all indications might be that a Layer 2 connection is operational, it is also good to actually confirm that Layer 2 traffic is being sent and received between two devices. Cisco Discovery Protocol (CDP) provides such functionality, verifying connectivity to locally connected Cisco devices. The following tasks related to verifying Layer 2 connectivity are now discussed:
Verifying a Layer 2 Connection is Established
On Cisco Catalyst switches, you can use the following commands to verify Layer 2 connectivity has been established:
Example 1-50 shows the output of the show interface command on Cisco IOS.
Example 1-50. The show interface command on Cisco IOS
Switch-A# show interface fastEthernet0/1 FastEthernet0/1 is up, line protocol is up (connected) Hardware is Fast Ethernet, address is 0009.b7aa.9c86 (bia 0009.b7aa.9c86) MTU 1546 bytes, BW 10000 Kbit, DLY 1000 usec, reliability 246/255, txload 4/255, rxload 4/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Half-duplex, 10Mb/s input flow-control is off, output flow-control is off ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:46, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue :0/40 (size/max) 5 minute input rate 160000 bits/sec, 1 packets/sec 5 minute output rate 160000 bits/sec, 1 packets/sec 5990 packets input, 8324144 bytes, 0 no buffer Received 10 broadcasts, 859 runts, 0 giants, 0 throttles 859 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 7 multicast, 0 pause input 0 input packets with dribble condition detected 6866 packets output, 8461183 bytes, 0 underruns 0 output errors, 479 collisions, 1 interface resets 0 babbles, 0 late collision, 70 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out...
In Example 1-50, the first shaded line provides an indication as to whether or not a Layer 2 connection is present. Notice that both the interface and line protocol are up, indicating Layer 1 and Layer 2 Ethernet is up. The next shaded line indicates the current speed and duplex settings. This is because Router-A has a 10BASE-T interface and is connected to interface fastEthernet0/1 on Switch-A (see Figure 1-11). Further down, you can see shaded portions of the output that indicate some errors on the port. Notice that there are 859 input errors, which have been exclusively caused by runts; a runt is frame received that is lower than the minimum 64 byte size permitted by Ethernet. Runts are normally caused by excessive collisions. Notice further that 479 collisions have been experienced. If you divide this figure by the highlighted total number of output frames (6866), this indicates a collision rate of approximately 7 percent, which is considered high.
Collisions are a normal occurrence on a half-duplex connection, but should never be experienced on a full-duplex connection. Collisions on a full-duplex port indicate a duplex mismatch, where one side of the connection is operating in half-duplex mode and the other in full-duplex mode. A high collision rate on half-duplex connections can indicate the link is overloaded or can be an indication of a faulty network card if the link is not loaded.
Example 1-51 shows the output of the show port command on CatOS.
Example 1-51. The show port command
Switch-B> (enable) show port 2/3 * = Configured MAC Address Port Name Status Vlan Level Duplex Speed Type ----- ------------------ ---------- ---------- ------ ------ ----- ------------ 2/3 connected 1 normal a-half a-10 10/100BaseTX Port AuxiliaryVlan AuxVlan-Status InlinePowered PowerAllocated Admin Oper Detected mWatt mA @51V ----- ------------- -------------- ----- ------ -------- ----- -------- 2/3 none none - - - - - Port Security Violation Shutdown-Time Age-Time Max-Addr Trap IfIndex ----- -------- --------- ------------- -------- -------- -------- ------- 2/3 disabled shutdown 0 0 1 enabled 10 Port Num-Addr Secure-Src-Addr Age-Left Last-Src-Addr Shutdown/Time-Left ----- -------- ----------------- -------- ----------------- ------------------ 2/3 0 - - - - - Port Flooding on Address Limit ----- ------------------------- 2/3 Enabled Port Status Channel Admin Ch Mode Group Id ----- ---------- -------------------- ----- ----- 2/3 connected auto silent 118 0 Port Status ErrDisable Reason Port ErrDisableTimeout Action on Timeout ---- ---------- ------------------- ---------------------- ----------------- 2/3 connected - Enable No Change Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize ----- ---------- ---------- ---------- ---------- --------- 2/3 - 1 2 1570 1403 Port Single-Col Multi-Coll Late-Coll Excess-Col Carri-Sen Runts Giants ----- ---------- ---------- ---------- ---------- --------- --------- --------- 2/3 309 355 1 1 0 164 1 Last-Time-Cleared -------------------------- Fri May 9 2003, 07:57:54
In Example 1-51, the Status column at the top indicates the port is currently in a connected state, indicating a Layer 2 Ethernet connection is up. Notice that the speed and duplex settings are also displayed. At the bottom of the output, you can see port error information.
Checking Connections for Layer 2 Errors
In the previous section you saw how the show interfaces and show port command can be used to view errors. On Cisco IOS, the show controllers ethernet-controller command provides more detailed information about port errors. Example 1-52 demonstrates the use of this command on Switch-A.
Example 1-52. The show controllers ethernet-controller command on Cisco IOS
Switch-A# show controllers ethernet-controller fastEthernet0/1 Transmit FastEthernet0/1 Receive 15648658 Bytes 15573710 Bytes 12148 Unicast frames 11102 Unicast frames 39 Multicast frames 1 Multicast frames 8 Broadcast frames 0 Broadcast frames 5 Discarded frames 7 No dest, unicast 0 Too old frames 0 No dest, multicast 18 Deferred frames 0 No dest, broadcast 1212 1 collision frames 151 2 collision frames 0 FCS errors 41 3 collision frames 0 Oversize frames 15 4 collision frames 1951 Undersize frames 4 5 collision frames 0 Collision fragments 4 6 collision frames 20 7 collision frames 8 Minimum size frames 5 8 collision frames 2 65 to 127 byte frames 1 9 collision frames 1009 128 to 255 byte frames 1 10 collision frames 1 256 to 511 byte frames 1 11 collision frames 0 512 to 1023 byte frames 0 12 collision frames 10090 1024 to 1518 byte frames 0 13 collision frames 0 14 collision frames 0 Flooded frames 0 15 collision frames 0 Overrun frames 0 Excessive collisions 0 VLAN filtered frames 0 Late collisions 0 Source routed frames 1212 Good (1 coll) frames 0 Valid oversize frames 243 Good(>1 coll) frames 0 Pause frames 0 Pause frames 0 Symbol error frames 0 VLAN discard frames 0 Invalid frames, too large 0 Excess defer frames 0 Valid frames, too large 0 Too large frames 0 Invalid frames, too small 50 64 byte frames 1951 Valid frames, too small 2 127 byte frames 0 255 byte frames 4 511 byte frames 1010 1023 byte frames 11129 1518 byte frames
In Example 1-52, notice that a lot of information is provided in relation to interface statistics when using the show controllers ethernet-controller command. Of particular interest in Example 1-52 is the collision statistics, which provide an indication as to not only the number of collisions, but also the number of retries required before a frame could be sent. For example, there are 20 * 7 collision frames, which means the switch attempted 7 retransmissions before the frame could be sent (i.e. 7 collisions occurred for the same frame). Notice that you can also view information as to the size distribution of framesin Example 1-52, you can see a high proportion of 1518 byte frames transmitted (11129 out of 12148), indicating a large data transfer has taken place or is in progress.
Using Cisco Discovery Protocol
An extremely useful feature of Cisco network devices to test point-to-point Layer 2 connectivity is called Cisco Discovery Protocol (CDP). CDP is used by many Cisco devices to verify Layer 2 connectivity over a variety of LAN and WAN media. On Ethernet networks, Cisco devices multicast CDP Hello packets periodically, which includes information about the device, including name, capabilities, software version, and Layer 3 parameters such as IP addressing. CDP messages are only propagated to locally connected Cisco deviceseach Cisco device that receives a CDP message reads the information in the message and then discards it, ensuring CDP messages to do not propagate further outwards.
CDP is discussed in more detail in Chapter 10, "Maintenance, Monitoring, and Troubleshooting."
CDP is enabled by default on Cisco network devices, and both CatOS and Cisco IOS enable you to view CDP neighbor information, which makes it an invaluable tool for troubleshooting. Both operating systems use the show cdp neighbors command to display a summary of the CDP neighbor table, as shown in Example 1-53.
Example 1-53. Viewing CDP Neighbors on Cisco IOS
Switch-A# show cdp neighbors Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone Device ID Local Intrfce Holdtme Capability Platform Port ID Router-A Fas 0/1 169 R 2610 Eth 0/0 JAB03375EKK(SwitcGig 0/1 162 T S WS-C4006 1/1
In Example 1-53, you can see that Switch-A has two directly connected neighbors indicated in the Device ID columnRouter-A and Switch-B (the serial number of Switch-B is displayed). The Local Intrfce column indicates with interface on the local switch each device is attached to, while the Capability column shows that Router-A is a router (R) and Switch-B is a switch (S) and transparent bridge (T). The Platform column indicates that Router-A is a 2610 router and that Switch-B is a Catalyst 4006, while the Port ID column indicates the port that the local switch is connected to on the remote device.
You can view more information about a CDP neighbor using the show cdp neighbors detail command. See Chapter 10 for more details.
Example 1-54 demonstrates the show cdp neighbors command on Switch-B.
Example 1-54. Viewing CDP Neighbors on CatOS
Switch-B> (enable) show cdp neighbors * - indicates vlan mismatch. # - indicates duplex mismatch. Port Device-ID Port-ID Platform -------- ------------------------------- ------------------------- ------------ 1/1 Switch-A GigabitEthernet0/1 cisco WS-C3550-24
In Example 1-54, notice that only a single neighbor (Switch-A) is shown, which is due to the fact that only Switch-A is directly connected to Switch-B (Server-A is as well; however, it is a non-Cisco device). You might expect Router-A to also be displayed as a neighbor, because Router-A is on the same VLAN as Switch-B; however, remember that CDP messages are propagated only to locally connected Cisco devices (whether they be switches, routers, or IP phones), at which point they are read and then discarded.
In Example 1-54, the output is abbreviated because the important line is the shaded line shown. The status field shows a value of connected, which indicates Layer 1 and Layer 2 connectivity to the switch. If Layer 1 and 2 are down, the status would show a value of not connected.
Verifying Layer 3 Connectivity
After verifying Layer 2 connectivity, you can next proceed to verify Layer 3 IP connectivity by using tools such as ping and traceroute. Refer to Chapter 10 for a discussion on Layer 3 connectivity tools.