MLS Using CEF

Cisco's switches support many types of MLS. However, Cisco's current crop of high-end switches, including the 3550, the 4000s, and the 6500, use CEF. A Layer 3 switching engine (sometimes referred to as the main processor) handles the control functions, such as building and maintaining the FIB, and pushes table information down to the line cards or ports, where data ASICs use this information to perform switching decisions in hardware. It's important to point out that CEF is used for hardware switching of unicast frames not broadcasts.

CEF separates switching into two components: control and data. Control components handle things such as building and maintaining the routing and FIB tables. Data components handle Layer 3 switching in hardware.

CEF Limitations

There are situations where switching decisions must be performed in software by the main processor. If your CEF switch sees any of the following traffic, the main processor is interrupted to handle it:

• IEEE 802.3 packets (for IP, make sure that all devices are using Ethernet II as a MAC-layer encapsulation)

• Nonsupported Layer 2 encapsulation types

• Packets that need fragmenting

• Packets destined to a tunnel interface

• IP header options enabled in the IP packet header

• An IP header in which the TTL field has expired

• Internet Group Management Protocol (IGMP) redirects

CEF Tables

CEF uses three tables to make its switching decisions: FIB, adjacency, and TCAM (commonly called CEF) tables. The FIB is built from the MLS switch's routing table and is sorted to optimize searches. The FIB table lookup for a destination is based on finding the longest matching prefix for the destination Layer 3 (IP) address. The FIB table is updated whenever one of the following three things occurs:

• The next-hop address for a routing entry changes

• A prefix changes for a routing entry

• ARP information changes for a next-hop address

• A route is no longer reachable

The adjacency table is built from the MLS switch's ARP table. This table contains Layer 2 information of neighbor's MAC addresses that will help the MLS switch rewrite Ethernet frames. The adjacency table is stored in double-data-rate DRAM. If the adjacency table becomes full, neighbors not listed in the adjacency table will have packets switched by the main processor whenever packets are sent to these neighbors (that is, they'll be software switched).

The CEF table contains IP destination prefixes that are sorted from the most specific to least specific to speed up searches. To provide for accurate tracking of statistics, the CEF table contains a separate entry for each adjacency. If the CEF table becomes full, a special entry, called a wildcard entry, is used to redirect switching decisions to the main processor (or ASIC), where switching occurs in software.

CEF Operation

The operation of CEF is similar to the process described earlier in the "MLS Implementation" and "Rewriting Frame and Packet Contents" sections. This section covers the operation of CEF as it relates to multilayer switching. Three basic steps occur during CEF's operation:

1. When a Layer 3 packet is received, find a match in the CEF (TCAM) table.

2. Based on the CEF entry, find the adjacent information that will be used to rewrite the frame.

3. The frame and packet are rewritten and forwarded to the next hop.

Of course, CEF's process is not as simple as the preceding three steps. Before any user frames are handled by CEF, the MLS switch first needs a MAC address that will represent itself when sending rewritten frames to a destination. The Layer 3 engine on the MLS switch assigns this MAC address from the chassis' MAC address range and this address is used by all VLANs remember that a MAC address has to be unique only in a broadcast domain (VLAN). Anytime frames are rewritten, the MLS switch will use this MAC address as the source MAC address in the frame.

Second, the MLS switch will install wildcard entries in its CEF table, which are for when a lookup occurs and connection information is not found. Basically, this tells the data ASICs that to switch the frame, the Layer 3 forwarding engine will have to handle the task (at a much slower rate).

Third, the Layer 3 forwarding engine will notify each interface that has been set up for CEF, as well as any CEF-specific features for that interface. Only interfaces enabled with CEF can have data ASICs (the ones on interfaces or line cards) perform the rewriting of frames. The MLS switch then sends the Layer 2 CAM table to the Layer 3 forwarding engine, which is used to build the CEF table.

Once traffic begins to cross VLAN boundaries, the MLS process begins. For each initial packet from a source to a specific destination, called a flow, the data ASICs must have the Layer 3 forwarding engine handle the switching of the frame. The Layer 3 forwarding engine will then populate the CEF and adjacency tables and forward the frame. At this point, any flow from the same source to the same destination can be rewritten by the data ASIC for the inbound port.

Load Balancing

MLS with CEF supports per-flow load balancing (sharing). Load balancing can be done on both an equal or unequal cost basis to a destination. For example, if your MLS switch's routing table has three paths to a destination, CEF can use all three paths in load balancing. CEF's FIB can contain up to six pointers to entries in the adjacency table for load balancing.

When load balancing, the MLS switch takes the source and destination IP addresses, as well as the transport layer source and destination port numbers, and runs them through a hash function. The result of this function is used to pick one of the multiple paths to the destination. As you can see from this function, this is more of a flow load balancing process. In other words, load balancing is not done on a packet-by-packet basis. Load sharing becomes more distributed as traffic from different sources and applications is sent to a single destination. Also, load sharing is automatically enabled when you configure IP routing on the Layer 3 forwarding engine.

CEF, by default, load balances across six paths to a destination. This load balancing is done on a connection-by-connection basis.

CEF Example

To illustrate this process in a little more depth, let's take a look at an example. I'll use the network shown in Figure 6.3. In this example:

1. PC-1 creates an IP packet destined to PC-2: 192.168.1.11 192.168.2.22. If PC-1 doesn't know the default gateway's MAC address, it ARPs for it and the MLS switch responds with its chosen MAC address. The PC then creates an Ethernet frame with its own MAC address as the source and the MLS switch's MAC address as the destination and forwards the frame.

2. The MLS switch receives the frame and begins processing it. Because this is the first time that PC-1 sent something to PC-2, the data ASIC on the inbound interface can't find an entry in the CEF table, so it interrupts the Layer 3 forwarding engine (L3FE) to process the inbound frame.

3. In step 3, the L3F3 examines its ARP table to see whether it knows about PC-2. If not, the L3FE ARPs for PC-2's MAC address, using its chassis address as the source. During this ARP process, the L3FE implements an ARP throttling policy. While waiting for the ARP response, if the L3FE receives any other packets to PC-2, it will not generate additional ARPs. This is used to prevent the L3FE from creating excessive ARPs and thereby possibly creating an ARP denial-of-service (DoS) attack. After the ARP response is received, the L3FE adds this information to its ARP table and creates an adjacency entry in its adjacency table. When the adjacency information is built, the L3FE uses the information in PC-1's frame and packet to create an entry in the CEF table that points to the newly created entry in the adjacency table. After the entry has been built, the L3FE rewrites the frame and packet with this information and forwards it to the destination.

4. In step 4, PC-2 receives PC-1's information. If PC-2 were to respond to PC-1, steps 2 and 3 would happen again. Remember that entries in the CEF table are unidirectional.

5. After the entry has been placed in the CEF table by the L3FE, any subsequent traffic from PC-1 to PC-2 would have the data ASIC directly rewrite PC-1's frame and packet information in hardware and forward it out the interface to PC-2. Note that in this situation, the L3FE isn't involved in the forwarding process.

Figure 6.3. CEF example.

CEF entries are unidirectional for communications between two devices, you'll have two entries. To repress unnecessary ARPs, the L3FE will generate only one ARP and wait for the response to that ARP.

CEF Configuration

One of the great features of configuring CEF is that the Catalyst switches that support it already assume that you'll be using it. Therefore, CEF is enabled by default. On the Catalyst 6500 with the Supervisor Engine II, CEF cannot be disabled if you have any of the following cards: Policy Feature Card 2 (PFC2), Multilayer Switch Feature Card 2 (MSFC2), or the Distributed Feature Card 2 (DFC2).

With the Catalyst 4000, you can disable CEF with the no ip cef command at Global Configuration mode this disables CEF on the entire switch. You can also use this command to disable CEF on an interface by first going into the interface and then executing this command. With the Catalyst 3550, you can disable CEF with the no ip route-cache cef command at Global Configuration mode this disables CEF on the entire switch. You can also use this command to disable CEF on an interface by first going into the interface and then executing this command.

CEF, by default, is enabled on the Catalyst 3550, 4000, and 6500 switches. You can disable CEF on the 4000 with the no ip cef command and disable it on the 3550 with the no ip route-cache cef command. You cannot disable it on the 6500.

CEF Verification

After you've enabled CEF, there are a handful of show commands that you can use to examine its operation. To display general statistics about Layer 3 traffic switched in hardware, use this command:

 Switch> show interfaces type slot_#/port_# | begin L3 

Here's an example of the output of this command:

 Switch> show interface fastethernet 3/1 | begin L3 L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 13 pkt, 760 bytes mcast L3 out Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes 4012302 packets input, 350170138 bytes, 0 no buffer Received 3385211 broadcasts, 2 runts, 0 giants, 0 throttles ...output omitted... 

To display CEF entries in the FIB table, use the show ip cef command. Here's an example of the use of this command:

 Switch> show ip cef ethernet 0/1 detail IP Distributed CEF with switching (Table Version 2338), flags=0x0   1380 routes, 0 reresolve, 0 unresolved (0 old, 0 new)   1380 leaves, 198 nodes, 370422 bytes, 2162 inserts, 942 invalidations   0 load sharing elements, 0 bytes, 0 references   universal per-destination load sharing algorithm, id 9B6C8123   2 CEF resets, 0 revisions of existing leaves   refcounts:  54376 leaf, 51514 node 192.168.2.2/32 version 1987, cached adjacency 192.168.2.2 0 packets,      0 bytes, adjacency-prefix      via 192.168.2.2 Ethernet0/1, 0 dependencies      next hop 192.168.2.2, Ethernet0/1 ...output omitted... 

The detail parameter lists all FIB information for all FIB entries.

To see the adjacency table, use the show adjacency command. These statistics are updated every 60 seconds. Here's an example of this command with the detail parameter:

 Switch> show adjacency detail Protocol  Interface        Address IP        FastEthernet3/3  192.168.2.2(3045)                            0 packets, 0 bytes                            000000000FF9200003                            00605C865B2800D0BB                            ARP 02:48:09 IP        FastEthernet3/3  192.168.2.3(11)                            0 packets, 0 bytes                            000000000FF9200003                            00801C93804000D0BB                            ARP 02:48:03 

In addition to listing the next-hop address for the adjacency, other types of adjacencies can appear, as shown in Table 6.2.

Table 6.2. Adjacency Types

Adjacency	Explanation
Discard	Drop the packet.
Drop	Check the prefix, but drop the packet.
Glean	For hosts directly connected to the RP, the subnet prefix is listed.
Null	Forward packets to the Null0 interface to filter packets (drop them).
Punt	Features are not supported in CEF and require the L3FE to process the packets.

CEF Troubleshooting

If you're experiencing problems with CEF, you can use debug and ping commands to troubleshoot the problem. Use this command to perform detailed troubleshooting of CEF:

 Switch# debug ip cef drops|receive|events|prefix-ipc|table|                         ipc|interface-ipc 

Table 6.3 explains the different parameters for this command.

Table 6.3. debug ip cef Options

Options	Explanation
drops	Displays dropped packets
receive	Displays packets that didn't use the FIB
events	Displays general CEF events
prefix-ipc	Displays updates to IP prefix information n the FIB
table	Displays processing on the FIB table, such as updates, flushing, and so on

Optionally, you can add an ACL to the debug command to limit the amount of output you see in your terminal session.

You can also use Cisco's extended ping command. This command is executed by itself at Privilege EXEC mode, and it prompts you for all the ICMP information for IP. One nice feature is that you can change the source IP address that will be used with the ping. This is normally the IP address of the exit interface of the IOS device, but you can change it to any IP address on the IOS device. This is useful for advanced testing of the reachability of a device.