Scenario 4: IP Multicast Issues in Frame Relay

Because multicast is a specific area of expertise, it is necessary that you first have a clear understanding of the terminology and underlying concepts of IP multicast. Internet Group Management Protocol (IGMP), Protocol Independent Multicast (PIM), dense mode, sparse mode, sparse-dense mode, rendezvous point (RP), Distance Vector Multicast Routing Protocol (DVMRP), Reverse Path Forwarding (RPF), shortest path tree (SPT) and their interactions are the key elements for troubleshooting multicast in Frame Relay. It is important to have a clear understanding of the group flags, and their role in the management of the process. The group flags include the following:

• D Dense

• S Sparse

• B Bidir Group

• C Connected

• L Local

• P Pruned

• R RP-bit set

• F Register flag

• T SPT-bit set

• J Join SPT

• M MSDP created entry

• X Proxy Join Timer Running

• A Advertised through Multicast Source Discovery Protocol (MSDP)

Recall that IP multicast, unlike unicast and broadcast, provides a scheme where a host sends packets to a subset of all hosts (group transmission), which are known as group members. Instead of listing all the IP addresses of the group members, all members are identified by a single multicast group address. Multicast packets are delivered to a group using best-effort reliability. In this environment, some members send multicast streams and others only listen, but while any host (whether a member of the group or not) can send to the group, only the members of a group receive the multicast streams. Membership in a multicast group is dynamic; hosts can join and leave at any time. A host can be a member of more than one multicast group at a time. For further information about multicast see Developing IP Multicast Networks, Volume I, by Beau Williamson (Cisco Press, 2000).

Some of the most common symptoms of existing problems with multicast are documented here. The remote user that's configured under Serial3/2:0.83 interface of the core router does not receive multicast traffic. The multicast software (such as Cisco IP/TV for example), regardless of the configuration efforts, does not report any incoming multicasts, or there is voice but no video.

The first action is to check if the SPT exists. Always start the troubleshooting process with the upstream neighbor of the remote user's router, and check if there is multicast traffic by checking the PIM neighbors using the command in Example 18-37. In this case, the upstream neighbor is the core router.

Example 18-37. Checking if There Is an Upstream PIM Neighbor Available

 7206-frame#show ip pim neighbor PIM Neighbor Table Neighbor          Interface                Uptime/Expires    Ver    DR Address                                                          Prio/Mode 161.70.192.10     FastEthernet0/0          00:04:20/00:01:24 v2    1 / B S 161.70.192.24     FastEthernet0/0          00:04:24/00:01:21 v2    N / 161.70.192.15     FastEthernet0/0          00:04:25/00:01:18 v2    1 / B S ! One of the PIM neighbor's IP is 161.70.192.15. ! The IP is learned through the F0/0 interface. ! The uptime of the neighbor is  04:25 minutes and the timer ! expires after 01:18 minutes. The PIM is version 2. 161.70.192.11     FastEthernet0/0          00:04:33/00:01:41 v2    1 / B S 161.70.192.1      FastEthernet0/0          00:04:33/00:01:41 v2     N / 161.70.192.16     FastEthernet0/0          00:04:35/00:01:40 v2    1 / B S 161.70.192.12     FastEthernet0/0          00:04:37/00:01:37 v2    1 / B S 

Checking the PIM neighbors is an important step to ensure that the upstream router or switch provides multicast for the core router. The availability of the downstream multicast can be checked by using the show ip mroute command, as shown in Example 18-38.

Example 18-38. Checking the Availability of the Downstream Multicast

 7206-frame#show ip mroute IP Multicast Routing Table Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,        L - Local, P - Pruned, R - RP-bit set, F - Register flag,        T - SPT-bit set, J - Join SPT, M - MSDP created entry,        X - Proxy Join Timer Running, A - Advertised via MSDP, U - URD,        I - Received Source Specific Host Report Outgoing interface flags: H - Hardware switched Timers: Uptime/Expires Interface state: Interface, Next-Hop or VCD, State/Mode <output omitted> ! The RP cannot be recognized  "RP 0.0.0.0" (*, 239.255.255.253), 09:39:29/00:02:27, RP 0.0.0.0, flags: S ! Sparse mode indicated RPF neighbor 0.0.0.0   Incoming interface: Null, RPF nbr 0.0.0.0   Outgoing interface list:     Serial3/2:0.83, Forward/Sparse-Dense, 09:39:29/00:02:27 <output omitted> (*, 224.0.1.39), 3d13h/00:02:00, RP 0.0.0.0, flags: D   Incoming interface: Null, RPF nbr 0.0.0.0   Outgoing interface list:     Serial4/1:0.37, Forward/Sparse-Dense, 3d13h/00:00:00     Serial3/2:0.83, Forward/Sparse-Dense, 09:40:01/00:00:00     Serial4/0:0.89, Forward/Sparse-Dense, 02:08:12/00:00:00     FastEthernet0/0, Forward/Sparse-Dense, 00:03:55/00:00:00     FastEthernet1/0, Forward/Sparse-Dense, 00:03:43/00:00:00 <output omitted> 

In the second part of the output, the flag S-Sparse indicates sparse mode and shows RPF neighbor 0.0.0.0. You can also see that the remote user connected through Serial3/2:0.83 issued a join request and, as a result, an outgoing entry is created in the multicast routing table. A similar situation is listed in the third group, where more interfaces (members) have joined, but the incoming interface still reports 0.0.0.0. It seems that the ip pim sparse-dense-mode command is missing in the upstream interfaces, which in this case, is a FastEthernet interface. After configuring the interfaces, take a look at the multicast routing table, as shown in Example 18-39.

Example 18-39. Now the FastEthernet 1/0 Interface Sees Its Upstream Neighbor

 7206-frame#show ip mroute IP Multicast Routing Table Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,        L - Local, P - Pruned, R - RP-bit set, F - Register flag,        T - SPT-bit set, J - Join SPT, M - MSDP created entry,        X - Proxy Join Timer Running, A - Advertised via MSDP, U - URD,        I - Received Source Specific Host Report Outgoing interface flags: H - Hardware switched Timers: Uptime/Expires Interface state: Interface, Next-Hop or VCD, State/Mode <output omitted> ! The FastEthernet 1/0 interface sees its upstream neighbor (*, 239.255.255.255), 4w3d/00:00:00, RP 10.68.10.13, flags: SJCL   Incoming interface: FastEthernet1/0, RPF nbr 161.70.192.33   Outgoing interface list:     FastEthernet0/0, Forward/Sparse-Dense, 00:25:50/00:00:00     Serial3/2:0.83, Forward/Sparse-Dense, 00:25:50/00:00:00     Serial4/0:0.89, Forward/Sparse-Dense, 00:25:51/00:00:00     Serial4/1:0.37, Forward/Sparse-Dense, 00:25:51/00:00:00 <output omitted> (192.168.165.15, 224.0.1.39), 00:00:51/00:02:08, flags: PTA   Incoming interface: FastEthernet1/0, RPF nbr 161.70.192.33   Outgoing interface list:     FastEthernet0/0, Prune/Sparse-Dense, 00:00:51/00:02:08     Serial3/2:0.83, Prune/Sparse-Dense, 00:00:55/00:02:07     Serial4/0:0.89, Prune/Sparse-Dense, 00:00:55/00:02:07     Serial4/1:0.37, Prune/Sparse-Dense, 00:00:55/00:02:06 <output omitted> 

The incoming interface reports a valid IP source, and the join messages from downstream members (including Serial3/2:0.83) are registered as an outgoing interface.

Next, check the remote user's router on the other end. The remote user's service is operational, but is experiencing trouble with multicast. The primary focus is on the remote user's configuration. If it is correct, the next action to take is to launch Cisco IP/TV (or any other multicast software), and check if the user issues a join request to the group (see Example 18-40).

Example 18-40. Check the IGMP Group Members

 1602-frame#show ip igmp group IGMP Connected Group Membership Group Address     Interface            Uptime     Expires    Last Reporter 239.193.0.10      Ethernet0            00:00:44   00:02:39   10.10.253.125 239.193.0.5       Ethernet0            00:00:45   00:02:35   10.10.253.125 224.0.1.40        Ethernet0            5w4d       never      10.10.253.121 1602-frame# 

The result is positive, so it is time to check if the core router has registered the join request, and identifies the downstream router as a neighbor (see Example 18-41).

Example 18-41. Check the IP PIM Neighbors of the Core Router

 7206-frame#show ip pim neighbor PIM Neighbor Table Neighbor          Interface                Uptime/Expires    Ver   DR Address                                                            Prio/Mode <output omitted> 161.70.192.47     FastEthernet1/0          16:33:17/00:01:20 v2    1 / B S 161.70.192.44     FastEthernet1/0          16:33:17/00:01:17 v2    1 / B S 161.70.192.43     FastEthernet1/0          16:33:17/00:01:16 v2    1 / B S 161.70.192.48     FastEthernet1/0          16:33:17/00:01:39 v2    1 / B S ! The core router sees the downstream neighbor Serial 3/2:0.83 10.10.253.121     Serial3/2:0.83           00:27:41/00:01:23 v2    N / 10.21.56.193      Serial4/0:0.89           16:33:15/00:01:43 v2    N / 161.70.194.209    Serial4/1:0.37           16:33:13/00:01:34 v2    N / 7206-frame# 

The result is positive, so now check how the remote user's router sees the upstream neighbors (see Example 18-42).

Example 18-42. Check if the Remote User's Router Sees the Upstream Neighbor, RP and RPF

 1602-frame#show ip mroute IP Multicast Routing Table Flags: D - Dense, S - Sparse, B - Bidir Group, C - Connected, L - Local,        P - Pruned, R - RP-bit set, F - Register flag, T - SPT-bit set,        J - Join SPT, M - MSDP created entry, X - Proxy Join Timer Running        A - Advertised via MSDP Outgoing interface flags: H - Hardware switched Timers: Uptime/Expires Interface state: Interface, Next-Hop or VCD, State/Mode (*, 239.193.0.10), 00:18:26/00:02:49, RP 10.68.10.13, flags: SJC ! There is a incoming interface, outgoing interface and a neighbor   Incoming interface: Serial1.83, RPF nbr 161.68.122.1   Outgoing interface list:     Ethernet0, Forward/Sparse-Dense, 00:03:36/00:02:49 (*, 239.193.0.5), 00:18:27/00:02:41, RP 10.68.10.13, flags: SJC   Incoming interface: Serial1.83, RPF nbr 161.68.122.1   Outgoing interface list:     Ethernet0, Forward/Sparse-Dense, 00:03:38/00:02:41 <output omitted> 1602-frame# 

Next, recall the RPF procedure. For traffic flowing down the source tree, the RPF check follows several steps. First, the router examines the source address of the arriving multicast packet to determine whether the packet arrived through an interface that is on the reverse path back to the source. If the check is successful, it forwards the packet; otherwise, it drops it. In the case of the PIM protocol, it typically uses the existing unicast routing table to determine the RPF check. The obvious next step is to compare the trace information between the two trace tools, which are available in the remote user's router.

A unicast trace uses the existing routing table, and the report is similar to that in Example 18-43.

The multicast trace report is different and it clearly reports the problem. The multicast stream is expected from 161.68.88.1, but the next hop is 161.68.86.70 (see Example 18-44).

Example 18-43. Using Unicast Trace for Multicast Troubleshooting

 1602-frame# trace 10.68.10.13 ! This address (10.68.10.13) points to the Rendezvous Point Type escape sequence to abort. Tracing the route to sj-mbone-loopback0.cisco.com (10.68.10.13)   1 7206-frame.cisco.com (161.68.88.1) 88 msec 84 msec 84 msec   2 access-gw1.cisco.com (161.70.192.1) 84 msec 88 msec 84 msec   3 bbi1-gw2.cisco.com (161.70.192.217) 84 msec 136 msec 128 msec <output omitted> -9  sj-mbone-loopback0.cisco.com (10.68.10.13) 1602-frame# 

Example 18-44. Using the Multicast Trace (mtrace Command) for Multicast Troubleshooting

 1602-frame#mtrace 10.68.10.13 Type escape sequence to abort. Mtrace from 10.68.10.13 to 10.10.253.121 via RPF From source (sj-mbone-loopback0.cisco.com) to destination (1602-frame.cisco.com) Querying full reverse path...  0  1602-frame.cisco.com (10.10.253.121) -1  1602-frame.cisco.com (10.10.253.121) PIM  [default] ! This !RPF! indication should not be missed. -2  161.68.86.70 PIM  !RPF!161.68.122.1 [default] -3  161.68.86.69 PIM  [default] <output omitted> -9  sj-mbone-loopback0.cisco.com (10.68.10.13) 1602-frame# 

The mtrace command output is reporting -2 161.68.86.70 PIM !RPF!161.68.122.1 [default], which indicates some issues with the RPF. RPF is based on the routing table information. Check the routing configuration again. The routing section is configured as shown in Example 18-45.

Example 18-45. The Configuration of the Static Route in the Remote User's Router

 1602-frame#show running-config <output omitted> ip classless ip route 0.0.0.0 0.0.0.0 161.68.122.1 ip route 10.10.253.120 255.255.255.248 Ethernet0 <output omitted> 

The mtrace command is based on the routing statement of the router that points to IP address 161.68.122.1, which is not on the reverse path, and consequently, the packets are discarded.

Finally, replace the routing statement and bound the next hop address in the static route with the outgoing interface, rather than with the IP address of the next hop. Another approach is to replace 161.68.122.1 with 161.68.88.1. After the configuration change, check the result with #show ip mroute, as shown in Example 18-46.

Example 18-46. Output from the show ip mroute Command Shows a New Neighbor

 1602-frame#show ip mroute IP Multicast Routing Table Flags: D - Dense, S - Sparse, B - Bidir Group, C - Connected, L - Local,        P - Pruned, R - RP-bit set, F - Register flag, T - SPT-bit set,        J - Join SPT, M - MSDP created entry, X - Proxy Join Timer Running        A - Advertised via MSDP Outgoing interface flags: H - Hardware switched Timers: Uptime/Expires Interface state: Interface, Next-Hop or VCD, State/Mode (*, 239.193.0.10), 00:30:56/00:02:14, RP 10.68.10.13, flags: SJC   Incoming interface: Serial1.83, RPF nbr 171.68.88.1   Outgoing interface list:     Ethernet0, Forward/Sparse-Dense, 00:16:06/00:02:14 (*, 239.193.0.5), 00:30:57/00:02:59, RP 10.68.10.13, flags: SJC   Incoming interface: Serial1.83, RPF nbr 171.68.88.1   Outgoing interface list:     Ethernet0, Forward/Sparse-Dense, 00:16:07/00:02:18 <output omitted> 

Consequently, the output of the mtrace command is as shown in Example 18-47.

Example 18-47. Output from the mtrace Command

 1602-frame#mtrace 10.68.10.13 Type escape sequence to abort. Mtrace from 10.68.10.13 to 10.10.253.121 via RPF From source (sj-mbone-loopback0.cisco.com) to destination (1602-frame.cisco.com) Querying full reverse path...  0  1602-frame.cisco.com (10.10.253.121) -1  1602-frame.cisco.com (10.10.253.121) PIM  [default] ! The next line shows that the upstream neighbor has changed. -2  7206-frame-1.cisco.com (171.68.88.1) PIM  [default] -3  161.70.192.33 PIM  [default] -4  161.70.192.221 PIM  [10.68.10.13/32] <output omitted> -10 sj-mbone-loopback0.cisco.com (10.68.10.13) 

NOTE

Another recommended command is 1602-frame#mstat 10.10.253.121 10.68.10.13.

The result after the configuration changes confirms that the multicast feature is on and the multicast applications are passing IP traffic.

The example provided is one of the possible scenarios, and it is designed to demonstrate the methodology for troubleshooting IP multicast in Frame Relay rather than including all possible issues and their solutions.