15.6 Troubleshooting Routing Protocols with the traceoptions CommandUp until now, we have focused mainly on hardware troubleshooting ”chassis, interface, and component problems. You also need to be able to troubleshoot the routing protocols running on your router effectively. We will focus on three ways to do this ”through show commands, through debug commands, and through the routing table. You may be familiar with using show commands with other vendor products. You probably also have a good idea of how to troubleshoot routing problems through the use of the routing table (described in Chapter 14). You may not, however, be familiar with the Juniper Networks routers' traceoptions command set, which can be used to monitor protocol traffic. The traceoptions command set is similar to Cisco Systems' debug command set. Both command toolkits can be used to perform diagnostics related to protocol status and network activity. Both can be CPU- intensive on the router itself because of the power and memory it takes to examine and capture the volume of traffic and tasks flowing through and taking place on today's high-speed routers. Enabling any kind of traceoptions command on a Juniper Networks router can be detrimental to performance and should be used with caution. While the Cisco Systems' debugging commands are complex enough to warrant a separate command reference, the traceoptions commands are simple yet powerful. Knowing how and when to use traceoptions commands is a key skill in good troubleshooting. Each protocol section in this chapter will discuss how to use them; knowing when to use traceoptions commands is partially based on experience. If you find yourself in a troubleshooting scenario that requires some advanced techniques, try the traceoptions commands. Using the traceoptions command set is a good way to become familiar with how valuable it can be. Note A good way to use traceoptions is to set the commonly used traceoptions commands and parameters ahead of time in the configuration and leave them disabled. When you need to run tracing, you simply have to go into your configuration and enable the trace commands needed. This is a great timesaver. 15.6.1 Viewing traceoptions OutputThis section will discuss protocol-specific information and how to get it later. Here it specifies what you need to know to retrieve the output gathered by the traceoptions command. There are two ways to see this information: by monitoring it in real-time or by viewing it from a saved file with the show log command. The monitor command allows you to display the end of either a system log file or a trace file, along with additional entries as they are being added. It can be a very valuable tool during active troubleshooting because you can watch the output and receive immediate feedback on what is happening in the router or network. The syntax of the monitor command is as follows : lab@Chicago> monitor (start stop) list The list qualifier prompts the system to provide you with a list of log files from which to select. This list will include not only traceoptions output files, but syslog-produced files as well. To stop the output, enter <ESC>-Q. The show log command can be used to list log files on the system, view the contents of the log files, or list user logins. The syntax of this command is as follows: lab@Chicago> show log [user username ] [ filename ] The user qualifier is optional, as is username . If you enter user and do not enter a username , you will see logged information about all users with recent logins. The filename is also optional. If you identify a filename, you will view the contents of that file. The use of no optional qualifiers simply provides a list of all available log files that you can view. Here is an example of the show log output: lab@Chicago> show log user lab ttyd0 Thu Apr 4 13:17 still logged in ted ttyd0 Tue Apr 2 11:24 13:13 (2+01:49) In this example, we see that on Chicago, two users are logged on. We see our session, user lab , and another session, user ted . User ted has been online for 2 days, 1 hour , and 49 minutes. Notice that, because we are still logged in, we do not see an accounting of the amount of time we have been logged in. 15.6.2 Enabling Global traceoptionsFor capturing traffic and activity on the Juniper Networks router, you can enable some global parameters for traceoptions , before editing specific parameters for protocols or interfaces. You would do this in the [edit routing-options] configuration mode as shown below: [edit routing-options] lab@Chicago# edit traceoptions file filename <replace> <size size > <files number > <no-stamp> <(world-readable no-world-readable)> flag flag <flag- modifier> <disable> Notice that you can specify a particular filename for the trace, such as /var/log/trace-all . Enabling the options at this level is optional, but can be beneficial if you want all routing options tracing to go into a single output file. Table 15-6 describes the parameters you can use with the traceoptions command. This table applies to these parameters regardless of the configuration hierarchy level you are at. Table 15-6. Configuration Parameters for the traceoptions Command
Table 15-7 lists the required and optional flag options you want to trace. Table 15-7. Required and Optional flag parameters for the traceoptions Command
An example of the syntax for a traceoptions command at this level is as follows: lab@Chicago# edit traceoptions file trace-all replace size 1m files 4 world-readable flag receive In this example, we have enabled traceoptions at the [edit routing-options] hierarchy. We are directing all output to a file called trace-all , which will replace any file already in existence with that name. We have set the maximum file size at 1MB, limiting the total number of file versions with this name to 4 . We are allowing all users to read this file and are looking at received traffic only. Now that you have traceoptions enabled globally, let's look at the ways you can use traceoptions with each routing protocol. 15.6.3 Using traceoptions with RIPProbably the least complex routing protocol you will run in the core is RIP, usually version 2, although Juniper Networks routers also support version 1. Section 15.6.2 described how to set up traceoptions globally. If you prefer, you may specify certain parameters that you want to see on your RIP protocol traffic at the [edit protocols rip] hierarchy. Bear in mind that the more specific you get, the easier it will be to use the file for troubleshooting purposes. More specific logging also takes up less space on the router's hard disk and is less CPU-intensive. Here is how you would enable traceoptions for RIP protocol traffic in configuration mode: lab@Chicago# edit protocols rip [edit protocols rip] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set traceoptions flag flag < flag-modifier > <disable> The RIP protocol-specific flags can help narrow down the output that you receive. Table 15-8 lists each flag and provides a short description. The following example shows how to enable the trigger , request , and update flags for RIP: [edit protocols rip] lab@Chicago# set traceoptions file rip-trace size 400k files 4 no-stamp world-readable lab@Chicago# set flag request update trigger Table 15-8. RIP traceoptions Flags
The following example shows how to disable the trigger flag for the above RIP traceoptions setting: [edit protocols rip] lab@Chicago# set traceoptions lab@Chicago# set flag trigger disable 15.6.4 Using traceoptions with OSPFMost large networks are running OSPF in the core. Using the traceoptions command, you can specify certain types of OSPF packets you want to capture. To do this, follow the instructions below to set up specific OSPF parameters within the [edit protocols ospf] hierarchy: lab@Chicago# edit protocols ospf [edit protocols ospf] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set traceoptions flag flag < flag-modifier > <disable> Flags allow you to set up some OSPF-specific information that you may want to see. Table 15-9 shows flags that are available for OSPF traceoptions . Alternately, you can enter a partial command for a list of the possible completions you can use as follows: lab@chicago# set flag <enter> Will give you a list of the available flags. Table 15-9. OSPF traceoptions Flags
The following example shows how to run traceoptions in OSPF to capture all OSPF state transitions, as well as all hello packets. The output is being logged to a file called ospf-trace . [edit protocols ospf] lab@Chicago# set traceoptions file ospf-trace lab@Chicago# set traceoptions flag event hello Although some OSPF show commands were discussed in Chapter 8, we wanted to include all of the possible commands here, so that you can use them in your troubleshooting process. To gather information about OSPF databases, processes and routes, you can use show commands. These commands can be used to get a current status of specific OSPF components at a given moment in time. Table 15-10 provides a brief description of each command. Table 15-10. OSPF show Commands
In this example, we are running a show ospf neighbor command on Chicago. We can see three neighbors, the interface through which we know each neighbor, the state, and the neighbor's router ID. We also see the value of the dead timer and that router's priority value. For more information on the meaning of these OSPF values, please refer to Chapter 8. [edit protocols] lab@Chicago# run show ospf neighbor Address Interface State ID Pri Dead 10.10.0.130 ae1.0 Full 10.10.1.1 128 38 10.10.0.138 at-1/2/0.235 Full 10.10.0.2 128 36 10.10.0.134 at-1/2/1.167 Full 10.10.0.3 128 38 Note that you can use the clear ospf command with the database , neighbors , or statistics qualifiers to clear out some or all of the data in the OSPF databases and routing table. This comes in very handy when troubleshooting, especially when you make a change and need to see the results right away. 15.6.5 Using traceoptions with IS-ISYou can also use the traceoptions command to monitor the behavior of the IS-IS routing protocol. The following example uses the [edit protocols isis] hierarchy: lab@Chicago# edit protocols isis [edit protocols isis] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set traceoptions flag flag < flag-modifier > <disable> Flags allow you to set up some IS-IS-specific information that you may want to see. Table 15-11 lists flags that are available for traceoptions . To see a list of all possible flags available from the CLI, hit enter after typing set flag to bring up a list of possible completions. In the following example, you can see that we are going to trace all IS-IS protocol traffic that is being sent. We are logging all traceoptions output to a file called isis-trace . [edit protocols isis] lab@Chicago# set traceoptions file isis-trace lab@Chicago# set traceoptions flag all send In addition to using the traceoptions command for monitoring IS-IS, you can also use the show isis command. While this command can be used alone, you can also add six different qualifiers to gather even more specific information. Table 15-12 lists these qualifiers. Table 15-11. IS-IS traceoptions Flags
Table 15-12. show isis Command Qualifiers
In this example, we use the show isis route command to view the IS-IS routing table. Notice that it gives us the network prefix, the version of JUNOS running on the router, metrics for the route, and the associated interface through which we connect to this network. [edit protocols] lab@Chicago# run show isis route IS-IS routing table Current version: L1: 38 L2: 55 Prefix L Version Metric Type Interface Via 10.10.0.2/32 2 55 10 int at-1/2/0.235 Montreal 10.10.0.3/32 2 55 10 int at-1/2/1.167 SanJose 10.10.0.4/32 2 55 20 int at-1/2/0.235 Montreal at-1/2/1.167 SanJose 10.10.0.160/30 2 55 20 int at-1/2/0.235 Montreal 10.10.0.164/30 2 55 20 int at-1/2/1.167 SanJose at-1/2/0.235 Montreal 10.10.0.192/30 2 55 20 int at-1/2/1.167 SanJose 10.10.0.224/30 2 55 30 int at-1/2/0.235 Montreal at-1/2/1.167 SanJose 10.10.0.228/30 2 55 30 int at-1/2/0.235 Montreal at-1/2/1.167 SanJose 10.10.2.132/30 2 55 20 int at-1/2/0.235 Montreal 17.185.36.224/30 2 55 20 int at-1/2/1.167 SanJose 192.168.18.4/30 2 55 20 int at-1/2/1.167 SanJose Note that you can use the clear isis command and the same qualifiers to clear out some or all of the data in the IS-IS databases and routing table. This comes in very handy when troubleshooting, especially when you make a change and want to watch the counters and so on. 15.6.6 Using traceoptions with BGPTo enable the monitoring of the BGP on the router, set up BGP-specific flags within the [edit protocols bgp] hierarchy as follows: lab@Chicago# edit protocols bgp [edit protocols bgp] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set traceoptions flag flag < flag-modifier > <disable> Notice that you may specify an output file of a name of your choosing. You may also instruct the router to replace the current output file, use a specified maximum size for the file, use a certain number of files before overwriting the first one, and indicate that no timestamp should be used. Flags allow you to set up some BGP-specific information that you may want to see. Table 15-13 lists flags that are available for traceoptions . To see a list of all possible flags available from the CLI, hit enter after typing set flag to bring up a list of possible completions. Table 15-13. BGP traceoptions Flags
In the following example, you can see that we are going to trace all BGP protocol traffic that is being received. We are logging all traceoptions output to a file called bgp-trace . [edit protocols bgp] lab@Chicago# set traceoptions file bgp-trace lab@Chicago# set traceoptions flag all receive In addition to using the traceoptions command for monitoring BGP, you can use the show bgp command. While this command can be used alone, you can also add four different qualifiers to gather even more specific information. Table 15-14 lists these qualifiers. Table 15-14. show bgp Command Qualifiers
Notice that in the following example, we provide you with a show bgp summary from Chicago. It gives us a look at the number of groups and peers. It also provides us with a little information about the peering sessions between routers, such as up time and state. [edit] lab@Chicago# run show bgp summary Groups: 3 Peers: 4 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 0 0 0 0 0 0 inet.2 0 0 0 0 0 0 bgp.l3vpn.0 18 18 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State#Active/Received/Damped... 10.10.0.2 7500 522 527 0 2 4:20:26 Establ bgp.l3vpn.0: 0/0/0 10.10.0.3 7500 530 533 0 2 4:23:18 Establ bgp.l3vpn.0: 9/9/0 10.10.0.4 7500 583 584 0 0 4:50:47 Establ inet.0: 0/0/0 10.10.0.130 3000 583 592 0 0 4:49:09 Establ inet.0: 0/0/0 inet.2: 0/0/0 bgp.l3vpn.0: 9/9/0 Note that you can use the clear bgp command and the same qualifiers to clear out some or all of the data or counters in the BGP routing table and processes. This comes in very handy when troubleshooting, especially when you make a change and want to watch the counters and so on. 15.6.7 Troubleshooting Commands for MPLS and VPNsChapter 12 on MPLS has a lot of background and configuration information for this popular new style of routing over Layer 2 virtual private networks (L2VPNs) using MPLS to carry the connection. Juniper Networks routers support traceoptions functionality for advanced MPLS and VPN troubleshooting. As with other protocols, you must first enable traceoptions globally before you set up the tracing of MPLS tunnels. Once global traceoptions are enabled, you can enable certain traceoptions parameters for MPLS as shown below: lab@Chicago# edit protocols mpls [edit protocols mpls] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set flag flag < flag-modifier > <disable> Again, by setting certain flags, you can selectively enable different elements to trace within the MPLS protocol. Table 15-15 lists the flags that are available for traceoptions . To see a list of all possible flags available from the CLI, hit enter after typing set flag to bring up a list of possible completions. Table 15-15. MPLS traceoptions Flags
You can also trace information about traffic on an L2VPN by using traceoptions . To do so, use the following syntax: lab@Chicago# edit routing-instances routing-instance-name protocols l2vpn [edit routing-instances routing-instance-name protocols l2vpn] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set traceoptions flag flag < flag-modifier > <disable> Flags that you can use for tracing L2VPN information are shown in Table 15-16. Table 15-16. L2VPN traceoptions Flags
While traceoptions are important, the additional load they put on the router can sometimes be more than you are willing to use to troubleshoot a problem. This is where show and clear commands come in handy. They are simple tools that can be used to monitor counters, link information, and LSP information in an MPLS or L2VPN environment. Some of the commands you can use when you want to keep it simple are shown in Table 15-17. If you want to see a list quickly of all possible commands to use with show or clear , you can press enter after typing in show l2vpn or clear l2vpn . The system will then display a list of possible completions. Table 15-17. MPLS and L2VPN show and clear Commands
The following is an example of the show mpls lsp command in use. It shows both ingress (incoming) and egress (outgoing) LSPs, their states, path information, and names . It also shows for the egress LSP, the LabelIn / LabelOut information and summary information at the end. [edit] instruct@Denver# run show mpls lsp Ingress LSP: 2 sessions To From State Rt ActivePath P LSPname 10.10.0.2 10.10.0.1 Up 0 * vpn-dev-mont 10.10.0.3 10.10.0.1 Up 0 * uunet-denv-sj Total 2 displayed, Up 2, Down 0 Egress LSP: 2 sessions To From State Rt Style Labelin Labelout LSPname 10.10.0.1 10.10.0.2 Up 0 1 FF 3 - vpn-mont-denv 10.10.0.1 192.168.161.19 Up 0 1 FF 3 - uunet-sj-denv Total 2 displayed, Up 2, Down 0 Transit LSP: 0 sessions Total 0 displayed, Up 0, Down 0 The next example shows the show mpls interface command. With this command, we can get a quick summary of all of our MPLS-enabled interfaces, along with their states and the administrative groups to which they belong. [edit] lab@Chicago# run show mpls interface Interface State Administrative groups at-1/2/0.235 Up <none> at-1/2/1.167 Up <none> fe-1/0/1.0 Up <none> fe-1/0/0.0 Up <none> ae1.0 Up <none> 15.6.8 Troubleshooting Commands for Multicast ProtocolsThere are several different multicast protocols supported by the Juniper Networks routers. In this section, we will discuss commands you can use to troubleshoot each of them. As with other protocols, the traceoptions commands are invaluable in advanced troubleshooting. You can enable specific multicast protocol traceoptions within the [edit routing-options] hierarchy, then set up protocol-specific flags within the [edit protocols multicast protocol] hierarchy. Each of these is discussed below. 15.6.8.1 DVMRPThis section addresses the commands used to turn on the traceoptions capabilities in DVMRP. With an output filename of your choosing, enable traceoptions by using the commands shown below. Then, set up specific output parameters with DVMRP-specific flags. lab@Chicago# edit protocols dvmrp [edit protocols dvmrp] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set traceoptions flag flag < flag-modifier > <disable> Flags allow you to gather output for some DVMRP-specific information that you may want to see. Table 15-18 lists flags that are available for DVMRP traceoptions . Table 15-18. DVMRP traceoptions Flags
The following example sets up a trace for DVMRP prune messages with detailed logging. All traceoptions output is being sent to a file called dvmrp-trace . [edit protocols dvmrp] lab@Chicago# set traceoptions file dvmrp-trace lab@Chicago# set traceoptions flag prune detail Additionally, you can use certain show commands in conjunction with DVMRP for real-time troubleshooting without taking up valuable disk space. Table 15-19 lists these commands. Table 15-19. DVMRP show Commands
15.6.8.2 MSDPAs with DVMRP, you can enable tracing for MSDP as follows: lab@Chicago# edit protocols msdp [edit protocols msdp] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set traceoptions flag flag < flag-modifier > <disable> In addition to enabling the tracing for MSDP globally, you can specify tracing for all peers in a group using the [edit protocols msdp group group-name] hierarchy, or for an individual peer using the [edit protocols msdp group group-name peer-address] hierarchy. Setting traceoptions for a group will override anything you set for a peer within that group. You can also add MSDP-specific flags that allow you to gather configurable output from the protocol. The flags used for MSDP are listed in Table 15-20. The following example shows how to trace all MSDP sa packets. This example logs all traceoptions output to a file called msdp-trace . [edit protocols msdp] lab@Chicago# set traceoptions file msdp-trace lab@Chicago# set traceoptions flag sa Table 15-20. MSDP traceoptions Flags
As you have seen with other protocols, show commands can be helpful in determining statistical information, current status, and cache entries. Table 15-21 lists the commands that are useful in troubleshooting MSDP: Table 15-21. MSDP Troubleshooting Commands
15.6.8.3 SAP/SDPIf your network is running videoteleconferencing and other multimedia conferences and presentations, you may use SAP/SDP in your network. There are no traceoptions capabilities for SAP/SDP, but you can take a look at the addresses to which the router is listening for multicast announcements by using the following command: User1@NewYork> show sap listen [brief detail] The output of this command will show you both the address and the port to which the router is listening. 15.6.8.4 IGMPIGMP is one of the oldest multicast protocols. It uses join and leave messages to indicate members entering or leaving the multicast group. In that respect, it is like a subscription-based service, sending information only to the members wishing to receive it. Tracing IGMP is similar to that in other protocols. lab@Chicago# edit protocols igmp [edit protocols igmp] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set traceoptions flag flag < flag-modifier > <disable> IGMP-specific flags for the traceoptions command are provided in Table 15-22. To see a complete list of flags available, press enter after typing set flag , and you will see a list of possible completions for the command. Table 15-22. IGMP traceoptions Flags
In the following example, you can see that we are going to trace all IGMP leave packets being sent. All traceoptions output is logged to a file called igmp-trace . [edit protocols igmp] lab@Chicago# set traceoptions file igmp-trace lab@Chicago# set traceoptions flag leave send Table 15-23 lists other helpful commands you can use to troubleshoot IGMP problems. Table 15-23. IGMP Troubleshooting Commands
15.6.8.5 PIMPIM is a popular multicast routing protocol because it is not dependent on a particular unicast protocol, such as IP. Whether you are running PIM-SM or PIM-DM, you can use the following commands to enable tracing: lab@Chicago# edit protocols pim [edit protocols pim] lab@Chicago# set traceoptions file filename <replace> <size size > <files number > <no-stamp> lab@Chicago# set traceoptions flag flag < flag-modifier > <disable> Flags that can be used when tracing PIM output are shown in Table 15-24, along with a brief description of each. Table 15-24. PIM traceoptions Flags
Here is an example of how to setup traceoptions for capturing all join packets received by a PIM-enabled router. All traceoptions output is logged to a file called pim-trace . [edit protocols pim] lab@Chicago# set traceoptions file pim-trace lab@Chicago# set traceoptions flag join receive Table 15-25 lists additional commands that can be helpful in narrowing down a problem within PIM. Table 15-25. PIM Troubleshooting Commands
|