Operating and Troubleshooting the Network

Now that all services are in effect operational at this point, it is time to take a look at the new RTPCom network through the eyes of those who will operate it. In truth, deploying the new services is rather easy; managing them is a different story. Then there is the important aspect of securing the infrastructure. New protocols and new services open new doors to possible attacks. These attacks could endanger both IPv4 and IPv6 infrastructures.

Securing the IPv6 Network

Until its expansion into IPv6 under the design presented so far, RTPCom was relatively protected from security threats. In the case of IPv4 service, RTPCom is not concerned at all with the customer traffic at layer 3, it is all encapsulated in PPP and L2TP. RTPCom has minimal provisioning responsibilities and user protection against attacks, if any is provided by the ISPs. In this context, RTPCom's infrastructure is not exposed to outside traffic and attacks.

The IPv6 service deployment changes completely the service model and with that it changes the security challenges. RTPCom is now switching the customer traffic at layer 3 throughout its network. This mode of operation exposes its infrastructure to attacks from the Internet as well as attacks sourced by its own customers. RTPCom is now responsible for customer provisioning and also for protecting him to a certain extent. The additional services, such as multicast, open the door to multiple types of threats. In this case, RTPCom has to implement a consistent and complete set of security policies that protects the IPv6 services as well as the IPv4 ones.

RTPCom's main concern is to secure its infrastructure. It will secure its access layer from attacks originated by its customers and it will secure its Edge from attacks originated by Internet hosts. Another area of concern is the data center in each L1 POP. It contains resources that are vital to the proper operation of various services and their management.

Securing the Access

RTPCom identified the following security concerns at the access layer:

• Attacks on its infrastructure (prefixes 2600:A000::/28 and 2600:A080::/28), attempts to access or impair network elements

• Spoofing attacks sourced by its customers using spoofed source addresses

• Link-local multicast attacks that would flood access routers with control traffic

• Multicast attacks where RTPCom's multicast enabled network is flooded with traffic sourced by its own customers

The important thing to remember is that in RTPCom's case, IPv4 facilitates IPv6 attacks sourced through its customers. IPv6 over IPv4 tunnels set up through the IPv4 service are invisible to RTPCom. Such tunnels can prove to be back doors to its IPv6 service for external hackers.

These four threats are addressed through the following policies and their respective implementations:

• Block all traffic destined to prefixes 2600:A000::/28 and 2600:A080::/28. This requirement was already built in to the two service ACLs used to implement unicast and Internet connectivity: Basic-Service and Internet-Service.

• To prevent spoofing attacks RTPCom enables uRPF on all customer facing interfaces (or subinterfaces) as shown in Chapter 9, "Securing IPv6 Networks."

• The link-local multicast is important to the proper operation of IPv6, so it cannot just be filtered out. A user could impair the access router by bombarding it with Router Solicitations, for example. The router can be protected by policing the link-local multicast traffic down to reasonable values.

• RTPCom is under no obligation to allow its customers to become sources of multicast traffic or to transport their multicast traffic for that matter. RTPCom could filter out customer traffic with multicast destinations. The ACL might turn out to be rather complex as it will have to permit link-local multicast as well as some of the well known control-plane multicast groups. A simpler way would be to disable PIM on all customer interfaces. The later approach does rely on the assumption that the CPE router does not need to use PIM to provide multicast service to hosts behind it. This is a reasonable assumption for the time being. Considering the limited resources of such devices, it is expected that they will more likely implement MLD proxy routing rather than PIM.

The implementation of these policies is shown in the configuration of NY-10-12-16 in Example 14-13.

Example 14-13. Relevant Security Configuration of Access Router (NY-10-12-16)

ipv6 access-list Access-mcast-DOS  permit icmp any any nd-ns  permit ipv6 any host FF02::2 ! class-map match-all Access-mcast-DOS-class  match access-group name Access-mcast-DOS ! policy-map prevent-mcast-attack  class Access-mcast-DOS-class  police 70000 1500 1500 conform-action transmit exceed-action drop ! interface ATM1/0.10 point-to-point  service-policy input prevent-mcast-attack  atm route-bridged ipv6  pvc 0/42   encapsulation aal5snap   service-policy output Parent-Basic   protocol pppoe ! ipv6 address 2600:A02A:30F0:A01::1/64  ipv6 enable  ipv6 verify unicast reverse-path  ipv6 traffic-filter Basic-Access in  ipv6 mld access-group CP3-Premium1  ipv6 nd other-config-flag  no ipv6 pim !

RTPCom's operations team is also concerned with attack vectors enabled by IPv6's extension headers. Routing headers are a particular concern. They intend to filter out packets with Type 0 (Source Route) routing header while the Type 2 (MIPv6) RH are allowed for Mobile IPv6 service. This security policy is implemented by adding the following line under all security ACLs defined:

deny ipv6 any any routing-type 0

RTPCom decided to delay defining policies that address the other extension header threats until those threats are better understood.

Securing the Edge

At the network edge, RTPCom peers with various ISPs. In their case, link-local attacks are less likely, even though they still are possible. On the other hand, attacks in a global scope are significant because of the exposure to the Internet. RTPCom identified the following threats in this part of its network:

• Attacks on its infrastructure network elements

• Multicast traffic flooding

• Link-local attacks in a smaller measure

These threats were already addressed while implementing various IPv6 services, so it is just a matter of adding the following final touches:

• While implementing the IPv6 Internet access service, RTPCom specifically blocked the infrastructure prefixes from being advertised outside its network. This policy will also be supported with inbound ACLs that will specifically block traffic destined to these prefixes.

• Considering the service model for its multicast service, RTPCom was able to safely disable it on the routers that provide access to the outside world. RTPCom can complement this measure with explicit filters applied inbound to its edge router interfaces.

• Despite being less likely, link-local attacks are still possible if the network of its institutional partner with whom it is sharing the link was compromised. As a measure of precaution, RTPCom can apply the policing methods used at the access layer.

The practical implementation of these policies is straightforward based on the examples already given for the access layer.

Securing the Data Center

A third area of security concern in RTPCom's network is the data center that hosts resources critical to its service offering. A first level of defense is provided by the perimeter security policies implemented at the access and edge layers. On the other hand, these resources require specific protection beyond layer 3, too.

To prevent attacks on the data center servers, RTPCom leverages PIX Firewalls upgraded to release 7.0 for IPv6 support. It also deploys the standard host protection policies on each of the servers. For more details on these mechanisms, see Chapter 9 of this book.

The multicast servers require particular attention because they can be accessed by the content providers. This is done over dedicated circuits to an access router connecting to server interfaces different than the ones used to transmit the multicast traffic. RTPCom has to treat the server as an unsecured resource. For this reason it will have to control the unicast traffic it receives from the servers. The same ACLs used at the access layer can be applied in this case.

RTPCom will have to continuously update its security policies based on the experiences learned operating IPv6. The dynamic character of these policies is also driven by the evolution of the existent IPv6 services and the emergence of new ones with their own security threats.

Managing the Network

Because most of the network elements are dual stack, RTPCom will continue to manage its network mostly over IPv4. The CiscoWorks NMS currently in use is upgraded to the latest release to leverage its IPv6 capabilities. NetFlow will be particularly used to monitor the traffic while Cisco IP SLA will be used for performance management. NetFlow captured data is sent to the collectors located in the data centers over IPv4. For topology mapping and IPv6 connectivity, the NOC team is testing Nagios. See Chapter 10, "IPv6 Network Management," for details on all these tools and their use.

Troubleshooting

After running the IPv4 for several years, RTPCom's network operations team defined clear processes for troubleshooting and resolving most common problems. IPv6's characteristics and the design of the services deployed require new troubleshooting methodologies that are significantly different from the ones used for the IPv4 services.

Provisioning

One of the rather new aspects of the IPv6 services is RTPCom's responsibility for customer provisioning. After a customer's line was configured based on the services it subscribed to and the physical interface is verified to be up, the first step in gaining access is to acquire the prefixes assigned. The investigation of the proper completion of this step should be at the top of the list when troubleshooting customer connectivity.

For customers that are assigned /64 prefixes, the operator can use the following troubleshooting steps:

• Verify the ND cache to see whether link-local control messages are exchanged with a subscriber's host. For NY-10-12-16 on the ATM1/0.10 interface where the customer is using stateless autoconfiguration:
  
  

  NY-10-12-16#show ipv6 neighbors IPv6 Address                             Age Link-layer Addr State Interface 2600:A02A:30F0:A01:20D:29FF:FEE1:4DC0    116 000d.29e1.4dc0  STALE ATM1/0.10 FE80::20D:29FF:FEE1:4DC0                 116 000d.29e1.4dc0  STALE ATM1/0.10

  
  The MAC address of the user is 000d.29E1.4DC0.
  
  
• The customer device is verified for the IPv6 autoconfiguration outcome.
  
  
• Turn on the following debug: debug ipv6 nd.
  
  

For customers that are assigned /56 prefixes via DHCP-PD, the operator can use the following troubleshooting steps:

• Verify the presence of the CPE in the access router ND cache.
  
  
• Verify the operation of DHCP-PD server on ATM1/0.100 interface of NY-10-12-16 that is provisioned for DHCP-PD:
  
  

  NY-10-12-16#show ipv6 dhcp interface ATM1/0.100 is in server mode   Using pool: DHCP-Customers   Preference value: 0   Hint from client: ignored   Rapid-Commit: disabled

  
  The DHCP-PD address pool is DHCP-Customers.
  
  
• Check the DHCP-PD bindings:
  
  

  NY-10-12-16#show ipv6 dhcp binding Client: FE80::20B:BFFF:FEAA:7400 (ATM1/0.100)   DUID: 00030001000BBFAA7400   Internet access PD: Internet access ID 0x000A0001, T1 302400, T2 483840     Prefix: 2600:A02A:30F0:6400::/56             preferred lifetime 604800, valid lifetime 2592000             expires at May 8 2005 03:05 AM (2591953 seconds)

  
  Prefix assigned via DHCP-PD: 2600:A02A:30F0:6400::/56.
  
  
• If there are no bindings, verify that the proper DUID was configured for this CPE: 00030001000BBFAA7400.
  
  
• Turn on the following debug: debug ipv6 dhcp detail.
  
  

Note

Debugs are a last resort troubleshooting tool on production routers because of their impact on the router's performance. It is also recommended that the debugging output is sent to the log rather than the console or terminal.

With the prefix assigned to the clients, the only thing left to verify is that they are reachable via ping from the access router.

Unicast Routing and Forwarding

Troubleshooting IPv6 unicast routing and forwarding is similar to troubleshooting IPv4. Each routing protocol can be verified independently. The BGP information on router NY-10-0-1 is shown in Example 14-14.

Example 14-14. IPv6 BGP Information on Router NY-10-0-1

NY-10-0-1#show bgp ipv6 summary BGP router identifier 2.10.0.1, local AS number 1600 BGP table version is 73, main routing table version 73 72 network entries using 9576 bytes of memory 72 path entries using 5184 bytes of memory 7 BGP path attribute entries using 392 bytes of memory 1 BGP AS-PATH entries using 24 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 15176 total bytes of memory BGP activity 73/0 prefixes, 73/0 paths, scan interval 60 secs Neighbor         V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd 2600:A002:A000:1100::1    4 1600   11852   11854      73    0   0 1w1d          71 2600:A002:A000:1200::1    4 1600   11852   11854      73    0   0 1w1d          71

The BGP information can be viewed separately for each address family, as shown in Example 14-15.

Example 14-15. IPv6 BGP Information for the Unicast Address Family on Router NY-10-0-1

NY-10-0-1#show bgp ipv6 unicast BGP table version is 73, local router ID is 2.10.0.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,               r RIB-failure, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete    Network          Next Hop            Metric LocPrf Weight Path *> 2600:A010::/28  2600:A002:A000:1100::1   3             0 201 ? *> 2600:A030::/28  2600:A002:A000:1100::1   3             0 201 ?

Similarly, the OSPFv3 process can be verified, as shown in Example 14-16.

Example 14-16. OSPFv3 Information on Router NY-10-0-1

NY-10-0-1#show ipv6 ospf  Routing Process "ospfv3 100" with ID 2.10.0.1  It is an autonomous system boundary router  Redistributing External Routes from,     bgp 1600  SPF schedule delay 5 secs, Hold time between two SPFs 10 secs  Minimum LSA interval 5 secs. Minimum LSA arrival 1 secs  LSA group pacing timer 240 secs  Interface flood pacing timer 33 msecs  Retransmission pacing timer 66 msecs  Number of external LSA 19392. Checksum Sum 0x2622F47C  Number of areas in this router is 1. 1 normal 0 stub 0 nssa     Area BACKBONE(0)         Number of interfaces in this area is 1         SPF algorithm executed 11 times         Number of LSA 27. Checksum Sum 0xF7C7F         Number of DCbitless LSA 0         Number of indication LSA 0         Number of DoNotAge LSA 0         Flood list length 0

Finally, the routing tables will indicate whether the network is operating properly from that perspective, as shown in Example 14-17.

Example 14-17. IPv6 Routing Table of Router NY-10-0-1

NY-10-0-1#show ipv6 route IPv6 Routing Table - 77 entries Codes: C - Connected, L - Local, S - Static, R - RIP, B - BGP        U - Per-user Static route        I1 - IS-IS L1, I2 - IS-IS L2, Internet access - IS-IS interarea, IS - IS-IS          summary        O - OSPF intra, OI - OSPF inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2 B   2600:A010::/28 [200/0]      via FE80::20F:34FF:FEB2:FA40 TenGigabitEthernet9/1      via FE80::20F:34FF:FEB2:F980 TenGigabitEthernet9/2

If routing was proven to be consistent with the design but connectivity was not reestablished, it is necessary to troubleshoot the forwarding path. Once again, the methodologies used are similar to the ones used in IPv4. Ping and traceroute for different destinations and from different sources will help narrow down the problem. If a problem router shows no routing issues, its routing information should be matched with the forwarding one. In the case of NY-10-0-1 and prefix 2600:A010::/28, the CEF information is as follows:

NY-10-0-1#show ipv6 cef 2600:A010::/28 detail 2600:A010::/28 RIBfib      nexthop FE80::20F:34FF:FEB2:FA40 TenGigabitEthernet9/1      nexthop FE80::20F:34FF:FEB2:F980 TenGigabitEthernet9/2

The IPv6 forwarding activity can be reviewed with the help of show ipv6 traffic.

Troubleshooting IPv6 unicast routing and forwarding should present no challenge for the experienced RTPCom NOC team. It can easily leverage the knowledge accumulated operation the IPv4 services.

Multicast Routing and Forwarding

Concept- and experience-wise, the multicast service is new to the RTPCom operations team. In preparation for this deployment, they went through technology training. Chapter 6 provides the technology details alongside a practical example of how troubleshooting is done.

Routers rely on the underlying unicast routing protocols to make multicast control or forwarding plane decisions. For this reason, the first step in troubleshooting multicast should the verification that the network is converged an operational from the unicast perspective. In the case of the L2 POP routers connecting to the backbone, it is worth specifically checking the prefixes advertised via BGP for IPv6 multicast (NY-10-0-1), as shown in Example 14-18.

Example 14-18. IPv6 BGP Information for the Multicast Address Family on Router NY-10-0-1

NY-10-0-1#show bgp ipv6 multicast BGP table version is 2, local router ID is 2.10.0.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,               r RIB-failure, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete    Network          Next Hop            Metric LocPrf Weight Path *> 2600:A002:00DD:1000::/64                     2600:A002:A000:1100::1 0             0 200 i

The multicast-specific troubleshooting should start with the verification of the proper operation of the involved protocols (MLD and PIM). The relevant information on router NY-10-12-16 is shown in Example 14-19.

Example 14-19. MLD Information on Access Router NY-10-12-16

NY-10-12-16#show ipv6 mld interface ATM1/0.10 is up, line protocol is up  Internet address is FE80::20E:FF:FE0E:E/10  MLD is enabled on interface  Current MLD version is 2  MLD query interval is 125 seconds  MLD querier timeout is 255 seconds  MLD max query response time is 10 seconds  Last member query response interval is 1 seconds  Inbound MLD access group is:   MLD activity: 7 joins, 0 leaves   MLD querying router is FE80::20E:FF:FE0E:E (this system) NY-10-12-16#show ipv6 pim interface Interface          PIM  Nbr   Hello  DR                         Count Intvl  Prior Gi6/1              on   1     30     1     Address: FE80::20F:35FF:FE3F:441A     DR     : FE80::20B:60FF:FEA7:D81A Gi6/2              on   1     30     1     Address: FE80::20F:35FF:FE3F:3616     DR     : FE80::20B:60FF:FEA6:E84A

The PIM activity can be monitored with the help of the command show ipv6 pim traffic. With the control protocol shown operational, it is time to move on to verify the multicast topology, as shown in Example 14-20.

Example 14-20. PIM Information on Access Router NY-10-12-16

NY-10-12-16#show ipv6 pim topology IP PIM Multicast Topology Table Entry state: (*/S,G)[RPT/SPT] Protocol Uptime Info Entry flags: KAT - Keep Alive Timer, AA - Assume Alive, PA - Probe Alive,     RA - Really Alive, LH - Last Hop, DSS - Do not Signal Sources,     RR - Register Received, SR - Sending Registers, E - MSDP External,     DCC - Do not Check Connected Interface state: Name, Uptime, Fwd, Info Interface flags: LI - Local Interest, LD - Local Dissinterest,     II - Internal Interest, ID - Internal Dissinterest,     LH - Last Hop, AS - Assert, AB - Admin Boundary (2600:A002:00DD:1000::101,FF3E:1::1) SSM SPT UP: 1w1d JP: Join(00:00:02) Flags: RPF: GigabitEthernet6/1,FE80::20B:60FF:FEA7:D81A   ATM1/0.10          1w1d      fwd Join(00:02:45)   Gi6/1              1w1d      off LI

This output reveals if the multicast group (2600:A002:00DD:1000::101,FF3E:1::1) is seen by NY-10-12-16 and if customers subscribed to it (interface ATM1/0.10). The forwarding information in Example 14-21 can further be compared with the control-plane one.

Example 14-21. Multicast Forwarding Information on Access Router NT-10-12-16

NY-10-12-16#show ipv6 mfib verbose IP Multicast Forwarding Information Base Entry Flags: C - Directly Connected, S - Signal, Internet access - Inherit A flag,              AR - Activity Required, D - Drop Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kbits per second Other counts: Total/RPF failed/Other drops Interface Flags: A - Accept, F - Forward, NS - Negate Signaling              IC - Internal Copy, NP - Not platform switched              SP - Signal Present Interface Counts: FS Pkt Count/PS Pkt Count (*,FF00::/8) Flags: C    Forwarding: 0/0/0/0, Other: 0/0/0 (*,FF30::/15) Flags: D    Forwarding: 0/0/0/0, Other: 0/0/0 (*,FF3E:1::/32) Flags: D    Forwarding: 0/0/0/0, Other: 0/0/0 (2600:A002:00DD:1000::101,FF3E:1::1) Flags:    Forwarding: 0/0/0/0, Other: 0/0/0    GigabitEthernet0/1 Flags: A    ATM1/0.10 Flags: F NS      Pkts: 0/0 MAC: 333300000001000F353F441A8100006786DD

This output provides the forwarding statistics, too. For more information on useful multicast troubleshooting commands, review Chapter 6 of this book.

Debugs are useful tools in troubleshooting the dynamic aspects of problems. However, they should be used with care because of their impact on the network equipment resources. Other tools, such as sniffers, can prove useful in providing the answer to what exactly is happening on a link.