The Carriers Carrier Architecture

In Chapter 4, the customers of MPLS VPN service providers were considered to be enterprises, but it is also possible that the customer of a service provider might itself be a service provider. This might be the case if, for example, a smaller, customer service provider wanted to take advantage of a larger service provider's MPLS VPN backbone to connect its geographically dispersed points of presence (PoPs). The customer service provider might or might not in turn be offering MPLS Layer 3 VPN service to its own customers.

The backbone service provider that offers an MPLS VPN service to customer service providers is known as a carriers' carrier (CSC). The service provider that is a customer of the CSC is referred to in this section as a CSC customer. Figure 5-1 illustrates a carriers' carrier architecture.

Figure 5-1. CSC Architecture

In Figure 5-1, SP1 is the CSC and offers MPLS VPN connectivity to its customer, SP2. SP2 takes advantage of SP1's MPLS VPN backbone to connect two of its sites, SP2 New York and SP2 London. SP2, in turn, may then offer connectivity to its own customers (mjlnet, in Figure 5-1) via its own and SP1's backbone network.

One of the most important advantages of deploying a CSC architecture is the fact that routes external to the CSC customer (such as Internet and mjlnet destinations shown in Figure 5-1) need not be held on the CSC Provider Edge (PE) routers.

There are two basic CSC architectures:

• When MPLS is not enabled between CSC customer routers (packet forwarding is IP based).
• When MPLS is enabled between CSC customer routers (packet forwarding is MPLS based). When MPLS is enabled between CSC customer routers, the CSC customer can, in turn, offer MPLS VPN service to its customers.

Note that irrespective of the CSC architecture, MPLS must be enabled between CSC PE and CSC Customer Edge (CE) routers.

The sections that follow examine both of these CSC architectures.

CSC Architecture When MPLS Is Not Enabled Within CSC Customer Sites

Figure 5-2 shows a CSC architecture when MPLS is not enabled within CSC customer sites.

Figure 5-2. CSC Architecture When MPLS Is Not Enabled Within CSC Customer Sites.

Two factors must be considered when discussing an MPLS CSC architecture (whether or not MPLS is enabled between routers within CSC customer sites):

• Route advertisement How routes are advertised between sites
• Packet forwarding How packets are forwarded between sites over the CSC MPLS VPN backbone network

Route advertisement and packet forwarding in a CSC architecture when MPLS is not enabled within the CSC customer sites are discussed in the following two sections.

Route Advertisement in a CSC Architecture When MPLS Is Not Enabled Within CSC Customer Sites

There are two important components of route advertisement in the CSC MPLS architecture:

• Advertisement of CSC customer external routes between CSC customer sites

  CSC customer external routes consist of the following:

  - CSC customer customers' routes (for example, the routes received from mjlnet sites 1 and 2 in Figure 5-2)

  - Any Internet routes

• Advertisement of CSC customer internal routes between CSC customer sites

  CSC customer internal routes consist of the following:

  - Routes to the border gateway protocol (BGP) next hops of CSC customer external routes (usually the loopback interfaces on CSC customer Autonomous System Boundary Routers [ASBRs]).

  - Routes to resources/networks internal to the CSC customer sites (management networks, shared services, and so on).

  Because reachability to resources within the CSC customer networks is provided using exactly the same mechanisms (either an interior gateway protocol [IGP] or static routes) used to advertise BGP next hops for external routes, the focus of discussion within this section is on providing reachability for the next hops of external BGP routes.

Now that you know what route advertisement consists of in a carriers' carrier MPLS VPN architecture, it's a good idea to take a look at an example. Figure 5-3 illustrates external and internal route advertisement between CSC customer sites.

Figure 5-3. External and Internal Route Advertisement Between CSC Customer Sites

The following two sections detail how external and internal routes are advertised between CSC customer sites.

Advertising External Routes Between CSC Customer Sites

CSC customer external routes are advertised directly between CSC customer sites using BGP. If you take a close look at Figure 5-3, you can see that there is a BGP session directly between the ASBRs (SP2.NewYork_ASBR#1 and SP2.London_ASBR#2, which also function as route reflectors in this example) at customer SP2's New York and London sites. This BGP session is used to exchange CSC customer external routes.

As shown in Figure 5-3, SP2's external networks consist of the following:

• Internet networks 172.16.40.0/24 through 172.16.45.0/24 and 172.16.50.0/24 through 172.16.55.0/24. Note that these (private) networks, though not in reality routable on the public Internet, are used here for illustrative purposes.
• SP2 customer (mjlnet) site networks 10.1.1.0/24 through 10.1.5.0/24 and 10.2.1.0/24 through 10.2.5.0/24.

Example 5-1 shows the configuration of the BGP neighbor relationship between SP2.NewYork_ASBR#1 (192.168.101.3) and SP2.London_ASBR#2 (192.168.201.3) for the exchange of external routes.

Example 5-1. Configuration of the BGP Neighbor Relationship Between SP2.NewYork_ASBR#1 and SP2.London_ASBR#2

!
! On SP2.NewYork_ASBR#1
!
router bgp 65001
 no synchronization
 neighbor 192.168.201.3 remote-as 65001
 neighbor 192.168.201.3 update-source Loopback0
 neighbor 192.168.201.3 next-hop-self
 no auto-summary

 

!
! On SP2.London_ASBR#2
!
router bgp 65001
 no synchronization
 neighbor 192.168.101.3 remote-as 65001
 neighbor 192.168.101.3 update-source Loopback0
 neighbor 192.168.101.3 next-hop-self
!

As you can see, the configuration in Example 5-1 is standardno special BGP configuration is required for the advertisement of external routes between CSC customer sites. Notice that the same autonomous system number is used at both CSC customer sites (65001), and so route exchange is via IBGP.

Caution

You must ensure that all routers (in the packet-forwarding path) within CSC customer sites possess external routes when MPLS is not enabled in the CSC customer networks. If some routers do not possess external routes, packet forwarding will fail (more on this later).

You can ensure that all routers possess external routes in a number of ways, including configuring a full mesh of BGP neighbors between CSC customer site routers or (in a more scalable configuration) configuring route reflectors at the CSC customer sites.

Note

SP2.NewYork_ASBR#1 and SP2.London_ASBR#2 are configured as route reflectors and to advertise routes directly between themselves in Example 5-3, but a more common real-world configuration would involve the advertisement of external routes between dedicated route reflectors at the CSC customer sites. In this case, all other CSC customer site routers (including ASBRs) would be route reflector clients.

For more information on the configuration of route reflectors, see the following URL:

http://www.cisco.com/en/US/products/sw/iosswrel/ps1828/products_configuration_guide_chapter09186a00800ca571.html#wp5155

External routes advertised between SP2.NewYork_ASBR#1 and SP2.London_ASBR#2 are not be installed into the routing tables of SP2 routers unless the next hops of these BGP routes are contained within their routing tables.

In this example, all routes received by SP2.NewYork_ASBR#1 and SP2.London_ASBR#2 from external peers (external customers and Internet peers) have their next hops reset using the neighbor ip-address next-hop-self BGP command, and so the next hop of the external routes correspond to the loopback interface addresses of the ASBRs, SP2.NewYork_ASBR#1 and SP2.London_ASBR#2.

In this example, therefore, in order for external routes to be installed into the routing tables of SP2 routers, loopback interface addresses of ASBRs must be advertised between SP2 sites. The loopback interface addresses are, as previously described, internal routes within the CSC customer sites, which brings us onto advertisement of internal routes between SP2 sites.

Advertising Internal Routes Between CSC Customer Sites

As previously mentioned, CSC customer site internal routes describe reachability to router loopback addresses (external route next hops) and other resources within the CSC customer sites. Internal routes are advertised between CSC customer sites via the CSC backbone network using any of the regular PE-CE routing protocols such as Open Shortest Path First (OSPF) or Intermediate System-to-Intermediate System (IS-IS). Alternatively, static routes can be configured on CSC PE and CE routers (and redistributed on the CSC CEs and PEs as appropriate).

In the example in this section, OSPF is the IGP used within both of the SP2 customer sites, and so this is also chosen as the PE-CE routing protocol on the connections between SP2.NewYork.CSC.CE and SP2.NewYork.CSC.PE, as well as SP2.London.CSC.CE and SP2.London.CSC.PE.

Example 5-2 shows the relevant interface and routing protocol configuration for SP2.NewYork.CSC.CE. Note that the configuration of router SP2.London.CSC.CE (not shown) is almost identicalonly the IP addressing differs.

Example 5-2. Interface and Routing Protocol Configurations for SP2.NewYork.CSC.CE

!
! On SP2.NewYork.CSC.CE (line 1)
!
!
!
mpls label protocol ldp (line 2)
!
interface Serial1/1
 description To SP2.NewYork.CSC.PE
 ip address 172.16.31.1 255.255.255.0
 mpls ip (line 3)
!
!
router ospf 100 (line 4)
 passive-interface Loopback0
 network 172.16.31.0 0.0.0.255 area 0
 network 192.168.0.0 0.0.255.255 area 0
!

For the most part, the configuration shown in Example 5-2 is straightforwardthe OSPF configuration beginning in highlighted line 4 enables OSPF on SP2 internal interfaces (192.168.0.0/16), and also on the interface connecting CSC CE and PE routers, SP2.NewYork.CSC.CE and SP2.NewYork.CSC.PE (interface serial 1/1 [172.16.31.0/24]).

There is, however, one seeming oddity with regard to the configuration of CSC CE SP2.NewYork.CSC.CELabel Distribution Protocol (LDP) is enabled on the interface between SP2.NewYork.CSC.CE and SP1.NewYork.CSC.PE (see highlighted lines 2 and 3). LDP must be enabled between CSC PE and CE routers so that they can exchange label bindings for CSC customer internal routes (including the next hops of external routes). If a label distribution is not enabled, packet forwarding will fail.

Note

As an alternative to LDP, BGP may be used to distribute IPv4 label bindings between CSC PE and CE routers (many service providers prefer BGP). LDP, however, is assumed in this section.

More information on the use of BGP as a label distribution protocol for IPv4 can be found in the section "The Inter-Autonomous System/Interprovider MPLS VPN Architecture" later in this chapter (see in particular Examples 5-23 and 5-24).

It is worth mentioning here that if you do choose to use BGP, be careful to filter the routes that you send to the CSC PE router so that you do not include external routes as well as internal routes.

Example 5-3 shows the interface and routing protocol configuration for SP1.NewYork.CSC.PE. Again, the configuration of router SP1.London.CSC.PE (not shown) is almost the same, with the only difference being the IP addressing.

Example 5-3. Interface and Routing Protocol Configurations for SP1.NewYork.CSC.PE

!
! On SP1.NewYork.CSC.PE (line 1)
!
ip vrf VPN-sp2 (line 2)
 rd 65535:500
 route-target export 65535:500
 route-target import 65535:500
!
interface Serial2/1
 ip vrf forwarding VPN-sp2 (line 3)
 ip address 172.16.31.2 255.255.255.0
 mpls ldp discovery transport-address interface (line 4)
 mpls ip (line 5)
!
router ospf 1 vrf VPN-sp2 (line 6)
 redistribute bgp 65535 subnets
 network 172.16.31.0 0.0.0.255 area 0
!
router bgp 65535 (line 7)
 no synchronization
 neighbor 192.168.1.3 remote-as 65535
 neighbor 192.168.1.3 update-source Loopback0
 no auto-summary
 !
 address-family vpnv4 (line 8)
 neighbor 192.168.1.3 activate
 neighbor 192.168.1.3 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf VPN-sp2 (line 9)
 redistribute ospf 1 vrf VPN-sp2 match internal external 1 external 2
 no auto-summary
 no synchronization
 exit-address-family
!

The configuration of CSC PE router SP1.NewYork.CSC.PE is again pretty standard.

In highlighted line 2, a VPN routing and forwarding table (VRF) called VPN-sp1 is configured, and the interface connected to CSC CE router SP2.NewYork.CSC.CE (serial 2/1) is associated with this VRF using the ip vrf forwarding vrf-name command (see highlighted line 3).

In highlighted line 6, OSPF is enabled as the PE-CE routing protocol for VRF VPN-sp2. OSPF is configured on the interface connected to SP2.NewYork.CSC.CE (interface serial 2/1), and MP-BGP is redistributed into OSPF. Nothing special here.

Then in highlighted line 7, BGP is enabled and a neighbor relationship is configured with SP1.London.CSC.PE (192.168.1.3).

In highlighted line 8 and 9, MP-BGP is enabled for VPN-IPv4 route exchange with SP1.London.CSC.PE (see below highlighted line 8), and the PE-CE routing protocol OSPF is redistributed into MP-BGP (within the IPv4 VRF address familysee below highlighted line 9). Again, nothing special.

But take a look at highlighted lines 4 and 5. Here's where the configuration of the CSC architecture differs from regular MPLS VPN configuration on PE routers.

In highlighted line 5, LDP is enabled on the interface connected to SP2.NewYork.CSC.CE (LDP is automatically selected as the label protocol when configuring the mpls ip command on a VRF interface). Remember that LDP (or BGP) must be enabled between CSC CE and PE to ensure that label bindings for CSC customer internal routes are exchanged and packet forwarding does not fail.

Note

Note also the mpls ldp discovery transport-address interface in highlighted line 4. This command may be necessary on the VRF interface connected to the CSC CE (and also possibly on the CSC CE interface connected to the CSC PE) to ensure that an LDP transport connection can be successfully established between the CSC PE and CE and label bindings can be exchanged.

By default, LDP uses the LDP router ID IP address (a loopback interface address or highest physical interface address) as the source/destination of its transport connection, but an LDP transport connection will fail if the CSC PE and CE do not have routes to each other's LDP router ID. There are a number of ways around this problem, but the easiest is to use the mpls ldp discovery transport-address interface command to configure LDP to use the (VRF) PE-CE interface address as the source/destination of its transport connection rather than the LDP router ID.

So, CSC customer internal routes are exchanged between sites in exactly the same way that customer routes are exchanged between sites in a standard (single service provider) MPLS VPN configuration.

Example 5-4 shows the output of the show ip route command on SP2.NewYork_ASBR#1 and SP2.London_ASBR#2.

Example 5-4. show ip route Command Output on CSC Customer ASBRs SP2.NewYork_ASBR#1 and SP2.London_ASBR#2

SP2.NewYork_ASBR#1#show ip route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
 ia - IS-IS inter area, * - candidate default, U - per-user static route
 o - ODR

Gateway of last resort is not set

O 192.168.104.0/24 [110/65] via 192.168.102.2, 00:24:13, FastEthernet0/0
 172.16.0.0/16 is variably subnetted, 15 subnets, 2 masks
B 172.16.52.0/24 [200/0] via 192.168.201.3, 00:07:19 (line 1)
B 172.16.53.0/24 [200/0] via 192.168.201.3, 00:07:19 (line 2)
B 172.16.54.0/24 [200/0] via 192.168.201.3, 00:07:19 (line 3)
B 172.16.55.0/24 [200/0] via 192.168.201.3, 00:07:19 (line 4)
B 172.16.50.0/24 [200/0] via 192.168.201.3, 00:07:19 (line 5)
B 172.16.51.0/24 [200/0] via 192.168.201.3, 00:07:19 (line 6)
B 172.16.44.0/24 [20/0] via 172.16.20.1, 00:03:32 (line 7)
B 172.16.45.0/24 [20/0] via 172.16.20.1, 00:03:32 (line 8)
B 172.16.40.0/24 [20/0] via 172.16.20.1, 00:03:32 (line 9)
B 172.16.41.0/24 [20/0] via 172.16.20.1, 00:03:32 (line 10)
B 172.16.42.0/24 [20/0] via 172.16.20.1, 00:03:32 (line 11)
B 172.16.43.0/24 [20/0] via 172.16.20.1, 00:03:32 (line 12)
O IA 172.16.32.0/24 [110/114] via 192.168.102.2, 00:24:14, FastEthernet0/0
O IA 172.16.32.2/32 [110/114] via 192.168.102.2, 00:24:14, FastEthernet0/0
O 172.16.31.0/24 [110/113] via 192.168.102.2, 00:24:14, FastEthernet0/0
 192.168.201.0/32 is subnetted, 3 subnets
O IA 192.168.201.3 [110/290] via 192.168.102.2, 00:24:15,
 FastEthernet0/0 (line 13)
O IA 192.168.201.2 [110/162] via 192.168.102.2, 00:24:15,
 FastEthernet0/0 (line 14)
O IA 192.168.201.4 [110/226] via 192.168.102.2, 00:24:15,
 FastEthernet0/0 (line 15)
O IA 192.168.202.0/24 [110/289] via 192.168.102.2, 00:24:15, FastEthernet0/0
O IA 192.168.204.0/24 [110/225] via 192.168.102.2, 00:24:15, FastEthernet0/0
C 192.168.102.0/24 is directly connected, FastEthernet0/0
 192.168.101.0/32 is subnetted, 3 subnets
O 192.168.101.4 [110/2] via 192.168.102.2, 00:24:15,
 FastEthernet0/0 (line 16)
C 192.168.101.3 is directly connected, Loopback0 (line 17)
O 192.168.101.2 [110/66] via 192.168.102.2, 00:24:15, FastEthernet0/0 (line 18)
SP2.NewYork_ASBR#1#

 

SP2.London_ASBR#2#show ip route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
 ia - IS-IS inter area, * - candidate default, U - per-user static route
 o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

O IA 192.168.104.0/24 [110/288] via 192.168.202.1, 00:33:24, Serial0/0
 172.16.0.0/16 is variably subnetted, 15 subnets, 2 masks
B 172.16.52.0/24 [20/0] via 172.16.60.1, 00:08:53 (line 19)
B 172.16.53.0/24 [20/0] via 172.16.60.1, 00:08:53 (line 20)
B 172.16.54.0/24 [20/0] via 172.16.60.1, 00:08:53 (line 21)
B 172.16.55.0/24 [20/0] via 172.16.60.1, 00:08:53 (line 22)
B 172.16.50.0/24 [20/0] via 172.16.60.1, 00:08:53 (line 23)
B 172.16.51.0/24 [20/0] via 172.16.60.1, 00:08:53 (line 24)
B 172.16.44.0/24 [200/0] via 192.168.101.3, 00:14:22 (line 25)
B 172.16.45.0/24 [200/0] via 192.168.101.3, 00:14:22 (line 26)
B 172.16.40.0/24 [200/0] via 192.168.101.3, 00:14:22 (line 27)
B 172.16.41.0/24 [200/0] via 192.168.101.3, 00:14:22 (line 28)
B 172.16.42.0/24 [200/0] via 192.168.101.3, 00:14:22 (line 29)
B 172.16.43.0/24 [200/0] via 192.168.101.3, 00:14:22 (line 30)
O 172.16.32.0/24 [110/192] via 192.168.202.1, 00:33:25, Serial0/0
O IA 172.16.31.1/32 [110/193] via 192.168.202.1, 00:33:25, Serial0/0
O IA 172.16.31.0/24 [110/193] via 192.168.202.1, 00:33:25, Serial0/0
 192.168.201.0/32 is subnetted, 3 subnets
C 192.168.201.3 is directly connected, Loopback0 (line 31)
O 192.168.201.2 [110/129] via 192.168.202.1, 00:33:25, Serial0/0 (line 32)
O 192.168.201.4 [110/65] via 192.168.202.1, 00:33:25, Serial0/0 (line 33)
 192.168.202.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.202.0/24 is directly connected, Serial0/0
C 192.168.202.1/32 is directly connected, Serial0/0
O 192.168.204.0/24 [110/128] via 192.168.202.1, 00:33:26, Serial0/0
O IA 192.168.102.0/24 [110/298] via 192.168.202.1, 00:33:26, Serial0/0
 192.168.101.0/32 is subnetted, 3 subnets
O IA 192.168.101.4 [110/289] via 192.168.202.1, 00:33:26, Serial0/0 (line 34)
O IA 192.168.101.3 [110/299] via 192.168.202.1, 00:33:26, Serial0/0 (line 35)
O IA 192.168.101.2 [110/241] via 192.168.202.1, 00:33:26, Serial0/0 (line 36)
SP2.London_ASBR#2#

The first part of Example 5-4 shows the output of the show ip route command on SP2.NewYork_ASBR#1, with highlighted lines 1 to 18 showing CSC customer external and internal routes.

In highlighted lines 1 to 12, you can see external routes 172.16.50.0/24 through 172.16.55.0/24 and 172.16.40.0/24 through 172.16.45.0/24.

Highlighted lines 13 to 18 show the internal routes corresponding to CSC customer (SP2) router loopback interfaces. Of particular interest are the routes shown in highlighted lines 13 and 17 (192.168.201.3/32 and 192.168.101.3/32)these two prefixes correspond to the loopback interface IP addresses on SP2.NewYork_ASBR#1 and SP2.London_ASBR#2 and are the next hops of the external BGP routes.

The output of the show ip route command on SP2.London_ASBR#2 is similar to that from SP2.NewYork_ASBR#1. Highlighted lines 19 to 36 show the external and internal routes.

You can see external routes in highlighted lines 19 to 30, and in highlighted lines 31 to 36, you can see internal routes corresponding to SP2 router loopback interface addresses. Again, the two most interesting internal routes are the prefixes shown in highlighted lines 31 and 35 (the loopback interfaces addresses of the ASBRs, which are the next-hop addresses for the external routes).

You can now see that SP2.NewYork_ASBR#1 and SP2.London_ASBR#2 have IP reachability (internal routes) to each other, as well as all CSC customer external destinations (external routes).

Before moving on to packet forwarding, it is important to just restate a few of the most important facts to remember about route advertisement in a CSC architecture when MPLS is not enabled in the CSC customer networks:

• CSC customer external routes provide IP reachability to destinations external to the CSC customer (external customers and/or Internet).
• CSC customer external routes are advertised directly between CSC customer sites using BGP.
• All routers within the CSC customer sites (in the IP forwarding path) must possess external routes otherwise packet forwarding will fail. You can ensure that all CSC customer routers contain external routes using a number of methods including configuring a BGP neighbor full mesh between CSC customer routers or preferably configuring route reflectors at CSC customer sites.
• CSC customer internal routes provide IP reachability to destinations within the CSC customer sites. Significantly, some internal routes (often loopback interface addresses on ASBRs) provide next-hop reachability for external BGP routes.
• CSC customer internal routes are advertised between CSC customer sites via the CSC backbone network. Standard PE-CE routing protocols can be used between CSC PE and CE routers.

Packet Forwarding in a CSC Architecture When MPLS Is Not Enabled Between Routers Within CSC Customer Sites

You must remember a few important things about packet forwarding in a CSC architecture when MPLS is not enabled between routers within CSC customer sites:

• Packet forwarding within the CSC customer sites is purely IP based.
• Packet forwarding between the CSC CEs and PEs is MPLS based.
• Packet forwarding within the CSC backbone network is MPLS based and conforms to the regular MPLS VPN backbone packet-forwarding paradigm.

To really understand how packet forwarding works in a CSC architecture, it is useful to examine how a packet is forwarded from a source in SP2's New York site to a destination external to SP2's London site.

Example 5-5 demonstrates how to use the traceroute command to examine packet forwarding hop by hop from source SP2.NewYork_ASBR#1 (SP's New York site) to destination IP address 172.16.51.1 (an Internet host reachable via SP2's London site [the direct Internet connection from SP2's New York site is down in this example]).

Example 5-5. Traceroute from Source SP2.NewYork_ASBR#1 to Destination IP Address 172.16.51.1 (a Host on a Network External to SP2 Site 2).

SP2.NewYork_ASBR#1#traceroute 172.16.51.1

Type escape sequence to abort.
Tracing the route to 172.16.51.1

 1 192.168.102.2 0 msec 0 msec 4 msec (line 1)
 2 192.168.104.2 16 msec 12 msec 16 msec (line 2)
 3 172.16.31.2 [MPLS: Label 32 Exp 0] 244 msec 328 msec 240 msec (line 3)
 4 192.168.2.2 [MPLS: Labels 19/25 Exp 0] 236 msec 228 msec 232 msec (line 4)
 5 172.16.32.1 [MPLS: Label 25 Exp 0] 204 msec 204 msec 208 msec (line 5)
 6 172.16.32.2 [MPLS: Label 18 Exp 0] 188 msec 188 msec 192 msec (line 6)
 7 192.168.204.2 88 msec 88 msec 88 msec (line 7)
 8 192.168.202.2 100 msec 100 msec 100 msec (line 8)
 9 172.16.60.1 115 msec 116 msec * (line 9)
SP2.NewYork_ASBR#1#

In highlighted lines 1 and 2, the packet is forwarded from SP2.NewYork_ASBR#1 to SP2.NewYork.C (192.168.102.2), and then from SP2.NewYork.C to SP2.NewYork.CSC.CE (192.168.104.2).

Notice that the output of the traceroute command does not show MPLS labels in highlighted lines 1 and 2. The lack of MPLS labels indicates that regular IP forwarding is used at each of these two hops (routers). The fact that IP forwarding is used is significant. It means that each of these two routers (in fact, all CSC customer site routers [in the packet-forwarding path]) must have knowledge of external routes (in this example, an external route to 172.16.51.1), otherwise packet forwarding will fail. So, as previously stated, CSC customer site routers must be configured as BGP peers to receive external routes.

In highlighted line 3, the packet is forwarded from SP2.NewYork.CSC.CE to SP1.NewYork.CSC.PE (172.16.31.2). Notice that an MPLS label (32) is shown in the output here. This label is very significant. It actually corresponds to the next hop of BGP route 172.16.51.0/24 (next hop 192.168.201.3 [SP2.London_ASBR#2]). You'll remember that Label Switching Routers (LSRs) do not (normally) assign labels to regular BGP routes but instead use the label corresponding to the next hops of those BGP routes.

The output of the show mpls forwarding-table command in Example 5-6 confirms that (outgoing) label 32 corresponds to prefix (next hop) 192.168.201.3.

Example 5-6. show mpls forwarding-table Command Output

SP2.NewYork.CSC.CE#show mpls forwarding-table
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
16 Untagged 172.16.31.2/32 0 Se1/1 point2point
17 Untagged 192.168.104.1/32 0 Se1/0 point2point
18 Untagged 192.168.102.0/24 4656 Se1/0 point2point
19 Untagged 192.168.101.3/32 9547 Se1/0 point2point
20 Untagged 192.168.101.4/32 0 Se1/0 point2point
21 29 172.16.32.0/24 0 Se1/1 point2point
22 30 172.16.32.2/32 0 Se1/1 point2point
23 31 192.168.201.2/32 0 Se1/1 point2point
24 33 192.168.201.4/32 0 Se1/1 point2point
25 35 192.168.204.0/24 0 Se1/1 point2point
26 34 192.168.202.0/24 0 Se1/1 point2point
27 32 192.168.201.3/32 0 Se1/1 point2point
SP2.NewYork.CSC.CE#

The significance of the label (corresponding to the next hop of the external BGP route) pushed onto the packet in Example 5-5, highlighted line 3 is that it will allow the ingress CSC PE router, SP1.NewYork.CSC.PE, to forward the packet without having any knowledge of CSC customer external routes (including a route to destination 172.16.51.1).

For SP1.NewYork.CSC.PE to possess (assign) labels for next hop 192.168.201.3, all it needs to have is a route to this next hop. How does SP1.NewYork.CSC.PE get routes to the next hops of CSC customer external routes? It gets routes to next hops via the advertisement of CSC customer internal routes from SP2.London.CSC.CE to SP1.London.CSC.PE (advertised using the PE-CE routing protocol), and the advertisement of those routes from SP1.London.CSC.PE using MP-BGP.

In Example 5-5, highlighted line 4, the packet transits the link from SP1.NewYork.CSC.PE to SP1.London.P. Notice that SP1.NewYork.CSC.PE has swapped the label (32) for a label stack consisting of an IGP label (19) and a VPN label (25). The function of the IGP label is to get the packet from the ingress PE (SP1.NewYork.CSC.PE) to the egress PE (SP1.London.CSC.PE), and the function of the VPN label is to identify the outgoing interface or VRF on the egress PE.

Example 5-5 highlighted line 5 shows the packet as it transits the link from SP1.London.P to SP1.London.CSC.PE. SP1.London.P has popped the IGP label, leaving just the VPN label (it is the penultimate hop, so nothing unusual there).

Now to Example 5-5 highlighted line 6. This line shows the packet as it traverses the link between SP1.London.CSC.PE and SP2.London.CSC.CE. If you look closely, you can see that SP2.London.CSC.PE has removed the VPN label and pushed a label (18) corresponding to a route to next hop 192.168.201.3 (SP2.London_ASBR#2). SP1.London.CSC.PE is, like SP1.NewYork.CSC.PE, exchanging internal routes and corresponding label bindings with its connected CSC CE (SP2.London.CSC.CE).

In Example 5-5, highlighted lines 7 and 8, the packet traverses the links between SP2.London.CSC.CE and SP2.London.C, as well as SP2.London.C and SP2.London_ASBR#2. Neither of these highlighted lines shows an MPLS label, and so you know that SP2.London.CSC.CE and SP2.London.C used regular IP forwarding. Again, SP2.London.CSC.CE and SP2.London.C must possess external BGP routes to do this.

Then, in highlighted line 9, the packet is shown as it transits the connection between SP2.London_ASBR#2 and an external router. The output terminates at line 9 because the external router is directly connected to the destination IP address 172.16.51.1.

Figure 5-4 illustrates packet forwarding across the network from SP2.NewYork_ASBR#1 to external destination 172.16.51.1.

Figure 5-4. Packet Forwarding Across the Network from SP2.NewYork_ASBR#1 to External Destination 172.16.51.1

So, now you know how route advertisement and packet forwarding work in a CSC architecture when MPLS is not enabled in the CSC customer networks. But, how about if MPLS is enabled in CSC customer networks?

CSC Architecture When MPLS Is Enabled Within CSC Customer Sites

CSC customers gain two main advantages by enabling MPLS within their sites:

• CSC CE routers and nonedge routers (such as SP2.NewYork.C and SP2.London.C in Figure 5-5) no longer need external BGP routes.

  Figure 5-5. A CSC Architecture When MPLS Is Enabled Within CSC Customer Sites

  
• The CSC customer can offer MPLS VPN services to its own customers. This is sometimes called a hierarchical VPN architecture.

Figure 5-5 illustrates a CSC architecture when MPLS is enabled within CSC customer sites.

It is important to understand route advertisement and packet forwarding in a CSC architecture when MPLS is enabled in the CSC customer sites to understand why external routes are no longer required on some CSC customer routers, and why the CSC customer can now offer MPLS VPN services to its own customers.

Route Advertisement in a CSC Architecture When MPLS Is Enabled Within CSC Customer Sites

Route advertisement in a CSC architecture when MPLS is enabled within is much the same as when MPLS is not enabled within CSC customer sites, with (as previously mentioned) one key difference: External BGP routes no longer need to be advertised to CSC CE and nonedge CSC customer site routers.

Figure 5-6 illustrates external and internal route advertisement between CSC customer sites when MPLS is enabled at CSC customer sites. You might want to compare Figure 5-6 to Figure 5-3 (where MPLS is not enabled at the CSC customer sites).

Figure 5-6. External and Internal Route Advertisement Between CSC Customer Sites (When MPLS Is Enabled CSC Customer Sites)

The reason that external routes are no longer required on CSC CE routers and nonedge CSC customer routers is that these routers now forward packets using MPLS. Just as with P routers in the standard MPLS VPN forwarding paradigm, CSC CE and nonedge CSC customer routers now only require internal routes (including routes to the next hops of external BGP routes) and label bindings corresponding to internal routes to successfully forward packets.

Therefore, the only configuration changes required when enabling MPLS in CSC customer sites are as follows:

• MPLS must be enabled (using the mpls ip command) on all CSC customer site router core-facing interfaces (those not facing external destinations).
• BGP configuration (that was required for external routes) can be removed on CSC CE routers (assuming that BGP is not the PE-CE routing protocol) and nonedge CSC customer site routers (with the exception of that on route reflectors!).

  Routers connected to external destinations (ASBRs) still require BGP configuration (and external routes) because they are required to perform regular IP packet forwarding to and from external networks.

Note

Although it cannot be used as the label protocol between CSC PE and CE routers, it is possible to use Tag Distribution Protocol (TDP) to advertise label bindings between LSRs within CSC customer sites.

Note, however, that the use of Cisco's proprietary TDP is deprecated in favor of IETF standard LDP.

 

Packet Forwarding in a CSC Architecture When MPLS Is Enabled Between Routers Within CSC Customer Sites

When MPLS is enabled within CSC customer sites, CSC customer site routers forward packets using MPLS. Forwarding within the CSC backbone network is exactly the same as when MPLS is not enabled within CSC customer sites.

Example 5-7 demonstrates using the traceroute command to examine packet forwarding hop by hop from source SP2.NewYork_ASBR#1 (SP2's New York site) to destination IP address 172.16.51.1 (a host on a network external to SP2's London site).

Example 5-7. Traceroute from Source SP2.NewYork_ASBR#1 to Destination IP Address 172.16.51.1 When MPLS Is Enabled Within CSC Customer Sites

SP2.NewYork_ASBR#1#traceroute 172.16.51.1

Type escape sequence to abort.
Tracing the route to 172.16.51.1

 1 192.168.102.2 [MPLS: Labels 26 Exp 0] 548 msec 388 msec 388 msec (line 1)
 2 192.168.104.2 [MPLS: Labels 24 Exp 0] 372 msec 368 msec 520 msec (line 2)
 3 172.16.31.2 [MPLS: Labels 32 Exp 0] 352 msec 356 msec 352 msec (line 3)
 4 192.168.2.2 [MPLS: Labels 19/35 Exp 0] 344 msec 344 msec 340 msec (line 4)
 5 172.16.32.1 [MPLS: Labels 35 Exp 0] 320 msec 384 msec 316 msec (line 5)
 6 172.16.32.2 [MPLS: Labels 25 Exp 0] 304 msec 304 msec 300 msec (line 6)
 7 192.168.204.2 [MPLS: Labels 20 Exp 0] 288 msec 288 msec 288 msec (line 7)
 8 192.168.202.2 220 msec 220 msec 224 msec (line 8)
 9 172.16.60.1 130 msec * 132 msec (line 9)
SP2.NewYork_ASBR#1

If you compare the output in Example 5-7 to that shown in Example 5-5 (when MPLS is not enabled within the CSC customer sites), you can see in highlighted lines 1, 2, and 7 that the difference is that the packet is now forwarded using MPLS (note the MPLS labels) between CSC customer site routers.

In highlighted line 1, the packet is forwarded between routers SP2.NewYork_ASBR#1 and SP2.NewYork.C.

Highlighted line 2 shows the packet as it is forwarded between routers SP2.NewYork.C and SP2.NewYork.CSC.CE.

From highlighted line 3 to 6, the packet is forwarded from SP2.NewYork.CSC.CE across the CSC backbone network to SP2.London.CSC.CE. Forwarding across the CSC backbone network is unchanged from Example 5-5 (when MPLS is not enabled within CSC customer sites).

In highlighted line 7, the packet is forwarded between routers SP2.London.CSC.CE and SP2.London.C.

Then in highlighted line 8, the packet is forwarded between SP2.London.C and SP2.London_ASBR#2.

Finally, the packet is forwarded between routers SP2.London_ASBR#2 and the external router that is connected to host 172.16.51.1.

Figure 5-7 shows packet forwarding across the network from SP2.NewYork_ASBR#1 to external destination 172.16.51.1 when MPLS is enabled within CSC customer sites.

Figure 5-7. Packet Forwarding Across the Network from SP2.NewYork_ASBR#1 to External Destination 172.16.51.1 When MPLS Is Enabled Within Sites

 

Enabling Hierarchical VPNs in a CSC Architecture

As previously mentioned, one major advantage of enabling MPLS within CSC customer sites is that the CSC customer can now offer MPLS VPN service to its customers.

In this case, route advertisement remains that same as previously described (when MPLS is enabled within CSC customers), with one key differenceMP-BGP must be enabled between ASBR/PEs connected to the CSC customer's MPLS VPN-enabled customers to advertise VPN-IPv4 routes.

Figure 5-8 shows hierarchical VPNs.

Figure 5-8. Hierarchical VPNs

As you can see in Figure 5-8, connectivity for VPN-mjlnet is provisioned over the SP2 (and SP1) network.

Figure 5-9 shows route and label advertisement in hierarchical VPNs.

Figure 5-9. Route and Label Advertisement in Hierarchical VPNs

Example 5-8 shows the configuration of SP2.NewYork_ASBR#1/PE when hierarchical VPNs are enabled. SP2.NewYork_ASBR#1/PE is connected to VPN-mjlnet site 1.

Note that the configuration of SP2.London_ASBR#2/PE (connected to VPN-mjlnet site 2) is almost identical, with the only difference being IP address configuration.

Example 5-8. Configuration of SP2.NewYork_ASBR#1/PE When Hierarchical MPLS VPNs Are Enabled

!
! On SP2.NewYork_ASBR#1/PE# (line 1)

!
ip vrf VPN-mjlnet (line 2)
 rd 65535:100
 route-target export 65535:100
 route-target import 65535:100
!
!
interface Serial1/1 (line 3)
 ip vrf forwarding VPN-mjlnet
 ip address 10.2.1.2 255.255.255.0
!
router bgp 65001 (line 4)
 no synchronization
 neighbor 192.168.201.3 remote-as 65001
 neighbor 192.168.201.3 update-source Loopback0
 no auto-summary
 !
 address-family vpnv4 (line 5)
 neighbor 192.168.201.3 activate
 neighbor 192.168.201.3 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf VPN-mjlnet (line 6)
 neighbor 10.2.1.1 remote-as 64512
 neighbor 10.2.1.1 activate
 neighbor 10.2.1.1 as-override
 no auto-summary
 no synchronization
 exit-address-family
!

Highlighted lines 1 and 2 show the configuration of VRF VPN-mjlnet and the association of interface serial 1/1 with VRF VPN-mjlnet.

In highlighted lines 3, BGP is enabled on SP2.NewYork_ASBR#1/PE.

Then, in highlighted line 4, MP-BGP is enabled for the advertisement of VPN-IPv4 routes between CSC customer sites, and in highlighted line 5, an IPv4 address family is configured for VRF VPN-mjlnet.

Notice that in this example (below highlighted line 6), the PE-CE routing protocol for VPN VPN-mjlnet is BGP. The customer (mjlnet) autonomous system is 64512, and because the same autonomous system is being used at mjlnet sites 1 and 2, the neighbor ip-address as-override is required to ensure that routes are not dropped.

So that is route advertisement for hierarchical VPNs. Now on to packet forwarding.

Example 5-9 demonstrates using the traceroute command to examine packet forwarding between VPN mjlnet sites 1 and 2.

Example 5-9. Traceroute Output from VPN mjlnet Site 1 to VPN mjlnet Site 2

VPN-mjlnet.NewYork.CE#traceroute 10.1.2.1

Type escape sequence to abort.
Tracing the route to 10.1.2.1

 1 10.2.1.2 4 msec 0 msec 0 msec (line 1)
 2 192.168.102.2 [MPLS: Labels 26/29 Exp 0] 548 msec 388 msec 388 msec (line 2)
 3 192.168.104.2 [MPLS: Labels 24/29 Exp 0] 372 msec 368 msec 520 msec (line 3)
 4 172.16.31.2 [MPLS: Labels 32/29 Exp 0] 352 msec 356 msec 352 msec (line 4)
 5 192.168.2.2 [MPLS: Labels 19/35/29 Exp 0] 344 msec 344 msec 340 msec (line 5)
 6 172.16.32.1 [MPLS: Labels 35/29 Exp 0] 320 msec 384 msec 316 msec (line 6)
 7 172.16.32.2 [MPLS: Labels 25/29 Exp 0] 304 msec 304 msec 300 msec (line 7)
 8 192.168.204.2 [MPLS: Labels 20/29 Exp 0] 288 msec 288 msec 288 msec (line 8)
 9 10.1.1.1 [AS 65001] [MPLS: Label 29 Exp 0] 220 msec 220 msec 224 msec (line 9)
 10 10.1.1.2 [AS 65001] 136 msec * 136 msec (line 10)
VPN-mjlnet.NewYork.CE#

Highlighted line 1 shows the packet as it transits the link between CE router VPN-mjlnet.NewYork.CE and SP2.NewYork_ASBR#1/PE.

In highlighted line 2, the packet crosses the connection between SP2.NewYork_ASBR#1/PE and SP2.NewYork.C. Notice that SP2.NewYork_ASBR#1/PE has pushed two labels onto the packet, an IGP label (26) and a VPN label (29). This VPN label identifies the outgoing interface or VRF on the egress PE router, SP2.London_ASBR#2/PE.

In highlighted line 3, the packet crosses the link between SP2.NewYork.C and SP2.NewYork.CSC.CE. Notice that the IGP label has been swapped (it is now label 24), but the VPN label (29) is unchanged.

From highlighted line 4 to highlighted line 7, the packet transits between routers SP2.NewYork.CSC.CE and SP2.London.CSC.CE via the CSC backbone network. Notice that the original VPN label (29) is maintained in the label stack and is not popped.

The packet then crosses the link between SP2.London.CSC.CE and SP2.London.C in highlighted line 8, and in highlighted line 9, it reaches router SP2.London_ASBR#1/PE. Again, notice that the original VPN label (29) has been maintained in the label stack.

Finally, in highlighted line 10, the packet is forwarded (unlabeled) to VPN-mjlnet.London.CE and its destination IP address 10.1.2.1.

Even though the sources and destinations are different, it is interesting to compare Example 5-7 (where a non-VPN packet is forwarded between MPLS-enabled CSC customer sites) to Example 5-9 (where hierarchical VPNs have been configured). You can see that the main body of each output is the same, with the only difference being the addition of an extra (VPN) label (label 29, in Example 5-9).

Figure 5-10 shows packet forwarding across the network from VPN-mjlnet site 1 to destination VPN-mjlnet site 2 when hierarchical VPNs are configured.

Figure 5-10. Packet Forwarding Across the Network from mjlnet Site 1 to mjlnet Site 2 When Hierarchical VPNs Are Configured