Equipment Scalability Versus Network Scalability

Scalability is an important factor when considering technology and equipment for deployment. Technology with limited scalability has severely limited applications. Specially, when MPLS is deployed by large service providers, you must understand how well MPLS scales and the role equipment scalability plays in scaling the overall network deployment.

We have stated in earlier chapters that equipment scalability must not hamper network scalability. However, when equipment runs into its bounds, adding more network elements can help scale the network; however, it also means more devices to manage. After a point, even adding more equipment can no longer provide the same linear change that is expectedthis is called the point of inflexion. The farther the point of inflexion on the scalability chart, the better the overall scale factor. Let us quantitatively define some things.

We have also shown in Chapters 4, "Layer 2 VPNs," and 5, "Layer 3 VPNs," that a network with MPLS network scalability is not an issue because devices can be added, and no single device is a bottleneck in building L3VPNs or L2VPNs.

Assume we are determining the scale factor of an L3 network. To establish the scale factor, we need to understand the network element scalability. The parameters influencing the scale of the network elements are as follows:

• Network element characteristics

• CPU power

• Memory (static and dynamic)

• Buffering capacity

• Number of line cards and interfaces nonblocking (interfaces include Channelized and Clear Channel)

• Network parameters

• Number of L2 VCs

  • FR DLCIs

  • ATM VCs

  • PPP sessions

  • Ethernet VLANs

  • High-Level Data Link Control (HDLC) sessions

• Number of L3 sessions

  • BGP peers (eBGP and iBGP)

  • IGP peers (OSPF, IS-IS, EIGRP, and RIP)

  • LDP peers

  • RSVP TE tunnels (head-end, mid-point, and tails)

• Number of routes

  • IGP prefixes and convergence

  • BGP prefixes and convergence

  • VPN prefixes and convergence

• MPLS scale

  • LDP scaleNumber of LDP sessions (neighbor and directed LDP sessions)

  • BGP assigned labels

  • RSVP assigned labels

  • PIM assigned labels

  • GMPLS signaling (RSVP-based)

  • MPLS forwarding

• SNMP Gets/Sets (simultaneous)

Network element characteristics determine the maximum number of network parameters possible on a device in isolation. However, even if a device is capable of supporting large numbers of each of these parameters in isolation, the device will not necessarily scale well in a real network. This is simply because none of these network parameters will be deployed/configured in isolation within the network. In a real network, the combination of these parameters matters. Hence, as an example, a device must not just be able to support a million prefixes, but must also support a large IGP table; label allocation for all those routes using LDP, BGP, or RSVP; thousands of L2 VCs; and hundreds or even thousands of IP interfaces all at the same time while passing traffic on each of the data paths.

You might think such a network device is nonexistent. In an ideal world, a network element can be configured with all the previously mentioned parameters with huge scale numbers for each and be able to pass traffic at line rate on all ports. However, practically speaking, irrespective of the vendor, it is rarely the case. Hence, the operational paradigm is important to determine what the reasonable scale numbers for a network element are. Decision-makers evaluating equipment for NGN must find their operational sweet spot. Having a highly scalable network element can drive the CAPEX up by running up the cost per unit, and whether it really drives the OPEX down is unclear because fewer devices are required in the network for the same customer base. This is because the OPEX is mainly associated with customer end-point management rather than network core or network edge (PE) management.

Moreover, configuring all services with large numbers of VPNs and L2 circuits equates to placing all your eggs in a single basket. The decision-makers must also determine whether the risk of placing all the services on a single node warrants the benefit gained by the cost reduction that can be attributed to using fewer devices for the same subscriber base. Should the highly scalable device fail or need to be taken out of service for any reason, the failure can affect many users.

Figure 13-5 shows the plot of network elements versus management cost. The fewer the network elements, the lower the management cost.

Figure 13-5. Management Costs and Risks

However, the fewer the network elements, the higher the risk of outage for subscribers.

Network Element Characteristics

Additionally, IP NGN managers and decision-makers must also determine how these network elements interact in the network. What happens when the IGP is too large or when a VPN is too large? They must design routing such that the network is appropriately segmented in areas or ASes such that any change in network does not impact the entire network but only that portion of the network where the change has occurred.

Let us now briefly discuss each of those parameters that determine network element scalability:

• CPU powerWith the hardware-based forwarding common in network elements today, CPU power is mainly used in handling control plane activity. Hence, the higher the CPU power, in terms of processing, the faster the network element can process routing updates, calculate paths, perform SPF, handle management, and converge. However, after custom ASICs and fabric, CPU is the single most expensive component in the network element. This implies that the higher the CPU power, the more expensive the route processor on the network element.

• Memory, static and dynamicMemory is required on network elements to store routes and MAC tables in addition to operating system parameters. Assuming the implementation is good, the larger the available memory on the box, the higher the number of routes that can be stored in the network element. The transient memory usage is also important to determine network element load conditions. Better transient usage enables the device to handle a larger churn in the network.

• L3 interfacesThe number of Layer 3 interfaces supported on the network element determines the number of ports and customers that can be connected to this network element for an L3 service. (The total number of L3 customers that can be connected to the network element equals the total number of L3 interfacesthe number of L3 interfaces used to handle core/edge adjacencies and VRFs.) It is common these days to find devices with 3000 to 5000 IP interfaces. In broadband aggregation devices, this number can grow to more than 10,000 interfaces.

Network Parameters

This section discusses network parameters and their scale requirements that must be met by network elements in designing a robust IP NGN.

• L2 VCsIn providing L2 services, L2 VCs provide a measure of the number of L2 circuits that can be transmitted across this device. For a multi-service edge device, this number can be reasonably large with as many as 10,000 VCs supported on a single network element. These L2 VCs can be Frame Relay DLCIs, ATM VCs, Ethernet VLANs, or PPP or HDLC connections.

• L3 sessionsA high number of L3 sessions must be supported to build large networks. For example, a network element can support up to hundreds of IGP adjacencies, thousands of BGP sessions, and thousands of TE tunnels and directed LDP sessions. Today, typical large-scale deployments have seen up to 1000 BGP sessions (eBGP for customer connections), tens of IBGP sessions if full mesh for L3VPNs. The largest IGP seen is 32,000, and the maximum number of IGP neighbors for any given node is 32. These are real provider numbers. For most networks today, the actual numbers might be less. However, for a decision-maker, setting a realistic expectation is important. For example, when evaluating equipment, it is important to buy equipment that provides room for growth, based on expected service demand and current network factors.

  For example, increasing the degree of connectivity for IGP to be greater than 2x is unrealistic. Having IGP prefixes increase by 2x in the network, however, is realistic. Decision-makers must consider all these factors carefully and weigh how much the factors strengthen manageability against equipment scale.

  TE tunnels also need to scale well. For example, with a network of 1000 PEs and a full mesh of TE tunnels, you need 999 TE tunnels from each device to connect to all the other devices. However, you must analyze how realistic this scenario is. Again, the largest deployed number today is 130 tunnel heads and about 3000 mid-points on any given network element. In network design you must consider how long a network element with a capability to signal 1500 to 2000 tunnels will last if the tunnel growth is 2x, or even 3x. Remember, these tunnels are not per subscriber or per user, but per PE-PE or P-P pair for a class of service as a worst-case scenario.

• Number of routesThe number of routes/prefixes supported by a network element is extremely important for deploying a scalable service. Despite the fact that with L3VPNs, equipment scale does not impact the network scale in a negative manner, having too many devices in the network can cause other issues, such as lower convergence and more control plane activity. For example, a network element can support one million VPN prefixes, the Internet table, and a 35,000 to 40,000 IGP table all at the same time.

• MPLS scaleThis is a generic category for MPLS control plane scale and MPLS forwarding scalability. Network elements must be capable of supporting a large number of LDP sessions for L2 scalability. When offering an L2VPN service using AToM (Any Transport over MPLS), LDP signaling must take place between PEs. If the number of PEs is large, say n, then the maximum number of directed LDP sessions required on any given PE is n-1. So, for a 1000 PE network, you need a network element to support 1000 directed LDP sessions. Similarly, based on the degree of connectivity, the number of neighbor sessions required for LDP is in the order of 100. Moreover, the network elements should be able to allocate the full label space for each application. With 20 bits of label space, the maximum number of local labels that can be allocated is one million.

  In reality, no application to date requires the full label space. Remember that the labels are local to the box and are allocated by the box. So in reality, the number of labels needed is equal to the number of connected prefixes, attachment circuits, or TE tunnels. This number is far lower than one million, even for the most densely packed interfaces in a network element. Another element of MPLS scale is the ability to store a large number of labels and process packets for those LSPs. The requirement here, however, is to support a large number of labels. So, if a network element supports one million prefixes, the same network element must also be able to store labels for these one million prefixes and forward packets to them on the label-switched paths. Anything short of that could result in supporting fewer than one million VPN routes in actuality.

Combinations of one or more factors can compound the effect on the router, and scale is affected. For example, in isolation a router might be able to handle 5000 QoS policies. However, when combined with BGP sessions, traffic engineering tunnels the number of QoS policies supported might be lower than in an isolated case. When sizing equipment, you must take these things into consideration. Perhaps the best and surest method for SPs, especially when scale is an issue, is to create a lab environment and test it with the configuration with which the router is going to be deployed.

Network-Wide Scale

Having discussed the various parameters required for the scale of network elements, let us also briefly discuss the network-wide scale issues:

• Similar versus dissimilar network elements (multi-vendor network)Having multi-vendor networks is common these days. However, with multi-vendor networks come interoperability issues. In particular, interoperability issues associated with implementation differences can cause scalability problems at a network-wide level and limit the usage of a device in a certain location or a certain role. For example a device that cannot process many routes and labels for all those routes might have difficulty speaking to another device that can do it. If a network element allocates one label per prefix and another allocates only one label per VRF due to other constraints, some scale issues might exist when these two devices talk to each other. The device that supports per-prefix label allocates many labels, and the device that supports per-VRF labels allocates only far fewer labels. If this device also cannot process many labels, it might not be able to accept all the route advertisements and labels from its peer that is allocating per-prefix labels.

• Full mesh of BGP sessions versus route reflectorsAlthough the detailed technical discussion of this topic is beyond the scope of this book, a brief description is warranted here. Full mesh of BGP sessions allows a faster update of the information between peers. However, the number of peers that can be supported per device limits the total number of routers that are in a BGP mesh. If the networks scale to a large number of PEs, having full mesh of BGP sessions might not be an option. Route reflectors allow reflection of VPN routes and labels to PEs. Because router reflectors exchange control plane function only with PEs, they can potentially peer with a large number of PEs. On the PE side, they need to peer only with a few route reflectors to obtain information about routes and other PEs. This reduces the number of BGP sessions required on the PEs but introduces a BGP hop in the control plane. This means network convergence is slower when compared to the full mesh of BGP sessions, so you are trading convergence for scalability.

• Hub and spoke design versus full meshMost Frame Relay and ATM VPNs are hub and spoke, and with MPLS L3 VPNs, building full mesh connectivity is much easier. However, with large VPNs, some aggregation might be required, hence, a hierarchical design might be neededfor example, a combination of hub and spoke and full mesh design in which remote sites are hubbed to a regional site and regional sites are meshed together.

Management and Scalability

Network elements must support, at a minimum, manageability with MIBs and applications. The degree of support acceptable is dependent on the provider. ILECs and PTTs require full SNMP support and full MIB support with counter information on packets, bytes, and LSPs including per VPN and per class of service. However, network elements must scale well when the network management (NM) stations poll these devices for statistics. The ability to process NM requests accurately and the ability to inform NM stations during failure conditions via SNMP notifications are critical to the operation of the network element.

Other scale issues in managing large networks exist when thousands of sites and thousands of VPNs are deployed. These factors are network and operation dependent. Having a provisioning system that can provision thousands of CEs and PEs to connect the VPN sites together is important. Operators cannot rely on the CLI and show commands anymore. Automated management and monitoring tools become necessarynot just necessary, but mandatoryas networks grow larger.

"I think that this (management systems) is actually the biggest issue in provider scalability. Once the hardware architecture is in place, if the network is designed well, network scalability can be achieved by adding more and/or bigger network routers and ancillary devices. However, if the management systems do not scale, the attempt is usually to try and use manpower to overcome the system's limitations. This usually results in high install failure rates. Managing over 10,000 ports with install rates of 500 to 1000 a month means that, without the proper provisioning and management system, installations are slow and problematic. A solid provisioning and automated service activation system is mandatory. The greater the manual intervention at high install rates, the greater the chance of misconstructed VPNs and dissatisfied customers. A look at recent Telemark studies will show that most providers don't fare well in this area as evaluated by customers." Joe Fusco, BT Infonet

When building IP NGN, decision-makers must evaluate which factors are most important to their operational environment and which factors provide the margin to grow their network up to the next upgrade period. As we all know, in reality no network elements can last through network growth forever, and therefore, a determination must be made on how to strike the right balance. A simple rule is to pick the sweet spot based on operational experience and to maximize the return on investment based on that sweet spot. For example, if 10,000 VCs are provisioned in the current Frame Relay edge switch for subscribers, the multi-service device may also support a similar number of VCs (maybe not the same because the multi-service edge [MSE] network element might also support Layer 3 services at the same time).

Layer 2 VPNsWhat to Expect

Often service providers worry whether they will be able to offer the same grade of service on a packet network with multi-service devices using MPLS that they offer today in the traditional Layer 2 switched networks. This is a valid concern. In addition, enterprise customers also would like to know how their service is being delivered and what to expect from the provider. In this section, we deal briefly with this topic and outline some points that decision-makers must consider to set the right expectations.

Same Grade of Service

Today SPs offer multiple grades of services on the Layer 2 network. These range from zero CIR for FR and unspecified bit rate (UBR) ATM service to well-defined bandwidth bounds using variable bit rate (VBR) or CBR services. The SPs also offer a leased connection with PPP or HDLC framing with a clocking/bandwidth rate on a Layer 1 network. Therefore, the question is whether this all can be emulated easily across a packet network. The answer, however, is not a simple one. Although some services such as UBR service or zero CIR service can easily be emulated, perfectly emulating a CBR service on a packet network is difficult, especially the cell delay variation tolerance (CDVT). However, providing a bandwidth bound equivalent to a VBR service is certainly possible. In small quantities, a CBR service is also possible, depending on the available bandwidth on the links in the network core during transient periods. For example, a provider might be able to get away with offering an ATM CBR service on an MPLS packet network by reserving a good deal of bandwidth to account for transient spikes in the network load or if the core links in the network are lightly loaded even during peak-hour traffic bursts.

Offering bandwidth guarantees on a packet network requires QoS capabilities on the network elements. The ability to police traffic per VC and rate shape traffic becomes important to keep the bandwidth bounds of the service. QoS techniques combined with MPLS TE and maybe even DiffServ TE becomes necessary to build a Layer 2 service with similar QoS guarantees as a traditional Layer 2 network. More details on this can be found in Chapter 4, "Layer 2 VPNs" and Chapter 9, "Quality of Service."

Enterprise customers expect to be able to do the same things as they do on a traditional Layer 2 network. As long as their traffic contracts are met, they do not care how the service is delivered across the network.

Planning and Sizing

With spatial and temporal gambling (also called statistical multiplexing), service providers overbook their capacity or use. With spatial gambling, service providers overbook and assume that not everyone uses the same space. Temporal gambling allows them to overbook the same space in different time slots, assuming not every one uses the network at the same time. This overbooking concept is well understood and is used in sizing the trunk capacity of the Layer 2 network.

The same calculation might not apply to MPLS networks. MPLS networks use load balancing techniques, and they transport Layer 3 traffic. These planning and sizing tools need to be modified to account for the nature of packet networks. We do not mean to scare you by presenting all the issues up front, but our aim is to ensure that you understand the issues involved and find your way around them.

Density

Current Layer 2 switches have a very high density of aggregation of Layer 2 VCs, be they Ethernet VLANs or ATM VCs. Current-generation hardware in the multi-service switches cannot match the same densities of traditional Layer 2 switches for Layer 2 services. The Layer 2 density is built to accommodate Layer 3 services. So, an MSE is not a direct replacement for a traditional switch; hence, the operation model must adjust to the new device densities. However, we have stated earlier that as providers are converging network architectures using MPLS, they are primarily using the cap and grow model, whereby they cap investment on the traditional Layer 2 networks and grow the multi-service network for all services.

Management

Current Layer 2 networks have evolved over a decade. The problems are well understood and there is considerable operational experience today with the current Layer 2 model. The management tools have also evolved to provide robust provisioning, configuration, and fault and diagnostics management. Many providers use these tools to provide fault monitoring and isolation as well as for billing, including usage-based billing. For IP networks, ubiquitous connectivity is more critical than usage-based connectivity. The two models are rather different when it comes to management, and the models have not come together. Network management vendors are making a concerted effort to allow provisioning of L2 and L3 VPNs and to provide a consistent feature set. It is only a matter of time before tools evolve to support both services in a consistent manner. Decision-makers must evaluate what is mandatory and what is highly desired when it comes to Layer 2 services.