9.5 Scaling BGP

Now that the concepts and configuration of BGP have been covered, this section will show how to scale BGP as the network grows. Consider that for each IBGP router in the full-mesh, the number of connections that must be active per router is n “ 1, where n is the total number of IBGP routers. If there are 30 routers in a carrier core , each with 29 IBGP sessions running, there are a total of 435 IBGP sessions up at one time [ n ( n “ 1)/2]. Assume that full tables need to be propagated throughout the entire AS, owing to architectural design requirements. Add in several thousand customer routes, and, in short, a lot of data has to traverse your core network just to cover the IBGP routing information. On top of that, you will have some IGP running in the core, as well. That is quite a bit of in- band data, not to mention the complexity of having to troubleshoot the platform and any potential instability issues. When BGP is scaled properly, the potential of an unstable routing domain decreases, while the ease of overall management increases .

To assist in scaling to these design requirements, two techniques are used: route reflectors and confederations. Route reflectors manipulate how IBGP sessions communicate with each other and eliminate the total number of IBGP sessions required to propagate the same information. This is done through a reflector-client relationship. Confederations, on the other hand, take what we know about BGP and allow the same type of implementations , including route reflectors, to be used in smaller, more manageable, private ASs, combining them all into a group of confederations. The real benefit to using route reflectors is that they assist in fixing IBGP full-mesh scaling issues. They are easier to manage in smaller, more static networks. Confederations, on the other hand, are a far superior approach and are very common in provider networks. They are inherently more scalable than route reflectors, as well. Where route reflectors assist in managing the IBGP full-mesh, confederations help to reduce the IBGP full-mesh and can also reduce the load on the IGP.

9.5.1 Route Reflectors

Route reflectors are typically implemented to reduce the overall cost of maintaining an IBGP full-mesh, allowing for scalability to greater proportions . One of the rules in IBGP is that no IBGP router shall advertise routes previously learned from an IBGP neighbor. This works, because each IBGP router has a session to all other IBGP routers (full-mesh). Essentially, all IBGP peers have the same routers as neighbors.

We will refer to Figure 9-26 to discuss IBGP route propagation. We see in this figure that router San Francisco advertises the prefix 172.17/16 to router Denver over an EBGP session. Router Denver is a border router for the AS, so it will run both EBGP and IBGP sessions. Because IBGP is set up with a full-mesh relationship to all other IBGP routers in the domain, it will, in turn , advertise the 172.17/16 prefix to both Washington D.C. and Atlanta. Even though Denver and Atlanta do not have a physical link connecting them, they are IBGP peers due to a logical connection provided by Washington D.C. and the IGP routing domain. Since this is a full-mesh, Washington D.C. and Atlanta received the prefix from Denver. Atlanta will not readvertise to Washington D.C. because of the previously discussed rule on IBGP route advertisements. However, Washington D.C. and Atlanta will receive prefix information from their EBGP peers and will propagate this information throughout the IBGP mesh, as well.

Figure 9-26. IBGP Route Propagation

Next , we want to take a look at Figure 9-27, where the concept changes, and we are able to scale our IBGP domain without the huge increase in session numbers and route advertisements. In this scenario, New York, San Francisco, and Boston are each advertising a network to AS100. We will follow the prefix advertised from San Francisco to Denver again. Denver receives prefix 172.17/16 from San Francisco. Denver, in a normal full-mesh environment, would advertise to all IBGP peers. Since this is a route-reflector scenario, Denver no longer peers with Denver and only peers with Washington D.C. So now Washington D.C., being the route reflector, will readvertise the prefix to Atlanta. For three routers, route reflection is not necessary. However, if there were 40 other IBGP routers in the AS, overall session count would be significantly reduced.

Figure 9-27. Route-Reflector Theory

With route reflection, we introduce a two new attributes into our BGP world: ORIGINATOR_ID and CLUSTER_LIST . These are used as loop avoidance mechanisms.

When a route reflector originates a route, it will set the ORIGINATOR_ID with the ROUTER_ID from where the route was received. The CLUSTER_LIST is a list of route-reflector clusters that a prefix has traversed. The CLUSTER_LIST is updated with a cluster ID that is assigned by the route reflector and unique to each route-reflector cluster. Typically, you want to use the RID or the loopback interface of the local system. According to the rules of BGP, if a route is received and the local system sees its own ID in this field, it will ignore the route. The next example shows the code necessary to set the cluster ID.

 lab@DC# set protocols bgp group  group-name  cluster  number  

Any single AS can implement route reflectors to manage their IBGP topology easily. They are fairly simple to implement and significantly reduce the resources necessary to manage an IBGP routing domain.

Next we will talk about confederations and how they can play a role in our scaling of BGP.

9.5.2 Confederations

Use of confederations is the other common technique for scaling IBGP. Confederations are not as easy to implement as route reflectors and take much time and analysis to implement successfully. However, due to design restrictions or scaling opportunities, confederations may be the best solution. Confederations can be used to enhance the overall simplicity of the AS routing domain. In ISP networks, confederations are by far the preferred method of scalability.

Before we talk directly about confederations, let's look at some of the benefits. When you are dealing with a large provider network or redesigning a large enterprise, confederations can be used to simplify a lot of functions. For instance, the IGP can be better scaled. In a large network, it can sometimes be difficult to manage an IGP's growth, regardless of the simplicity of the actual protocol. With confederations, you can scale the IGP down into smaller, more manageable subdomains. These subdomains can take advantage of their BGP connections to the rest of the AS confederations by advertising fewer routes, summarizing in more places, and making routing easier to administer.

The main reason for using confederations is to scale the BGP implementation. BGP itself is a fairly easy protocol to manage. However, as the BGP domain grows, it easily can become cumbersome. For the same reasons confederations help an IGP scale, it will help BGP scale. You can reduce the number of advertised prefixes within a single confederation AS, thus making routing simpler. The ability to segregate the network addressing scheme into manageable portions becomes easier, as well. The IBGP full-mesh can also be reduced. Some people hesitate to use confederations, perhaps because of a lack of understanding. Confederations work in the same way as BGP, for the most part. Aside from the special rules concerning confederations and route advertisements, it is a lot like setting up small BGP domains.

Now that we have simplified things a bit, remember that confederations are used to help in scaling. Pay attention when configuring for confederations. If this is done improperly, then the likelihood of making a difficult situation worse is fairly certain. If you are lucky enough to get involved in a large network design from the ground up, then you can take advantage of confederations from the start.

Working with confederations requires using EBGP inside an AS. Confederation EBGP sessions are created between each sub-AS boundary router. In addition to the use of EBGP between sub-AS boundary routers, two new path attributes are introduced: AS_CONFED_SEQUENCE and AS_CONFED_SET . Both of these new attributes are defined in RFC 3065. AS_CONFED_SEQUENCE is the ordered set of member ASNs in the local confederation that the UPDATE message has traversed; AS_CONFED_SET is the unordered set of member ASNs in the local confederation that the UPDATE message has traversed. These are used in the same manner as the AS_SET and AS_SEQUENCE ; however, their information relates to the confederations. These attributes are removed prior to sending route information externally from the confederation AS (see Figure 9-28).

Figure 9-28. Confederation Diagram