Internetworking is geographical by nature, so most design practices have to do with matching topology to needs. (Recall from Chapter 5 that a topology is a map of an internetwork's physical layout.) The layout of an internetwork largely dictates how it will perform and how well it can scale. In networking, scale, or scalability, means how much an internetwork can grow without having to change the basic shape of its topology (that is, without having to replace or excessively reconfigure an existing infrastructure).
One last run-through of what we've learned thus far is in order. Doing so is especially important here, because in this chapter we'll be looking at a variety of design factors and options. Therefore, we need to be clear on the various components that make up an internetwork.
Switches form LAN segments, the basic building block of every internetwork. Each switch port is a segment brought through a process called microsegmentation. A LAN segment could be a departmental LAN or a high-speed LAN backbone servicing dozens of other LAN segments within an enterprise.
To review, a LAN segment is a physical medium shared among a group of devices. Most LAN segments are formed by hubs or switches. Strictly speaking, a LAN segment is a LAN. Usually, though, the term LAN is used to refer to a local network consisting of many LAN segments.
A collision domain is a shared network medium in which Ethernet packets are allowed to collide; a broadcast domain is the area within which messages may be sent to all stations using a so-called broadcast address. Collision domains should be kept small because collisions limit the use of bandwidth. The more hosts that are connected to a LAN segment, the slower the traffic moves.
Most collision domains are formed by hubs that connect host devices to the internetwork. Hubs are the functional equivalent of the Ethernet cable segments used in the early days of local area networking. Switches segment collision domains on each switch port.
Some broadcasts are useful, but too many can bog down a network in useless overhead traffic-an unwelcome phenomenon called a broadcast storm. Broadcast domains are, by default, the same as a network's collision domain for shared media (in other words, hubs), but the scope of broadcast addresses can be made smaller than a collision domain using switches. Routers normally limit broadcasts, but a broadcast domain can be extended by configuring a router to let broadcast messages pass.
Switches also connect hosts to networks, but in a fundamentally different way. A switch "time slices" network access among its attached hosts in such a way that each switch port forms a channel with a collision domain of one. This is called switched bandwidth, as opposed to the shared bandwidth of hubs. Switched networks are estimated to be ten times faster than shared networks over the same medium.
Moreover, switched networks support virtual LANs (VLANs), enabling admin istrators to group users rationally instead of being forced to group them according to the host devices to which they are attached.
In addition to forming them by connecting hosts, larger switches connect LAN segments to form internetworks. To avoid confusion, the two types of switches are sometimes called access switches and LAN (or backbone) switches.
The third basic device in internetworking is the router. Routers connect LAN segments instead of connecting hosts, as do hubs and access switches. Routers are used to isolate intramural traffic and to provide internal security. In addition, they can extend broadcast and multicast domains to specified LAN segments to help bind those networks into a functional unit.
Routers are deployed both inside internetworks and at the edge of autonomous systems. Inside routers are sometimes called internal routers or access routers. Routers that concentrate on communicating with the outside are called edge, or gateway, routers. For example, an Internet service provider (ISP) will use gateway routers to connect to the Internet. By contrast, a big company will place at least one internal router at each of its major sites to help manage in-house traffic.
Routers are more intelligent than hubs and switches because they are able to interpret network addresses. They read network addresses in order to filter traffic, control access to networks or services, and choose the best path to reach a destination. Routers bring internetworks to life. It's no coincidence that the three most basic devices in internetworking operate at different levels of the seven-layer OSI reference model, as illustrated here:
Routers operate at the network layer (layer 3). Today, most internetworks use IP network addresses-all Internet routers do. But many internal routers must still use legacy desktop protocols such as IPX, AppleTalk, or DECnet. For that reason, Cisco and its competitors have invested heavily in engineering multiprotocol products to allow legacy LANs to interoperate with IP. Cisco's IOS feature sets exist mostly to give network designers options in purchasing system software that fits their network protocol needs.
Whether IP or a legacy layer-3 protocol, network addresses are inherently hierarchical. One way to look at it is that as a router works its way rightward through an IP address, it zeros in on the LAN segment to which the destination host is attached. Over long-haul routes, moves through the address are manifested in hops between routers. A one-hop route would require only finding the LAN segment on which the destination resides.
Routes are often summarized before being shared with other routers. This improves performance by greatly reducing the number of address entries carried inside a router's route table. Route summarization, called route aggregation, works by relying on a gateway router to know the target LAN segment's full address and allowing interim routers to carry fewer, summarized entries in their respective routing tables, thereby improving performance. Address translation is also frequently used, where internal addresses are either altered or grouped into a global address in packets sent outside an internetwork. Mechanisms such as Port Address Translation (PAT) and Network Address Translation (NAT) are used at edge routers or firewalls to make these translations in the packet address fields in both directions.
Switches operate at the data-link layer (layer 2), dealing in MAC addresses instead of network addresses. A MAC address is a long number that uniquely identifies physical hardware devices. MACs combine a manufacturer code with a serial number. Even routers use MAC addresses for a message's last step-resolving an IP address to the physical MAC address to locate the destination host within the LAN segment.
MAC addresses are topologically flat. The logical profile of a MAC address appears as if all hosts are connected to the same cable; it offers no clue as to where hosts are located because it's basically a serial number. Switched networks, therefore, must operate by brute force, flooding broadcasts of MAC addresses to all ports when a destination MAC is unknown.
VLANs give switched networks hierarchy by limiting broadcasts to discrete groups of users. This combines the speed of switched bandwidth with the hierarchical topology heretofore available only in shared bandwidth networks. In addition, VLANs flexibly assign users to logical workgroups instead of having to group users by device.
Internetworks use control protocols to route messages. There is so much dynamic change in internetworks-through growth, changing traffic patterns, a device going down, and the like-that they must self-operate to some degree by constantly updating device routing tables. Routed networks rely on routing protocols to keep track of paths through internetworks. For example, many small internetworks use RIP 2; most large ones use EIGRP or OSPF (EIGRP is Cisco proprietary; OSPF is an open standard). These trade in lists of routes, mostly within an autonomous system, and are used to connect LAN segments. BGP trades in lists of autonomous systems and is used to connect the Internet.
As you've learned, internetworks maintain a degree of self-awareness by way of discovery protocols, which find new devices and keep checking the status of known ones. These protocols-Cisco Discovery Protocol (CDP) is an example-are the supporting cast to routing protocols. When an event takes place, it's discovered and the news is passed around until the device population converges on a new list of routes. Sometimes loops appear where a suggested route turns back toward its source device, creating nonsensical routes that can slow or even crash an internetwork. Routing protocols use metrics to tune internetworks. RIP uses only hop count, but the more sophisticated protocols use several metrics that can be combined into a weighted matrix to steer traffic along desired links.
Switched networks aren't so sophisticated. Switches maintain only lists of MAC addresses, with the most recently used MACs appearing toward the top, the highest one being the first choice. Switched networks use the Spanning Tree Protocol (STP) to prevent loops.
In just the past few years, the design requirements of the typical enterprise have changed radically. These changes have occurred at opposite ends of the topology. At the bottom, segmentation using hubs with access switches has greatly increased the number of LAN segments and, therefore, the amount of traffic to go over the backbone between segments. At the top, whole new computing architectures are becoming standard, with Web-based intranets replacing traditional client-server management systems, extranets transforming traditional electronic data interchange (EDI) systems, and virtual private networks (VPNs) replacing leased-line wide area networks (WANs).
Driving even more change is the fact that new network applications have changed traffic characteristics. For example, videoconferencing is becoming popular, increasing the need for configurations optimized to handle multicasts-where a single copy of a message is forwarded to a subset of destination hosts.
Hierarchical topologies are inherently better than flat ones for a number of reasons, the main one being that hierarchy contains traffic to its local area. The rule of thumb designers use is that broadcast traffic should not exceed 20 percent of the packets going over each link-the implication being that segmentation will naturally boost throughput by isolating traffic to its most likely users. This rule of thumb applies only to the amount of broadcast packets in the traffic mix, and is not to be confused with the 80/20 rule. The 80/20 rule states that 80 percent of all traffic stays home and only 20 percent goes beyond the local area.
A flat topology-one in which each device does more or less the same job-increases the number of neighbors with which an individual device must communicate. This increases somewhat the amount of payload traffic the device is likely to carry and greatly increases overhead traffic. For example, each time a router receives a broadcast message, its CPU is interrupted. For many small internetworks, a flat topology is sufficient, and the added expense and complexity that hierarchy requires isn't warranted. But it doesn't take many LAN segments to hurt an internetwork's performance and reliability, with devices and hosts bogged down in unnecessary traffic.
This is why the industry adheres to a classical hierarchical design model. The model has three layers: the access, distribution, and core layers. This separates local traffic from high-volume traffic passing between LAN segments and areas, and lets network devices at each layer concentrate on doing their specific job. The hierarchical model is depicted in Figure 14-1.
Figure 14-1: The classical three-layer hierarchical topology is based on segmentation
Hierarchy is made possible by segmentation-the practice of dividing hosts into smaller LAN segments. Fifteen years ago, most LAN segments were actual cable spans running through walls and ceiling plenums. Today, most are formed by "cable-in-abox" hubs and access switches. Segmentation and hierarchical topology yield several benefits:
Performance Traffic is isolated to source areas, thereby narrowing Ethernet packet collision domains and speeding throughput.
Reliability Most faults are isolated to the segment from which the problem originated.
Simplicity By separating dissimilar areas, network elements can be replicated as needed throughout the internetwork.
Scalability Modular design elements can be added as the internetwork grows over time, with minimal disruption of existing networks.
Security Access can be controlled at well-defined junctures between the layers.
Internetworks naturally tend toward a two-level hierarchy. Hubs and switches connect host devices into LAN segments, and the backbone connects the segments into a local network, whether within a floor, building, office campus, or even a metropolitan area. This is a relatively flat topology in the sense that, even though collision domains are limited, excessive broadcast traffic still chews into available bandwidth. This makes the distribution layer the key. By isolating traffic, the distribution layer also isolates problems and complexity.
Hierarchy also helps reduce costs. By dividing hosts and traffic, variations are limited to fewer LAN segments, or even to a single segment. Variations include such things as desktop protocols (IP, IPX, AppleTalk), traffic volumes (workgroup versus backbone), and traffic type (big graphical files, e-mail, HTTP). Hierarchy allows the network designer to tune the configuration for the particular job at hand. Adjustments are made in the model of network device purchased and in how it is configured in terms of memory, modules, software, and config file parameter settings.
The access layer is made up mostly of hubs and switches, which serve to segment host devices, such as PCs and servers, into many LAN segments made up of either shared or switched bandwidth. This is where MAC-layer filtering can take place.
If an internetwork has remote sites, such as branch offices or home offices, the access layer would also include access servers and access routers. WANs must use some type of long-distance transmission medium. There is a wide selection of media now, such as leased digital T1 or T3 lines and Frame Relay public digital networks. Dial-in remote users employ analog modem lines and, in certain areas, higher-bandwidth technologies, such as Digital Subscriber Line (DSL) and Integrated Services Digital Network (ISDN). Figure 14-2 shows access-layer functionality.
Figure 14-2: The access layer provides both local and remote connectivity to hosts
In large internetworks, the access layer can include routers. These internal routers serve mostly to isolate overhead, control traffic, and enhance internal security. The access layer encompasses a mix of technologies in most internetworks. Dial-ondemand routing (DDR) has become popular for remote connections, because it keeps a link inactive except when traffic needs to be sent, thereby reducing telecommunication costs.
Most enterprises have legacy technologies that are being gradually phased out as new ones are implemented. For example, many big companies still use their leased-line T1 WANs alongside growing VPNs, substituting shared network usage for dedicated leased lines. From a practical standpoint, this is necessary because the routers must be upgraded along each VPN link.
The distribution layer is made up mostly of routers and layer-3 switches. They're used to separate slow-speed local traffic from the high-speed backbone. Traffic at the access layer tends to be bandwidth-intensive because that's where most LAN and host addresses reside. Network overhead protocol traffic for discovery protocols, SNMP, routing protocols, and other network control systems is heavier at the access layer.
Because routers are intelligent enough to read network addresses and examine packets, they also improve performance by sending traffic as directly as possible to its destination. For example, distribution-layer routers define broadcast and multicast domains across LAN segments. Domains are, by default, limited to LAN segments; routers can extend domains across segments as the hierarchy design dictates. Figure 14-3 depicts distribution-layer functionality.
Figure 14-3: The distribution layer is the key to providing a functional hierarchy
In configurations using multilayer switches, distribution-layer devices route messages between VLANs. Multilayer switching is a technology in which packets are filtered and forwarded based on both MAC and network addresses. The Catalyst 6500 is perhaps the best example of a multilayer switch, incorporating the Multilayer Switch Feature Card (MSFC2) in addition to those with typical switch electronics.
Most value-added services are provided by devices at the distribution layer. Address translation takes place at this layer, usually on a gateway router or a firewall (itself a type of router). Address aggregation also takes place here, as well as area aggregation if the internetwork is running OSPF routing domains. Other services are also performed on distribution-layer routers: translation between protocols such as IPX and IP; encryption for VPN tunneling; traffic-based security using access lists and context-based firewall algorithms; and user-based security using security protocols such as RADIUS, TACACS+, and Kerberos.
The core layer is the backbone layer. In large internetworks, the core incorporates multiple backbones, from campus backbone LANs up through regional ones. Sometimes, special backbone LANs are configured to handle a specific protocol or particularly sensitive traffic. Most backbones exist to connect LAN segments, usually those within a particular building or office campus. Figure 14-4 depicts how the core layer might look in a typical large-enterprise internetwork.
Figure 14-4: The core layer includes campus LAN backbones and WAN backbones
To run fast, a backbone LAN should be configured to experience a minimum of interruptions. The goal is to have as many backbone device CPU cycles as possible spent transferring packets among segments. The distribution layer makes this possible by connecting workgroup LAN segments and providing value-added routing services. A minimum of packet manipulation should occur at this level. This is why most new backbones are switched LANs. The need for address interpretation at the core is minimized by the processing already performed by distribution-layer routers, so why not use switching technology to move data over the backbone much faster?
ATM (Asynchronous Transfer Mode) and Gigabit Ethernet battled to become the switched backbone technology of choice. ATM had an edge for multimedia applications because it uses fixed-sized cells instead of Ethernet's variable-length packets. The obvious advantage of Gigabit Ethernet switched backbones is easier compatibility with the millions of Ethernet LANs already installed throughout the world. In the end, however, Gigabit Ethernet won the war.
ATM is an international cell relay standard for service types such as video, voice, and data. The fixed-length 53-byte cells speed data transfer by allowing processing to occur in hardware. Although ATM products exist to take data all the way to the desktop, the technology is optimized to work with high-speed transmission media such as OC-48 (2.5 Gbps), T3 (45 Mbps), and T3's European counterpart, E3 (34 Mbps).
Over the years, the networking industry has developed a set of concepts and best practices for use in internetwork design. Most internetworks are works in progress; very few are designed from a clean sheet of paper. As internetwork topologies evolve through time and circumstance, it becomes difficult to maintain a rigorous hierarchical network design-especially in large enterprises with distributed management structures or in shops that have high personnel turnover in their network teams.
Redundancy is the practice of configuring backup equipment. This is done to provide fault tolerance, where traffic will shift to the backup device if the primary unit fails, a process called failover. For example, most high-speed backbones have dual-configured switches at each end in case the primary switch goes down. Another common safeguard is to have redundant power supplies within a device, so that if one fails, the device keeps running.
Cisco's technology to support redundancy is the Hot Standby Router Protocol (HSRP), a suite of commands in IOS. Hot standby is a computer industry term meaning that the backup unit is always up and running, thereby allowing automatic failover in the event of a failure. HSRP works by creating a group of routers where one is elected as the active router and another is elected as the standby, or "phantom," router. They all share a virtual IP and MAC address that the active router will serve. The active router is monitored by others in the group, and should it fail, the standby router will take over the traffic processing duties, and another backup router (if more than two) will be elected as the new standby router. Failovers are achieved with no human intervention and are generally accomplished in a few seconds.
Because redundant configurations are expensive, fault-tolerant configurations are usually limited to critical devices. Redundancy is most commonly configured into backbone devices and firewalls, where device failure would have the broadest effect on the overall network.
Load balancing is a configuration technique that shifts traffic to an alternative link if a certain threshold is exceeded on the primary link. Load balancing can be achieved through various means, such as tuning routing metrics in router config files within routing protocol domains.
Load balancing is similar to redundancy in that an event causes traffic to shift directions, and alternative equipment must be present in the configuration. But in load balancing, the alternative equipment isn't necessarily redundant equipment that only operates in the event of failure.
A good design will incorporate a meshed topology to achieve redundancy and load balancing. A mesh is where two network devices-usually routers or switches-are directly connected. In a fully meshed topology, all network nodes have either physical or virtual circuits connecting them to every other node in the internetwork. You can also have a partially meshed topology, in which some parts of the topology are fully meshed but some nodes are connected only to one or two other nodes. Figure 14-5 depicts the two.
Figure 14-5: Fully meshed and partially meshed topologies each offer their pros and cons
At first blush, all meshing seems to be an inherently good thing. Looking at the example in Figure 14-5, you can readily see the benefits in the full-mesh topology:
Performance It's only a single hop to any network attached to one of the other routers, and the fewer the hops, the faster the speed.
Availability Having redundant paths means that if any one router goes down, one or more alternate routes are always available.
Load balancing Alternative paths can also be used for normal operations, where routing parameters can be configured to use alternate paths if a preset traffic load is exceeded on the primary router.
The partially meshed internetwork on the bottom of Figure 14-5 doesn't have these advantages. For example, to go from router A to C takes two router hops, not one. If routers on both sides of router F go down, it will be unable to communicate with the rest of the internetwork. Also, fewer mesh connections reduce opportunities for load balancing. However, although meshing can bring benefits, it must be used carefully, because it comes at the following costs:
Expense Every router (or switch) interface dedicated to meshing is one that can't be used to connect a LAN segment. Meshing consumes hardware capacity.
Overhead traffic Devices constantly advertise their services to one another. The more mesh links a device has, the more advertisement packets it broadcasts, thereby eating into payload bandwidth.
Vulnerability Meshing makes it more difficult to contain problems within a local area. If a misconfigured device begins propagating indiscriminate broadcast messages, for example, each element in a mesh will cause the broadcast storm to radiate farther from the source.
Complexity Additional connections make it more difficult to isolate problems. For example, it would be harder to track down the device causing the broadcast storm in a fully or heavily meshed internetwork, because there would be so many trails to follow.
For these reasons, few internetworks are fully meshed. The general practice is to fully mesh the backbone portion of topologies to provide fault tolerance and load balancing along these critical links, but only partially mesh the access and distribution layer topologies.
Circumstance sometimes dictates deviating from the strict hierarchical model. The two most common topology deviations are so-called backdoors and chains. A backdoor is any direct connection between devices at the same layer, usually the access layer. A chain is the addition of one or more layers below the access layer. Figure 14-6 depicts the two.
Figure 14-6: Backdoors and chains violate the ideal three-layer hierarchical topology
Sometimes, it makes sense to configure a backdoor. For example, you might want to directly link two remote sites if the links to the distribution-layer routers are costly or slow. Backdoors also provide a degree of redundancy: If the backdoor link goes down, the two remote sites can failover to the distribution-layer router and keep communicating. More often than not, however, backdoors and chains emerge because of poor network planning or a renegade manager who installs networking equipment without involving the network team.