6.5 Example resilient network design

6.5 Example resilient network design

Some organizations will go to great lengths to provide complete resilience. It is not unusual for a financial trading network to completely duplicate its floor infrastructure and locate it in separate sites, connected by one or more fast backbones. This is clearly an expensive option, but some organizations simply cannot afford to be offline under any circumstances. Such organizations may be targeted by terrorists and extortionists, since disruption of their networks can be so important as to threaten not only their livelihood but also the regional economy. For these networks it is useful to base the design on military metaphors. All core systems must be duplicated. Command and control systems must be proactive, automated, and fully distributed. The worst-case scenario of a catastrophic site failure must be accounted for in the design.

6.5.1 Backbone design

In our example we illustrate some of the techniques used to design such a fault-tolerant network (note that this design is fictitious and simplified for brevity). In this network we have two major exchanges: New York and London. New York generates 80 percent of the business and is, therefore, considered to be sufficiently important to warrant a mirrored site configuration (XChange-A and XChange-B). Information exchange between London and New York is critical; therefore, both a private backbone and backups via dial-up primary rate ISDN are deployed. Secure VPNs are also configured over the Internet. The VPN is provisioned by a service provider offering a Service-Level Agreement (SLA) with tight bandwidth and availability guarantees. The main backbone design is outlined in Figure 6.17. Note in this figure that both New York exchanges are dual linked by local high-speed Gigabit Ethernet links. Boston and New York are connected via a private T1 bundle, with the option for ISDN top-up and backup via multiple PRI links. New York is dual linked to the Internet, using different service providers for additional resilience.

Figure 6.17: Fault-tolerant backbone design.

Note that this network is expensive and could be optimized further. At current tariff rates the three 2-Mbps international leased circuits alone are likely to incur annual charges in excess of $2 million (excluding any local taxes, such as VAT). Once the basic design is complete, it would be worth checking on further service options, particularly for the most expensive links. Higher-bandwidth leased lines (say a single 6-Mbps service) could be cheaper, and Frame Relay should provide a more flexible and cost-effective solution in this scenario.

6.5.2 Building design

If we now examine New York XChange-A in more detail, we can see in Figure 6.18 that there are high levels of resilience built into a single multistory building. Floor resilience is provided by dual-switching hubs, and wiring at the floor level is interleaved to ensure 50 percent of desk positions are covered in the event of a complete riser failure. Hub resilience is achieved via Spanning Tree protocols. Dual routers are provided for subnet routing and wide area access. The links between routers are in fact live for routing traffic and backup for data (forced using standard routing metrics). In the computer rooms all services are mirrored. Both intranet and Internet access are dual connected. Both UPS and backup DC power are available.

Figure 6.18: Total resilience in a building environment.

Specifically, the following guidelines are in place:

• All key network elements (routers, switches, firewalls, servers) are duplicated.

• Server mirroring is operating between XChange-A and XChange-B.

• All key external communication paths are duplicated.

• Both local and intersite bandwidth are overprovisioned to cope with both fast market conditions and a catastrophic site failure.

• Trader dealing desks are fed from alternate risers and different physical equipment to cope with complete riser or device failure.

• Multiple power sources are deployed. Different power supply companies service the central computer rooms. Backup power is available as short-term UPS battery backup and a longer-term DC diesel generator.

• All routers are running OSPF for intranet applications. This offers fast recovery around failures (typically in the order of seconds).

• As well as being resilient this design is scalable, since performance is balanced between different locations; services appear to be identical externally.

Note that this is only a top-level physical view. We would also need to cover the operation of the command and control protocols running over this infrastructure in much greater depth. A complete design would go into much finer detail and would include a complete survivability analysis together with crisis management plans.

6.5.3 Floor design

The cost of providing complete end-user resilience for a large user population is generally too prohibitive and often leads to an inflexible design that is hard to maintain. As Figure 6.18 illustrates, the user networks at New York XChange-A are diversely routed to different risers with some overlap; the purpose of this overlap will now become clear. For applications such as online trading, it is important that simple network failures do not wipe out whole groups of dealers. Since these users may be physically located in very close proximity, this is a definite possibility, absent careful design.

In this particular design we have chosen to interleave the wiring for desk positions so that no single point of failure will take out a complete trading group (as illustrated in Figure 6.19). The cabling for each adjacent desk position is diversely routed back to different equipment rooms so that any single point of failure in the network would take down only a fixed percentage of dealer terminals. In this example there are in fact four risers and four communications rooms per floor. This means that a single point of failure would take out only an average 25 percent of terminals in any dealing area. Note that this is partly influenced by the number of desks organized into a block. In this example desks are arranged in groups of ten. This means that should switches SW1 or SW2 fail, then three out of ten desks will be lost. Should switches SW3 or SW4 fail, then two out of ten desks will be lost.

Figure 6.19: First floor conceptual wiring plan at New York XChange-A, Floor 1.

One of the problems particular to financial trading networks is fairness. For example, it would not be fair if one trader were to consistently receive share prices several seconds earlier than a colleague. This is a particular problem for the network designer, since the very act of diversely routing transmission paths for high availability may incur additional hops and cable lengths. The fact that different paths are taken also means that the traffic dynamics along those paths are likely to vary. All of these factors affect latency, and it is the designer's job, in cooperation with the users, to provide performance that is within acceptable windows.

6.5.4 Testing the design

A critical part of a high-availability design is the testing phase. While in theory everything may look fine, it is imperative that you test all of the identified failure scenarios in order to establish confidence in the design and to assess the true recovery times. You will need to run live traffic through the network and break components of the design systematically, recording all key data and ensuring that the results are clearly documented.

In practice our example design needs to be explored in considerably more depth. We would need to cover routing infrastructure, dealer applications, external feeds, and traffic engineering problems in considerable detail. However, this should hopefully give a sample of the kinds of issues that need to be addressed when designing any fault-tolerant network.