An Example Site

Introduction

This example site is intended to be generic so that the core architectural components and infrastructure can be illustrated. Nevertheless, it is representative of the key architectural features from many operational sites that we have reviewed. Owners are naturally reluctant to disclose actual details of their sites for competitive and security reasons.

Our example illustrates a large site and demonstrates both topological and component redundancy. It is a highly available system: Critical services can survive most failure modes, short of a major disaster. Servers in each of the ISP 1 through ISP N groupings support each of the site's critical functions, so even the loss of an ISP will not take the site down. Providing nonstop service through most disaster scenarios requires replication of the entire site in multiple geographies (geoplex). Cisco's Distributed Director is commonly used to support the geoplex. Unfortunately, site replication can more than double a site's cost and may introduce data consistency issues for Web applications.

Both smaller as well as much larger sites can be derived from this example. Smaller sites may not require as many servers in each cluster. Those that do not need very high availability need only eliminate the redundant elements, notably the entire top half of the diagram in Figure A.3, starting from Internet ISP 1. Those that do not have high database security can eliminate the secure SQL clusters on the secure network. Much larger sites, on the other hand, can scale significantly by adding:

• Clones to each IIS Web cluster.
• Web clusters.
• Internet connection access points.
• Front-end components, such as firewalls.

Moreover, as the network traffic and the number of managed devices increases, the management network must grow in both size and complexity.

Figure A.3 shows the architecture of the example site.

Figure A.3 Example of a large Web site network topology

In Figure A.3, different line styles, thickness, and annotations show the IP addresses and connections for different parts of the network. In particular:

• External (Internet-facing) network (thin).
• DMZ network (medium).
• Secure (internal) network (thick).
• Management network (thin dashed).
• Cluster heartbeat private network (thin—local to each cluster).
• Connections to the corporate network (lightning bolts).

The remainder of this section provides a tour of the example site, starting from the Internet and working through the DMZ and to the secure network, which includes the corporate network and management network.

Internet

The tour begins with connections to one or more Internet Service Providers (ISPs). Our example illustrates multiple connections for redundancy, labeled ISP 1 and ISP N. These should be provisioned from diverse (physically separate) networks. Domain Name Servers (DNS—not shown in e-commerce in Figure A.3) provide forward and reverse mapping between domain names and one or more TCP/IP addresses. For example, http://www.microsoft.com/ currently maps to the following addresses, each of which is a cluster:

207.46.130.14     207.46.131.28

207.46.130.149    207.46.131.30

207.46.130.150    207.46.131.137

In the event of multiple IP addresses, DNS will resolve successive queries for an IP address for www.microsoft.com by traversing the list—hence the name Round Robin DNS (RRDNS). A disadvantage of RRDNS is that it cannot detect loss of an ISP connection and it will continue to serve up the nonworking IP address. This is not fatal, however, because the user need only request a reload of the Web page. Third-party solutions such as Cisco's Local Director or F5 Networks' BigIP provide more intelligent solutions that route connections dynamically.

DMZ

Servers on front-end networks are exposed to the Internet. Firewalls are essential security components that provide network isolation by filtering traffic by packet type as well as source and destination addresses. They form one boundary of a DMZ (demilitarized zone).

Firewall

The first components in the path are Router/Firewalls, whose functions may be distinct or combined in single devices. Internet-facing routers support the Border Gateway Protocol (http://www.ietf.org/rfc/rfc1654.txt). High-speed front-end switches support connections to each server in the front-end Web clusters. The cross-connection with the Router/Firewalls provide an alternate path in the event of failure of the ISP connection or any of the components in the path.

Front-End Network

The front-end provides the core Web services, such as HTTP/HTTPS, using Microsoft Internet Information Server (IIS) to serve up HTML and ASP pages, and LDAP (Lightweight Directory Access Protocol) servers to authenticate customers. Site Server Commerce Edition may also be loaded on the front-end servers to provide additional database-driven services.

Front-end servers are grouped by service and function—for example, http://www.microsoft.com/, http://search.microsoft.com/, SMTP (email), or FTP (download). SSL service (HTTPS) is similarly segregated from normal HTTP traffic. This allows specially configured servers with expensive hardware security accelerator modules to support high-speed encryption functions. Further, SSL sessions are inherently stateful and may require special failover treatment.

Each of the Web clusters, running Windows 2000 in our example site, employs NLBS (Network Load Balancing Service—in Windows NT this is also known as Windows Load Balancing Service). Each clone is configured identically within each NLBS Web cluster delivering the same content. This provides transparent failover for stateless Web applications, radically improving service availability compared to individual servers. Web clusters support extensive scalability by adding clones to share the cluster's workload.

Client requests are made to each Web cluster using a virtual IP address that all of the front-end servers in the NLBS cluster can respond to. The front-end servers access the site's content data located on back-end clustered file share servers and back-end clustered SQL servers.

All COM objects necessary to provide Web services, including objects called from ASP pages, are installed and registered on each front-end server. ASP pages for the site can be either loaded on the front-end servers' local disks or kept on the back-end cluster file share servers.

Each front-end server is specially hardened for security and connects to three networks:

• Front-end network—Internet access.
• Back-end network—access to DMZ servers and, through inner firewalls, to the Secure Network.
• Management network—supports management and operations functions.

This network segregation improves security while increasing total available bandwidth and improving redundancy.

Note that the only publicly accessible IP addresses on any of the servers in this site are the NLBS virtual IP addresses, to which only the front-end servers can respond. IP filtering applied to Internet-facing NICs (Network Interface Cards) ensures that only the correct type and source of traffic for the functions supported can enter the front-end server. IP forwarding has also been disabled between these networks.

Back-End Network

The back-end network supports all DMZ servers through use of a high-speed, private 10.10.1.x LAN. This architecture prevents direct network access from the Internet to the DMZ servers, even if the firewall were to be breached, because Internet routers are not permitted to forward designated ranges of IP addresses (see "Address Allocation for Private Internets," available at http://www.ietf.org/rfc/rfc1918.txt) including the 10.x.x.x range. As with the front-end network, redundant switches provide access to all front-end and back-end servers. All back-end switches share a common network, so back-end traffic loading can become problematic for active sites, especially if a separate management network is not available for logging and other front-end management network traffic.

The major components on the back-end network are security-hardened server clusters that provide services for storing Web content and temporary persistent state, such as intra-session transactional data (the contents of a shopping cart, for instance). Because all persistent data is available elsewhere, there is no need to provide backup facilities. Scalability is achieved by adding clusters, through partitioning the databases.

These servers employ Microsoft Cluster Service on Windows 2000 to achieve exceptionally high availability with failover capability. The failure of a server does not cause failure of the data services or even interruption in service. When the failed server is placed back online, it resumes delivering data services. Since hard drives can and do fail, the use of RAID drive arrays provides needed redundancy protection for data.

File shares within the cluster support file storage services. Microsoft SQL Server running on the cluster provides database services. Each cluster server employs at least four NICs: one for each switch, one for the private heartbeat LAN (which should use another private network address—for example, 192.168.10.x), and one for the management LAN. In addition to server physical addresses, clusters have multiple virtual IP addresses to support the cluster itself and an address pair (for redundancy) for each clustered service.

Hardened DMZ Utility DC servers support local domain accounts for all DMZ servers, local DHCP and name services (WINS or, preferably, DNS) and local utility file services. One-way trust relationships to internal corporate domains provide authenticated access to secure internal systems.

Secure Network

Another firewall forms the inner boundary for the DMZ and isolates what we term the secure network from the back-end network. The firewall is configured to only allow required communications between permitted port and source/destination pairs. The secure network again comprises a private network (10.10.2.0 in this example), a pair of coupled switches, a variety of servers, and a device labeled VPN/Router that provides connectivity to the internal corporate network. The secure network is logically part of the corporate network. Servers on the secure network are often members of an internal corporate domain, so domain controllers and address and name servers are assumed to be internal.

Other servers may be desired in this section to support additional functionality. There are many possibilities for processing and then moving transactional data between secure data stores and internal systems. Transactions range from traditional synchronous (MTS — Microsoft Transaction Service) to asynchronous (MSMQ — Microsoft Message Queue) to batch-based or e-mail-based store and forward. These are beyond the scope of this document.

However, it is important to note that the Internet, for many organizations, is but one delivery channel among many to provide customer services. As examples, consider a bookseller or a bank. Most business logic and processing take place internally on legacy systems. The Internet solution must interoperate with and serve these existing systems.

Secure Data Stores

The secure SQL clusters are optional and are only required for more complex, transactional sites. They provide high availability, persistent storage for authentication databases, and long-lived transaction storage, and they keep customer information and account data confidential. Unlike server clusters in the DMZ, these servers must be backed up, either using directly connected removable storage devices or through the corporate network. They are otherwise similar to DMZ clusters. Each server again connects to both switches on the secure network for redundancy. Scalability is again achieved through partitioning the database and adding clusters.

Staging Servers

Staging servers appear in the secure network section, although they could be located in the corporate network or even in the DMZ. They receive and stage content from within the corporate network, or from an external content provider, and then deploy content to the Web servers so that the site is in a consistent state. A number of mechanisms are commonly used, including the Microsoft Content Replication System and tools such as RoboCopy.

Corporate Network Connectivity

A device shown as a VPN/Router that connects the site to the corporate network is actually a router that, if required, may incorporate VPN security features to sign and encrypt traffic. Alternatively, using the Windows 2000 built-in IPSec features may add VPN functionality. This supports end-to-end security on an as-needed basis, while eliminating the cost of VPN hardware support.

For a site hosted in a corporate data center, connecting to the corporate network is very simple. In this case, the VPN/Router is connected directly to the corporate network.

Large business sites are frequently hosted remotely from corporate data centers. Dedicated lines generally connect the site to the corporate network, especially if high-performance, low latency access is required. Alternatively, the Internet itself may be used for transport, in which case it is essential to secure all communications using VPN technology.

Management Network Connectivity

We end our tour with a discussion of the management network, which provides the essential capability to monitor and manage the site. For simplicity, we show only computers with LAN connections to a separate management network. These are implemented with separate NICs. Some sites do not employ a separate management network. Instead, they collapse management traffic onto the back-end network. We do not recommend this practice for security, network loading, and management reasons.

Not shown are management connections to routers, switches, and firewalls. Also not shown are serial port dial-up connections, used for emergency out-of-band (OOB) access. Their absence does not imply they are not needed. When the management network (or the back-end network that is used for management) is unavailable, each host can still be accessed using this setup.

Summary

The following sections use the model described in the previous example to discuss in detail how the architecture described in this document meets these four goals:

• Linear scalability: continuous growth to meet user demand and business complexity.
• Continuous service availability: using redundancy and functional specialization to contain faults.
• Security of data and infrastructure: protecting data and infrastructure from malicious attacks or theft.
• Ease and completeness of management: ensuring that operations can match growth.