3.2 The Big Picture - Applications, Member Systems, and the Goal of the Cluster

3.2 The Big Picture – Applications, Member Systems, and the Goal of the Cluster

The first task is to determine the member systems and their configuration as computing resources. This is based on the performance needs of the application(s), the need to ensure capacity for future application growth and to ensure the overcapacity required for availability. This section will discuss meeting these needs by posing a series of questions:

• What are the applications?

• Are multiple applications going to be run in the cluster?

• Are any of the applications parallel (multi-instance)?

• What are the computing resource requirements of each application?

• Will multiple applications run on a single member?

• Is there room to scale out the cluster's size in terms of number of members?

• Is there room to scale up an individual member's computing capacity?

• Will there be homogeneity of member systems (will the systems be similar)?

• Is there required overcapacity in terms of computing resources available in the cluster for failover?

Be forewarned, the process of sizing the basic member systems is often an iterative one as different goals and options are considered.

3.2.1 What Are The Applications?

Sometimes we get caught up in the component technology of a cluster. To state the obvious, therefore, a solution is configured to run applications. So the starting point of designing a TruCluster Server cluster-based solution is to specify the applications and then determine the basic platform or platforms and their resources required to run the applications at the desired performance levels.

So what are the applications intended for your cluster? The variety of possible applications is as diverse as that for a standalone server – a Database server, WebServer, Application Server, Compute Server, NFS Server, Mail server, Network Services Server such as DHCP, DNS, BIND etc., and many more possibilities.

3.2.2 Are Multiple Applications Going to be Run in the Cluster?

One important issue is whether more than one application will exist in the cluster. Just like a standalone platform, a cluster can host multiple applications. This can have both up and downsides. On the benefit side, running multiple applications within the cluster can be viewed as a server consolidation strategy. On the negative side, multiple applications within the cluster can make it more complex and error prone to manage.

Running multiple applications on a single large system is commonly referred to as "Server Consolidation." Most system vendors including HP with its Alpha-based GS series and PA-RISC-based Superdomes have implemented special facilities in system partitioning and resource management to compartmentalize the running of multiple applications on a single system to increase robustness, control, and predictability. In a sense, a cluster has its own feature to meet these goals. Each member system can run a specific application or applications of the total workload and be its own unit of resource application, control, and robustness.

Multiple applications do not come without a downside. They will require a larger and more complex environment – and size and complexity are enemies of stability. For instance, applications may have very different system resource needs. If they run on the same cluster member, then it can be difficult to tune and size that member for the combination of the workloads, which may have diametrically opposed optimal tuning requirements. Running workloads with drastically different characteristics on different members solves the immediate problem, but what about when failover is required? If the member systems are then tuned or sized for a specific application, then when failover occurs, behavior will be sub-optimal.

Even if you think you are running one application, be sure to outline all the minor pieces of the application solution that could be thought of separately. For instance, an application where users first log in to the host and then use tools to access a database could be thought of as two pieces, the user login and the database server. Another example would be backup of the data being used by the application. If the data in the cluster is small or unchanging, then backup could be a secondary consideration. On the other hand, if backup requires large resources and time, it may be wise to list it as a separate application.

3.2.2.1 Example

For the Biotech research cluster project, the following applications are identified for the new cluster:

• Home written compute application

• Small standard database application

• Web server application

• User logins for development environment and job dispatching

3.2.3 Are Any of the Applications Parallel (Multi-Instance)?

Parallel applications run as separate instances on multiple cluster members. Together, the instances cooperate to do the work of the application utilizing the resources on each of the cluster member systems on which they run. Multi-instances versions of applications are typically offered as a performance enhancement feature over the single-instance version of the same product. To achieve more throughput or processing by the application, simply start additional instances on additional members in the cluster. In addition to performance benefits, multi-instance applications can also offer faster failover. Rather then having to cold-start the application on a surviving member (when a traditional single-instance application or the member it runs on fails), for multi-instance applications, an instance is already running on additional members at the time any given instance or its member host fails.

The common example for TruCluster Server would be Oracle Server and Oracle RAC^[3]. Oracle Server is the standard single-instance version of the database product. Oracle RAC is the parallel multi-instance implementation. Although Oracle RAC is the most common case, there are other parallel applications including:

• NFS Serving

  In a TruCluster Server, all members can be used to export files over NFS to NFS clients mounting exported file systems from the cluster.

• Web Serving

  Popular Web Servers can be configured to run on multiple members exporting the same HTML^[4]files from a common file system to browsers running on clients.

• Advanced Server for UNIX (ASU)

  The HP product for Serving PC file systems and NT Domains from Tru64 UNIX can be configured as single-instance or multi-instance in a cluster.

• NIS Server

  Once built, all NIS servers on all members will export NIS data to clients.

• Multiple user logins

  In a TruCluster Server cluster, the common shared file system means that users can be logged in to multiple cluster members and see the same home directories and shell configuration.

There are some potential downsides for multi-instance applications. The ability of a multi-instance application to scale is dependent on the workload that it is given. The workload you are presenting to the application may not run any faster with multiple instances. The second is that the application vendor may charge a premium in terms of license fee for a multi-instance version of the applications. Finally, the additional complexity of running multiple instances can increase administrative overhead over that of traditional single-instance applications. In the case of TruCluster Server, this has been minimized with the single common file system. In the case of Oracle, the database itself can be placed in the common file system rather than on raw devices. Please refer to Chapter 13 for more on the cluster file system.

At this step in the cluster solution design process, we would like to simply mark which applications are potentially multi-instance. Note that depending on the workload, etc., it may not be worth the additional complexity.

3.2.3.1 Example

Which of the applications planned for the Biotech cluster could potentially run as multi-instance and which, if any, have been chosen by the site to run multi-instance?

• Home written compute application

  This application is launched in a batch manner and accesses a large collection of input files in read-only mode. It then writes to an output file that can be specified as an argument to the process. The application is inherently multi-instance as long as separate output files are specified for each instance. Multiple instances of the application can be started on a single system, or in the case of the cluster, on multiple members sharing access to common input data and each having its own private output file. These private output files, such as log files, can be implemented by way of Context Dependent Symbolic Links (CDSLs), which are discussed in Chapters 6 and 12. The decision to run the application as a single-instance on a single large member system or multi-instance on multiple smaller systems will come down to the cost of the systems and the scaling across multiple members to obtain the best price-to-performance ratio.

• Small industry standard database application

  The database vendor supplies a multi-instance as well as single-instance version of their product. For this particular usage, the database is small and has very low performance requirements. It is unlikely that going with a multi-instance configuration is worth the additional cost and complexity.

• Web Server application

  The Web Server in question exports a series of read-only HTML files and also has active content to query or update the database mentioned previously. The Web Server could be run multi-instance because the HTML files are read-only, and the database in the backend can guarantee transactional properties if multiple queries or updates come in simultaneously from multiple Web Server instances. On the other hand, the amount of traffic and performance needs are such that it is unlikely that multiple Web Servers will be needed.

• User logins

  User logins could be made to be multi-instance. As long as a user can log in to any system and see the same home directories, shells, and tools, then it doesn't matter whether a group of active logged in users are all on one member system or spread over multiple member systems. The major activity of the logged in users is to submit computational jobs. In addition, some users also access the database or work on developing the applications. For this cluster, the user application could be deployed on a single host or multiple hosts. The final decision can be based on cost-effectiveness by comparing the price of one larger platform with multiple smaller platforms.

3.2.4 How Much Computing Resources Do The Applications Require?

This must be done for each application individually. If the single application were running on a standalone platform (or multiple, if multi-instance), what class, model, and configuration of system(s) would be required to meet its performance needs? The details include CPU, Memory, Network I/O Bandwidth, Storage I/O Bandwidth, and total Storage.

Application sizing performance for a platform is not always easy. This task is independent of the cluster technology except for the scaling factor as the member count is increased for multi-instance applications. For some standard popular software applications, such as Oracle, some references exist. In other cases, sizing a new platform for an application can be based on the site's experience running the application on an existing smaller or older platform.

3.2.4.1 Example

Like many sites, the Biotech Company has previous experience running either the same or the current incarnation of the applications on standalone servers in the environment. To size the new platforms for the new cluster, they will start out by studying the performance resource consumption rates on the current environments. Next, they will evaluate the latest server models and choose memory and cpu counts for the servers in the new cluster.

• Home written compute application

  If run on a single system, it would require a GS160 class system. The GS160 would require twelve CPUs, 32GBs of Memory, six HBAs, and approximately one TB of storage. Alternatively, the application could be split over multiple systems in a cluster, the best option being three ES45 class systems. Each of these servers would have four CPUs, 8GB of Memory, and two HBAs. The network interface requirements are low. A single, logical interface implemented with two Fast Ethernet physical NICs would be adequate.

• Small standard database application

  Assuming it is run on a single node as a single instance, a DS20 system with two CPUs, 4GB of memory, and two HBAs would be adequate. The total storage required is 400 GB. Network interface requirements would be low. A single logical interface implemented with two Fast Ethernet physical NICs would be adequate.

• Web Server application

  Assuming it is run on a single node as a single instance, a DS10 system with one CPU, 2GB of memory, and two HBAs would be adequate. The total storage required is 200 GB. Network interface requirements would be low. A single logical interface implemented with two Fast Ethernet physical NICs would be adequate.

• User logins and development environment application

  If run on a single system, a single ES45 with four CPUs, 8GB of memory, and two HBAs would be more then adequate. The total storage required would be 400 GB. Network interface requirements would be moderate. A single logical interface implemented with two Fast Ethernet physical NICs would be adequate.

3.2.5 Will Multiple Applications Run on a Single Member?

When we plan how the applications will be mapped to cluster members, we have the choice of dedicating specific members to a specific application or running multiple applications on a single cluster member. One to one mapping of applications to servers in a cluster (or standalone servers) can simplify tuning, support, and computing resource management for the members. On the other hand, maximizing the investment in platforms is an incentive to run multiple applications on the same box.

This decision can have a lot to do with the nature of the applications and the services available in the operating system to simplify and control the running of multiple applications using resource management.

• Do the applications have extreme resource consumption patterns that make sharing a host difficult to optimize?

• Would utilizing the Tru64 UNIX Operating system's resource management facilities (including processor sets, class scheduling, and the ARMTech fair share scheduler) simplify resource management on a server?

• Would multiple applications conflict because they run at different times of the day?

• Even if a set of applications can share a host, how does the combination of applications affect the system if they were to failover to another cluster member, which is already running a different set of jobs?

• Would the doubling up of applications allow the overall cluster to be smaller, thus simplifying the cluster at large configuration and management?

3.2.5.1 Example

In the case of the Biotech cluster, some of the applications could be run on the same member without major risk:

• The web server and database can run on a common platform because neither is expected to be an excessive resource consumer. Also, the web server interfaces with the database so running them on the same platform will improve the performance seen by the web server's clients.

After this, the resulting workloads could be reduced to:

• Home written compute application

  No change from earlier description.

• Small standard database application and Web Server Application

  A single ES40 system with four CPUs, 8 GB of memory, two HBAs, and 600GB of storage would be adequate.

• User logins and development environment application

  If run on a single system, one ES45 with four CPUs, 8GB of memory, and two HBAs would be more than adequate. The total storage required would be 400GB. Network interface requirements would be moderate. A single logical interface implemented with two Fast Ethernet physical NICs would be adequate.

3.2.6 Will There Be Homogeneity of Member Systems?

The TruCluster Server software supports homogeneous or heterogeneous cluster configurations. Homogeneous clusters have members that consist of the same model and computing resources and have the advantage of simplifying management of the cluster in a number of ways. First, the site does not need to worry about keeping track of which systems have which computing resources when assigning applications within the cluster either statically or when failover occurs. Moreover, tuning and hardware and software maintenance through patches can be more standardized. Heterogeneous environments can be more cost-effective as the site either pulls together existing hardware of different model types to make the cluster, or exactly matches systems to the applications when they are drastically different needs.

3.2.6.1 Example

In the Biotech cluster, no legacy servers will be included in the cluster, and the ease of management of a homogeneous cluster is deemed a priority so the design will center on a homogeneous configuration.

3 ES45s	– Compute application
1 ES45	– Database and Web Server
1 ES45	– User Logins

Note that in the case of the Database and Web Server system, the server will have more power than is absolutely required by the application in the interest of keeping the cluster homogeneous.

3.2.7 Is Availability Insured By Providing Additional Computing Capacity?

Almost all cluster solutions have high availability as a design criterion. In this case, the cluster must have enough spare computing power so that if a member fails, the applications running on it can be moved to another cluster member and continue to run. The major question is what level of service is required for the application when moved to a new member. If it is the same as before the original member failed, then free computing capacity of the same amount as the failed member must be available to failover to. For example, if the cluster has one application running on one member, another member of the same size is required to meet this requirement. The key is that this additional member is idle until such time as the first member fails, causing the application to failover. Having 100% idle computing power is the worst case and can be reduced if you do the following:

• As cluster size increases, the overhead of the spare capacity can be amortized over a number of members. For instance, if the cluster has two applications on two members, it requires a third idle member to take either of the two applications if the current member fails.

• The site may be willing to live with degraded level of service or the termination of less essential applications when a failure condition exists. For instance, a two-node cluster with two applications could either allow both applications to run on a single member when one fails with degraded performance, or terminate the less important application when failover occurs.

Complicating all of this failover planning is the computing resources required by the applications of the cluster and whether or not the cluster will be homogeneous in member systems. For instance, if one workload requires an extremely large server, and other applications in the cluster do not, an additional large server would be required for failover of the large application – independent of the smaller needs of the remaining applications.

3.2.7.1 Example

The Biotech cluster design has already been narrowed down to a homogeneous cluster. This makes failover planning easier. The minimum number of members to support the applications without considering failover is already 5. The cost of adding a 6^th member as a failover host is not excessive to the site's plans so that solution will be taken. This results in a cluster as follows:

3 ES45s	– Compute application
1 ES45	– Database and Web Server
1 ES45	– User Logins
1 ES45	– Failover host for the other jobs

3.2.8 Is There Room to Scale Out Cluster Size In Terms of Number of Members?

Many sites like to be able to design into a solution the ability to grow in computing capacity over time. Sometimes it is known up front that the application load will increase over time; other times it's simply the desire to be ready in case it does, due to an unforeseen event. With a cluster, the computing power can be grown in two dimensions – the size of the members and the number of members.

The changes that can trigger a need to grow the cluster by adding new members are:

• The addition of new applications

• The number of members running instances of a multi-instance application

• Moving applications that were sharing member systems onto their own member systems

In the case of the number of cluster members, the currently supported maximum is eight nodes. This means that if a site would like to be able to grow, it needs to start with a cluster size of less than this value.

3.2.8.1 Example

In the Biotech cluster solution, as it stands there are six members. This leaves room to add two more members either for new applications or because in the case of the multi-instance compute jobs, more computing power is to be supplied by adding instances.

3.2.9 Is There Room to Scale Up Individual Member Computing Capacity/Size?

As we discussed previously, the capability to grow in computing resources is often a requirement for computing solutions. Growing the cluster in number of members can be useful for certain types of requirements but not others. A key growth that can only be met by increasing the size of an individual member is:

• A single-instance application that needs additional computing resources on the member where it is running.

In this case, we are looking at the traditional growth mechanisms for a standalone platform: increasing the number of CPUs, memory, HBAs, etc., or replacing the system with a larger server model.

The largest single platform in the AlphaServer stable has been the GS320. Recently coming onto the scene is the even larger GS1280 "Marvel" class systems. In the case of both models, they are moduler and scalable. A site can start with a small configuration, which can later be expanded/upgraded to many times its initial installed size.

3.2.9.1 Example

The Biotech cluster solution is currently centered on ES45 servers. The next size platform would be a GS-series platform, which would have plenty of scalability beyond the ES45 in terms of CPU count, memory, etc.

^[3]Real Application Clusters

^[4]HyperText Markup Language