Section 10.2. Best Practices

10.2. Best Practices

In this section, the best practices in the following areas will be explored:

• ESX Server hardware

• ESX Server software

• Virtual machine builds

• Administrative tasks

10.2.1. ESX Server Hardware

In this section, we'll discuss the best practices related to ESX Server hardware, including configuration, storage management, and documentation.

10.2.1.1. Purchasing Supported Hardware

When considering hardware for your virtual infrastructure, the first thing you want to consider is the hardware supported by VMware. Most major computer hardware vendors, including Dell, HP, and IBM, all have platforms that are supported by VMware. To ensure you're purchasing supported hardware, read the Hardware Compatibility Guide which you can obtain from the following URL: www.vmware.com/pdf/esx_systems_guide.pdf.

You should also familiarize yourself with the minimum and maximum system requirements outlined in the ESX Server Installation Guide, which is available at www.vmware.com/pdf/esx25_install.pdf.

You can avoid many problems and save loads of time by purchasing supported hardware. Nothing is more frustrating than troubleshooting hardware issues only to find out from VMware Support that the piece of hardware you're having an issue with isn't supported.

After you receive your hardware, burn it in for a while. There are tools you can use to benchmark or simply load an operating system onto it and thereby keep it active for a few days.

10.2.1.2. Multiple NICs

Although it isn't absolutely mandatory, it's highly recommended you get multiple NICs for your ESX Server. It's possible to have the Service Console NIC use a VMnic, which is great to test with and so on. But if this is a production ESX Server, or for that matter a serious test ESX Server, use multiple NICs. NICs are inexpensive and for network-intensive virtual machines, you may want to allocate a separate NIC to achieve greater service levels. Placing the NIC for the Service Console on a different network (subnet) than your virtual machines is also a recommended best practice and another reason to have them on separate physical NICS.

10.2.1.3. Memory

ESX Server uses memory efficiently and allows you to overallocate the physical memory you have to your virtual machines. That said, if you were to over-spec any aspect of our ESX Server, we'd recommend you choose to have more memory than, say, the fastest CPUs. By having enough memory for both the Service Console and your virtual machines, you can prevent excessive paging (thrashing) which causes performance degradation.

If you plan on loading systems management software on your ESX Server itself, you may want to increase the amount of memory you give the Service Console. This is especially true if you plan to run a large number of virtual machines on your ESX Server. Although not stated in any documentation we've read, we believe a best practice for memory in high-end production ESX Servers (especially these days since memory prices are low) is to allot the maximum amount of memory to the Service Console, the maximum being 800MB.

10.2.1.4. Documentation

Document the hardware configuration of your ESX Server. Keep that document up-to-date if you make any changes or upgrades to the ESX Server hardware. The document can be as elaborate or basic as you wantthe key is to make it accurate. If you have a document versioning product such as Microsoft's SharePoint Server, keep the document in there and allow the versioning feature to track any changes you or others make to the document.

10.2.1.5. Server Sizing

If you're doing a server consolidation project, you need to gather the historical utilization trend information from the physical servers you're going to virtualize. This data allows you to better gauge the performance requirements of your ESX Server and you'll be better able to size your server(s) accordingly.

10.2.1.6. Disks

Ensure that the local storage on your ESX Server has redundant drives and that they are set up in a RAID-1 configuration. Hot-swappable spares are always ideal.

Try to prevent your file system from filling up and running out of room.

If your virtual machines will be run on local storage, provide a second controller card to help improve performance. This is especially important if the virtual machines are hard-core disk I/O intensive.

10.2.2. ESX Server Software

In this section, we'll discuss best practices related to ESX Server software, including installation, security, the Service Console swap space, and backups.

10.2.2.1. Practice, Practice, Practice

Become familiar with the installation by building and rebuilding your ESX Server. The installation that will eventually go into production is what all your virtual machinesthus, your production serviceswill be running on. In addition to the installation, practice the commands such as vmware-cmd and vmkfstools with their various arguments (options). Although you can do probably 95 percent of the functions ESX Server offers through the MUI, that other 5 percent is very handy to know and can pull you out of tough spots. To that end, learn the file system and common commands you'll need to navigate through it and make any changes. Although you don't need to be a Linux guru, knowing the basics can really help you out. Pick up a book on Linux, learn the commands taught in this chapter and in Chapter 13, and check out the wealth of free information available on the Internet. For a quick reference, go to www.oreilly.com/catalog/debian/chapter/book/appe_01.html.

10.2.2.2. Security

By default, when you install ESX Server the Security is set to High. What this means is that Telnet, FTP, and NFS file sharing are turned off and that only encrypted sessions to the Service Console and MUI are permitted. Don't change the default security setting unless you have a very good reason to do so.

Keep the root password safely guarded and use the su command if you need to perform functions as root. Otherwise, log in with a normal account to perform tasks on the Service Console.

Avoid turning on any unnecessary services or daemons and run as few things as possible on the Service Console.

Adhere to Linux administration best practices which include things like having a strong password policy, having root kit detection software, and using PuTTY and WinSCP to access the Service Console from a Windows desktop (as we discussed in Chapter 8) and ensure disaster recovery measures such as backing up the Service Console are followed. Backups will be discussed more in Chapter 12.

You can review VMware's Security Response policy at www.vmware.com/support/policies/security_response.html.

10.2.2.3. The Service Console's Swap Space

The Service Console swap file should be set to one or two times the amount of RAM allocated to the Service Console. Say you've allocated the maximum amount of memory, 800MB, to the Service Console. Then your Service Console swap file should be between 800MB and 1.6GB. If you set your memory for your Service Console to 192MB and the swap file to 384MB, which is an adequate setting for hosting eight virtual machines, be careful not to exceed the number of virtual machines on your ESX Server. Doing so may cause performance issues. If you're going to have more virtual machines than you originally planned, increase the Service Console memory allocation and adjust the Service Console swap file. This helps ensure good performance.

10.2.2.4. VMkernel Swap Space

The VMkernel swap file is used by your virtual machines if you overcommit memory and paging occurs from your virtual machines. The recommended swap file size is one to two times the total amount of RAM you have on your ESX Server.

10.2.2.5. Backups

Back up the Service Consolewithout fail. See Chapter 15.

10.2.2.6. Modifying or Updating the Linux Kernel on Your ESX Server

Don't modify or update the Linux kernel on your ESX Server. The kernel has been highly customized and should not be touched. Bad things will happen.

10.2.3. Virtual Machine Builds

Building a virtual machine involves several key steps. In this section we'll discuss using templates, defragging, allocating swap space, and mixing your ESX Server's workloads.

10.2.3.1. Using Templates

Create templates or gold builds for your virtual servers. These templates should include many items from your build books for your existing physical servers. You'll want to include some or all of the following: the operating system, of course, with the latest security patches and service packs; antivirus software; management software; IDS software, which would include root kit detection components; and all other packages deemed necessary by your particular environment. You should build a new build book for virtual machines that can be used for provisioning future demand and that incorporate the elements of physical-to-virtual migration.

Without fail, load VMware Tools in every one of your virtual machines.

10.2.3.2. Defrag

Include defrag space in your virtual machines. Don't forget, these virtual machines are files, and for optimum performance, they need to be defragged on occasion. Set up a scheduled task or cron job to ensure this occurs.

10.2.3.3. Virtual Machine Swap

Include in your virtual machines enough swap space necessary for your guest operation system. Generally, the amount is one to two times the amount of RAM you allocate to the virtual machine.

10.2.3.4. Guest OS Memory Requirements

Discover and document the guest operating system memory requirements and how they'll be affected by ESX Server's ability to share common memory pages between similar OSes. VMware has stated that 30 to 60 percent of the memory pages of Windows idle virtual machines are identical.

10.2.3.5. Mixing Your Virtual Machine Workloads

Mix your virtual machine workloads on your ESX Server. Try not to load virtual machines that all require heavy disk I/Ofor example, all on the same ESX Server. If you mix your workloads, you will see better performance marks from your virtual machines and a more efficient use of your ESX Server's hardware. Additionally, you may also group your virtual machines based on function. Let's say you want to have all your Citrix servers on one or two ESX Servers. You would do this for ease of administration, not necessarily because you believe it the most efficient use of your ESX Server's hardware. Although this is not necessarily a best practice, it's another option for you to consider.

10.2.3.6. ISOs

Use ISOs, without fail, whenever and wherever you can. See Chapter 8.

10.2.3.7. Turning Off the Unnecessary

Do you really need millions of colors for your virtual machine's displays or a wireless service running? Of course you don't.

What you need is to turn off unnecessary services and minimize configuration parameters, which will allow for more scalability for your ESX Server and better performance of both your virtuals and your hosts.

10.2.3.8. Keeping Off the Unnecessary

ESX Server is optimized to run virtual machines. If you can, avoid turning on services available from the Service Console that may degrade the performance of your hostfor example, NFS, X.

10.2.4. Administrative Tasks

In this section, we'll discuss administrative tasks, such as using VirtualCenter or vmkusage to monitor the performance of your virtual machines and following licensing procedures.

10.2.4.1. Monitor…It's Not so Reptilian

Monitoring the utilization of servers in the past was not the most exciting work. A maxed server in the past could mean a lot of work for an administrator. Now, allocating more memory, an additional CPU, hard drive, or NIC is a process that takes minutes, not days or weeks. Monitor the performance of your virtual machines to ensure you meet your service-level agreements. Use VirtualCenter for thisit's excellent. You can also use vmkusage, which provides for the utilization of your ESX Server as well as all running virtual machines.

To start vmkusage, from the command line of your ESX Server's Service Console type vmkusagectl install.

This creates the utilization reports in the /var/log/vmkusage directory. These reports are updated every five minutes by default. You can view the reports with a Web browser by typing https://esxservername.corp.com/vmkusage.

Substitute the URL with the appropriate information in your environment. You should see a window similar to the one shown in Figure 10.1

Figure 10-1. vmkusage

From here, you can pull very specific utilization reports that you'll find important for capacity planning, future demand availability, server sizing, and hardware purchasing.

10.2.4.2. Licensing

Don't forget, just because you can copy and paste a server doesn't mean you are properly licensed. Follow the licensing procedures you've established so as not to end up with a hefty bill if the BSA comes knocking.