Section 23.1. Backing Up VMware Servers

23.1. Backing Up VMware Servers

The popularity of VMware virtual servers has grown significantly in the last few years, prompting questions on how to back them up. First, we'll describe the architecture of VMware and follow that with a discussion of how to back it up.

23.1.1. VMware Architecture

VMware currently comes in two basic flavors, VMware Server and VMware ESX Server. VMware Server is a free version of VMware that offers basic virtual server capabilities and runs inside Linux or Windows. Each virtual machine is represented as a series of files in a subdirectory of a standard filesystem that you specify; the subdirectory carries the name of the virtual machine. For example, if you've chosen to store your virtual machines in /vmachines, and you have a virtual host called Windows 2000, its files will be located in /vmachines/Windows 2000.

While VMware Server runs inside standard Linux or Windows, VMware ESX Server uses a custom Linux kernel and a custom filesystem, VMFS, to store virtual machine files. You can also store virtual machine files on raw disk partitions. Neither the raw disk partitions or files in a VMFS filesystem can be accessed by all backup commands, so you probably need to back them up in a special way.

23.1.2. VMware Backups

When backing up a VMware machine, you have to back up the operating system of the VMware server (known as the service console on ESX systems) and the VMware application itself. You also need to back up each virtual machine's files. However, you cannot simply back up the virtual machine files or raw disks with a standard backup program. The virtual disks are constantly open and changing while the virtual machines are running, and you will not receive a consistent backup of them. Even open file agents won't necessarily work properly if the virtual machines are gigabytes in size. You therefore have three options for backing up virtual machines running inside VMware:

• Back up virtual machines as physical machines.

• Back up virtual files while virtual machines are suspended.

• Use VMware's built-in tools to copy a running virtual machine's files.

23.1.2.1. Back up virtual machines as physical machines

This is, of course, the easy method. Simply pretend that each virtual machine is a physical machine, and back it up as such. This method has both advantages and disadvantages.

If you use this method, don't forget to exclude the virtual machine files when you're backing up the VMware Server or ESX service console.

The first advantage of this method is that you can use the same backup system as the rest of your data center. Just because the machines are virtual doesn't mean you have to treat them as such. This simplifies your backup system. It also allows you to take advantage of full and incremental backups. Unless your backup software is able to perform subfile incremental backups, the other two methods perform full backups every day because the entire virtual machine is represented by a single file that will certainly change every day. Finally, it allows you to back up the systems live.

The disadvantage is that you have to configure backups for each virtual machine. Some may prefer to configure one backup for the entire VMware server. If you're using a commercial backup software package, this also increases your cost because you must buy a license for each virtual machine. A final disadvantage is that you also need to configure a bare-metal recovery backup for each virtual machine. Each of the other two methods won't need such a backup because restoring the virtual machine is all that's needed to perform its bare-metal recovery.

23.1.2.2. Back up suspended virtual machine files

If you can afford the downtime for each virtual machine, all you have to do is suspend a virtual machine prior to backing up its files. You can then back up the virtual machine files using your favorite backup program because the files won't be open or changing during your backup. The suspend function in VMware works just the same as on your laptop (and some servers). The current memory image and running processes are saved to a file that is then accessed when you power it on, causing all running processes to resume where they left off just before you suspended the machine.

The advantages to this method are that you don't need to configure backups for every virtual machine, and you don't need to worry about bare-metal recovery of these machines. The first disadvantage is that it performs a full backup of each virtual server every night, unless you have a backup that can perform subfile incrementals. The only open-source product that may be able to do that is rdiff-backup; see Chapter 7 for more information. The second disadvantage is that it requires suspending virtual machines, which renders them unusable during the backup. This downtime may be undesirable in many environments.

One way to have your cake and eat it too is if your virtual machines are residing on an LVM volume that supports snapshots. You could suspend all the virtual machines, take an LVM snapshot, then unsuspend the virtual machines. This minimizes the amount of time the virtual machine is suspended but allows you to take as long as you need to back up the LVM snapshot.

To suspend a running machine from the command line, run the following command, where the .vmx file is the configuration file stored in the virtual machine directory:

C:> vmware-cmd path-to-config/config.vmx suspend

You can now back up the virtual files using any method you choose. After backups have completed, you can restart the machine with the following command:

C:> vmware-cmd path-to-config/config.vmx start

If you use ESX Server, your backup tools may have trouble accessing the VMFS files. Make sure you test any new backup method.

23.1.2.3. Copy/export a running virtual machine using VMware's tools (ESX only)

Finally, if you're running VMware ESX Server, you can use the Perl APIs to copy the virtual machine files while the machine is running. They do this by creating a snapshot of the changes that happen to the virtual machine during the backup, storing them in a redo log, then copying or exporting the virtual files to another location. This method has the same advantages mentioned for the previous method, and it comes with the additional advantage of being able to back up systems while they're running.

This process requires using ever-changing Perl scripts, so we won't cover implementation details here. The VMware web site (http://www.vmware.com) includes some example scripts. There's also an open-source tool called vmbk, written by Massimiliano Daneri, that can automate this process. Learn more about it at http://www.vmts.net/vmbk.htm.

23.1.3. Using Bare-Metal Recovery to Migrate to VMware

One of the really nice things about using VMware (or other virtual server solutions) is that you don't have worry about bare-metal recovery of the virtual servers. As long as you can get them to not change during a backup, all you have to do is back up their files.

However, you can use the bare-metal recovery procedure documented in Chapter 11 to migrate physical machines into virtual machines. We just did this and turned 25 very old physical servers into one very nice VMware server. The following is the story of that migration.

I get asked all kinds of questions about backup products and how they behave on different operating systems and applications, and I use a lab to answer these questions. In addition to the usual backup hardware (SAN, tape libraries, VTLs), it consists of some Sun, IBM, and HP hardware running Solaris, AIX, and HP-UX. Up until just recently, we also had about 25 Intel machines running various versions of Linux and Windows and their associated applications (Exchange, SQL Server, Oracle, etc.). I never had enough machines, and I never had the right machines connected to the right hardware. We were constantly swapping SCSI and Fibre Channel cards, as well as installing and uninstalling applications. I could have used 100 machines, but that would obviously be prohibitive in many ways. (The cooling alone would be crazy.)

So we recently decided to see if we could get rid of all these servers with VMware. We bought a white box with a 3.5 GHz Dual Core AMD processor, 4 GB DDR2 RAM and 1.75 TB of internal SATA disks. I installed into that server two existing Fibre Channel cards and two SCSI cards. I then followed the alt-boot recovery method to move all of those physical servers into virtual servers, virtually upgrading each of their CPUs, storage, and memory in the process. Here are the steps I followed for each server:

1. I used the alt-boot full image method to create an image of the entire /dev/hda hard drive to an NFS mount on the new VMware server. (These images were typically 410 GB. They were old servers!)

2. I used VMware to create a virtual machine specifying a virtual IDE hard drive that was much bigger than the original, usually about 20 or 40 GB.

3. I used VMware to create a virtual CD drive that pointed to an ISO file that was actually a symbolic link to an ISO image of a Knoppix CD on the hard drive.

4. I booted the virtual machine into Knoppix using the virtual Knoppix CD.

5. I used dd to copy the image of the real hard drive to the virtual hard drive in the virtual machine booted from the virtual CD. (We did this by mounting the NFS drive where we stored the image.)

6. I "removed" the Knoppix CD by changing the symbolic link to point to an ISO image of a nonbootable CD and rebooted the virtual server.

7. In almost every case, the virtual server came up without incident, and voila! I had moved a physical server into a virtual server without a hitch! One Windows server was blue-screening during the boot, but I pressed F8 and selected Last Known Good Configuration, and it booted just fine.

8. I installed VMware tools into each virtual machine, which made their video and other drivers much happier.

9. Once I verified the health of each machine, I changed the CD symbolic link to point to Knoppix again and booted into Knoppix. I then used either qtparted (for Linux systems) or fdisk and ntfsresize (for Windows systems) to grow the original hard drive to the new size, as discussed in the sidebar "Restoring to Larger Hard Drives" in Chapter 11.

With 4 GB of RAM and a 3.5 GHz dual-core processor, I can run about eight virtual servers at a time without swapping. I typically only need a few at a time, and what's important is that I have Exchange 2000, SQL Server X, or XYZ x.x running; they don't need to run that fast. (That's how I was able to get by with those old servers for so long.) Each virtual server can have access to either one of the Fibre Channel cards or SCSI cards, which gives them access to every physical and virtual tape drive in the lab. They will also have more CPU, disk, and RAM than they ever had in their old machine. (I can even temporarily give any of them the entire 3.5 GHz processor and almost all of the 4 GB of RAM if I need to, and I don't have to swap chips or open up any CPU thermal compound to do it!)

I also get to have hundreds of virtual servers and not have any logistical or cooling issues, since each server only represents 2050 GB of space on the hard drive. I can have a Windows 2000 server running no special apps, one running Exchange 5, one running SQL Server 7, a server running Windows 2003 with no special apps, one with Exchange 2000, and one running Windows Vista. I could have servers running every distribution of Linux, FreeBSD, and Solaris x86and all of the applications those servers support. I think you get the point. I've got enough space for about 300 virtual server combinations like that. It boggles the mind.