Running Software in Emulators | Hacking Ubuntu: Serious Hacks Mods and Customizations (ExtremeTech)

Although nearly compatible software helps collaborators bridge a communication gap, nothing beats running the exact same application. For example, if you need to edit Word documents, then nothing does a better job than Microsoft Word. In the days of old, this true compatibility was accomplished with dual-boot systems. A single computer would have multiple operating systems installed and the user would reboot the computer into whatever operating system they needed. Although Ubuntu does support dual-boot environments (see Chapter 1), this is not as effective as being able to run applications from multiple operating systems at the same time. Today, people either run two separate computers with different operating systems and network connectivity (hardware is cheap), or they use hardware emulators.

A hardware emulator is an application that pretends to be an entire computer. Emulators replicate a computer in a virtual machine (VM): CPU, memory, devices (for example, hard drives), and even BIOS. A perfect emulator can run any operating system, and the operating system should not be able to tell that it is running in an emulator. For example, operating systems like Ubuntu and Windows expect standardized hardware. If the emulator mirrors the hardware, then either operating system will run without a problem. Most OSs do not need to know that the hardware is emulated.

There are many reasons to run an emulator, such as playing with hostile viruses, opening suspicious e-mails, or installing software without screwing up your main system. A true emulator is a perfect sandbox, where software can run without hurting the host operating system. But one of the main reasons to use an emulator is compatibility. Not all software can be used natively under Ubuntu, but an emulator running under Ubuntu can be installed with a non-Ubuntu operating system and used to run native applications. For example, if I really need to use Microsoft NetMeeting, then I can start a VM running Windows and use NetMeeting from within the emulation.

Choosing an Emulator

Emulators have three main components. The host operating system is where the emulator is running and the guest operating system runs in the emulator. For example, an Ubuntu host can run Windows 2000 as a guest within an emulator. The final component is a virtual hard drive. This is usually a file on the host system. I frequently install a guest operating system to a virtual hard drive (file), configure it just the way I like it, and make a backup of the file. Later, after playing with software in the guest system, I can copy back the image in order to reset the system. This is faster and cleaner than trying to remove undesirable software or perform a restore from a backup system.

Tip

If you want to see if a patch will make your system unstable, first install it in a VM. I have a VM installed with the same software as some of my critical computers. Before patching the real thing, I make sure the VM works after being patched.

There are three main emulators for Ubuntu: Qemu, VMware, and Xen. Although each provides solid emulation, they are all different. Table 6-2 shows some of the differences. All three emulators offer the same basic features. They all support multiple drives, offer solid i386 emulation, and support network access. The main differences come from licensing, architecture support, and speed. Qemu is the slowest of the emulators, but offers the widest hardware support. Xen 3.0 is arguably the fastest, but only supports Linux or BSD systems. VMware offers a commercial quality, fast emulation, and refined interface, but is proprietary and limited to Linux and Windows host systems.

Table 6-2: Comparison of Hardware Emulators
Open table as spreadsheet
Feature	Qemu 0.8.1	VMware 1.0.2	Xen 3.0.2
Licensing	GPL	Commercial	GPL
Open Source	Yes	No	Yes
VM Type	Emulator	Emulator	Virtualizer
Host Architectures	i386, PowerPC, Sparc; Linux, BSD, Windows, Solaris, MacOS X; portable to other host systems	i386; Linux and Windows	i386; Linux and BSD^[4]
Guest Platforms	i386, PowerPC^[1], ARM, MIPS, Sparc, and more	i386	i386^[5]
Kernel Boost Module	Yes^[2]	Yes	Yes
Ubuntu Support	Universe repository^[3]	Generic Linux binary	Download source or Debian binary
Installation and Removal	`apt-get` for install and remove	Install script; manual removal	Manual install and removal
^[4]Xen 2.0 supported FreeBSD, but Xen 3.0 broke that support. Check with the Xen homepage for updates on BSD support. ^[1]Qemu 0.8.1 only supports Linux as a guest OS on a PowerPC. ^[5]Xen virtualizes hardware, so it can only run multiple instances of the host's hardware. ^[2]Qemu 0.8.1 includes a kernel boot module, but it does not work consistently under Ubuntu. ^[3]The Ubuntu universe repository may not have the latest version of Qemu available.

There are two types of virtual machine: emulators and virtualizers. An emulator uses software to represent a computer system. A true emulator, such as Qemu and VMware, provide virtual hardware, BIOS, and even multiple CPUs. The benefit is that any operating system that is supported on the real hardware will work within an emulator. For example, if the emulator acts as an SMP Pentium Pro PC with 512 MB RAM, then it will support Windows, Linux, BSD, OS/2, E/OS, and any other PC operating system.

The biggest limitations with emulators are system resources and speed. If you configure an emulator to have 512 MB RAM but you actually have 128 MB RAM, then you can expect to spend time swapping RAM to disk. As speed goes, emulators interpret running opcodes. This means, the guest operating system runs slower than if it was on a dedicated computer. The speed can be less than 25 percent of the actual operating system's clock speed. For example, a 2.8 GHz host system may have a guest that appears to run at 300 MHz. Systems can run even slower if they are performing hardware-intensive commands. For example, when installing any guest operating system, the detection of attached peripherals may take a very long time. To overcome some of the speed limitations, kernel modules are available for boosting performance. These modules allow the emulator to directly link some functions into the host operating system. Drive, memory, and video access can all be performed faster with a kernel module. In the best cases, a kernel module can give you a nearly 1-to-1 performance ratio.

In contrast to emulators, virtualizers do not emulate hardware. Instead, they manage the existing host hardware and allow different guest systems to share the same resources. Xen is an example of a virtualizer. Xen allows multiple guest operating systems to run independently on the same hardware. Although Xen won't allow you to run different applications for collaboration, it can assist in dividing workloads, testing networked applications, and even benchmarking software. Xen is often used for security. Each independent operating system performs a specific task. If one Xen VM is compromised, it will not impact other VMs.

Understanding Virtual Disks

In general, there are a few types of virtual drives you can use with emulators. The first is a disk image of an actual hard drive. After installing the guest OS, this image will contain a partition table, boot loader, and the guest OS-similar to the block device /dev/hda. This is the most flexible option. Unfortunately, this option is not always desirable since you cannot easily copy files off of the disk image. (Linux does not allow you to mount a file containing a partition table.)

The second option is a plain disk partition. This is a file (or device) missing the partition table and boot loader-similar to the block devices /dev/hda1 and /dev/hda2, which are partitions under /dev/hda. This option allows you to mount the file as a loop-back device. If you want to share files between the guest and host OSs, you can mount the partition image. For example, if the partition-file is called disk.img, then you can use:

 sudo mkdir /mnt/img   # make sure the mount point exists sudo mount -o loop disk.img /mnt/img  # mount the disk

By mounting the partition, you can copy any files you need to and from the VM.

A third option is to use an existing directory. The directory is used by the VM and treated as a disk partition. While not supported by all emulators, this does make it easier to copy files between the host and guest systems.

Differences Between VNC and VM

All of these emulators have the ability to grab the mouse. This means that the mouse's input is completely used by the VM. This is different from VNC, where the remote mouse is separate from the local mouse, and moving the mouse outside the window allows you to leave the remote window. To release the mouse from the emulator, press Control and Alt at the same time.

Note

For Qemu, only the Ctrl+Alt keys on the left side of the keyboard release the mouse. The ones on the right are sent to the guest operating system. With VMware, any combination of Ctrl and Alt keys will release the mouse.

The other big difference concerns the clipboard. Under VNC, you can cut and paste between the remote (guest) and local (host) operating systems. Qemu 0.8.1 does not support this feature-the host and guest do not share a clipboard. Xen 3.0 and VMware do offer shared clip- boards if the guest OS kernel is modified.

Emulating with VNC

The coolest feature provided by all three emulators is the ability to use VNC as the display. Normally when a VM is started, a window appears that acts as the display for the operating system. Qemu, VMware, and Xen allow you to specify a VNC display instead of a normal window. For example, under Qemu you would use qemu -vnc 2 to start the display on the VNC server localhost:2. Now, if you want to collaborate, you can use any application on any supported guest operating system-you are not limited to sharing your Ubuntu desktop.

Using Qemu (Open Source)

The Q-Emulator (Qemu; http://fabrice.bellard.free.fr/qemu/) is the most flexible emulator option. Qemu supports a wide variety of host and guest architectures. Generally, if a pre-built Qemu binary is unavailable, you can download the source code and compile it. For Ubuntu, you can install Qemu from the universe repository; Qemu is the only emulator that offers Dapper-specific binaries.

 sudo apt-get install qemu

You may want to compile Qemu from scratch since the Ubuntu package does not always contain the most recent code release.

Note

You will need to have a developer's environment installed. See Chapter 4 for Programming with C.

Download the latest source code from http://fabrice.bellard.free.fr/qemu/.
Follow the installation instructions. They should be as simple as:
- Extract the source code and cd into the source code directory.
- Configure the build files with ./configure.
- Compile Qemu using make clean ; make.
- Install it using sudo make install.
You will need to do one manual configuration step to specify the keyboard mapping for the Qemu VNC server:
```
 sudo ln -s /usr/share/rdesktop/keymaps /usr/local/share/qemu/keymaps 
```

Installing a Qemu VM

To get started using Qemu, you first need a virtual disk. There are a bunch of options for doing this:

Use qemu-img-The program qemu-img creates a blank file that will act as a disk image. To create a two-gigabyte file called disk.img, use:
```
 qemu-img create disk.img 2G 
```
Create a blank disk-You can use dd to create a blank disk. For example, dd if=/dev/zero of=big1 bs=512 count=4194304. This will do the same thing as using qemu-img (but not as fast).
Copy a working drive-Use dd to make a copy of a working hard drive. This is much faster than installing a guest OS within the VM. If the disk is located at /dev/hdb, then you can copy it to disk.img using:
```
 dd if=/dev/hdb of=disk.img 
```
Use a real hard drive-If the disk is installed as /dev/hdb, then you will just need read/write access to /dev/hdb.

Warning

Do not use the same boot device as your host system! If your host OS is using /dev/had, do not tell the VM to use /dev/hda. The drive may become corrupted if two operating systems use it at the same time.

If you have configured a blank disk, then you will need to install an operating system on it. This is usually done using a CD-ROM drive or ISO image. For example, if you downloaded the Ubuntu server ISO, then you can burn it to a CD-ROM or install directly from the ISO:

 qemu -hda disk.img -cdrom /dev/cdrom -boot d qemu -hda disk.img -cdrom ubuntu-6.06-server-i386.iso -boot d

Both of these commands will start the Qemu i386 VM and begin installing Dapper Drake. The main options for Qemu specify the images for hard drives (-hda and -hdb for the primary and secondary IDE drives), CD-ROM (-cdrom), and floppy drives (-fda and -fdb). By default, Qemu boots from the first hard drive (-boot c). If you want, you can specify booting from the floppy (-boot a) or CD-ROM (-boot d). There are other options for supporting USB devices, network cards, and video.

Tip

Qemu 0.8.1 does not support emulating DVD drives. If you need to access a DVD image, mount it to your local file system (for example, sudo mount /dev/dvd /mnt/dvd) and access it as a directory by using qemu -hdb fat:/mnt/dvd.

Installing Ubuntu under Qemu can take a very long time. Although a real Ubuntu installation may complete in under a half-hour, a Qemu-based installation may take two hours or longer. You can usually speed up VMs by increasing the amount of emulated RAM. Qemu defaults to 128 MB for each VM. With this default setting, I do not recommend booting off the Ubuntu desktop Live CD-ROM-you may grow old and die before the desktop loads (in can take over four hours). If you increase the VM's RAM to 256 MB (-m 256), then it should come up fully in a few minutes.

Running a Qemu VM

Although installations usually take a long time, installed operating systems are fast enough for real-time use. After you have installed the operating system, you can boot off the image drive using:

 qemu -hda disk.img

The only limits to the number of operating systems you can run simultaneously are the speed of the host system and the amount of shared memory. Although you can increase the shared memory size (see Chapter 7), you cannot increase your computer's speed. On a dual 2.8 GHz computer with 1 GB RAM, I would not recommend running more than two graphical operating systems (or four text systems) at one time. Since the boot sequence for most guest systems consumes the most resources, I would also recommend booting them one at a time. After booting, it is very easy to run two emulators at once. Figure 6-8 shows two Qemu sessions running. One VM is running E/OS (a BeOS clone) with a VNC display. The other is running Windows 98 with the Firefox web browser. The host operating system is Ubuntu's Dapper Drake.

image from book
Figure 6-8: Running two Qemu sessions

Creating Partitions

Although having a disk partition for a file is great for loop-back mounts, Qemu and VMware cannot use a disk image as a drive unless it has a partition table. The challenge becomes: How do you create a partition table when all you have is a partition? The answer involves the dd and fdisk commands.

Let's assume you have a disk partition in a file (for example, part.img created from /dev/hda1) and you want to turn it into a disk image for Qemu (disk.img). First, you need to allocate space for the partition table. The partition table consumes the first 63 sectors of the drive. Using dd, you can replace your partition file (part.img) with a disk image (disk.img):

 dd if=part.img of=disk.img bs=512 seek=63

Your disk file (disk.img) is 32,256 bytes larger than the partition image and has space for a partition table.

The next step is more complicated. Disks have sectors, heads, and cylinders. While old drives had a direct correlation between heads and physical read-write heads, newer drives simply use it as a numerical offset. Groups of sectors are grouped into cylinders. There are a maximum sizes are 63 sectors per cylinder, 255 heads, and 16,383 cylinders. For disks larger than 125 Gigs, sector sizes greater than 512 bytes are used.

Take a look at the size of your disk.img file and compute the number of cylinders. For example, let's say the size of disk.img is 1,073,774,080 bytes (a 1 GB partition plus partition table). Qemu prefers 16 heads and 63 sectors: 1,073,774,080 bytes ÷ (512 bytes per sector · 63 sectors · 16 heads) = 2080.5704 cylinders. Since you cannot have a fractional cylinder, round up to 2081. Now you can use fdisk to create the partition table:

 fdisk -C 2081 -H 16 -S 63 disk.img

Tip

The fdisk command works on both block devices and files. You don't need to be root to modify a file, but fdisk will still try to synchronize disks. If you see errors about ioctl(), don't worry-only root has permission to call it and your changes are still saved.

Create one partition that spans the entire disk, from cylinder 1 to 2081. The final partition table should look like:

   Device Boot      Start     End     Blocks   Id  System disk.img1               1    2081    1048823+  83  Linux

Note

Based on the type of partition, you may need to change the system identifier in the partition table. If you plan to boot from this drive, you will also need to make the partition "active".

Your saved disk image is now usable as a disk by Qemu. Even though it may not be bootable (since there is no boot manager for Linux), you can boot from a live CD-ROM and setup a boot manager.

Using VMware (Commercial)

VMware is a commercial emulator. Although it has a more professional feel than Qemu and offers more configuration options, installing VMware is not as painless as Qemu.

Note

Before you begin, check your system requirements. VMware only runs under Ubuntu on an i386. If you are using an IA64 or PowerPC system, then you cannot use VMware.

Go to http://www.vmware.com. VMware offers three different types of emulators. The VMware Player is free, but cannot be used to create a new image. In contrast, the VMware server and workstation are not free, but can create new images and make it easy to configure emulated hardware. Download the type of VMware system you plan to use. Be sure to download the Linux TAR image.

Tip

If you plan to install the VMware server, get your serial number code first! While you can re-run the installer, the server will not run until you enter your serial number. The VMware workstation offers a short trial period, but then must be registered to use.

Extract the TAR file. For example, if you downloaded VMware-server-1.0.1- 29996.tar.gz, then you would extract the contents-this creates a directory called vmware-server-distrib. Similarly, the VMware-player will create a directory called vmware-player-distrib.
```
 tar -xvf VMware-server-1.0.1-29996.tar.gz 
```
For the VMware kernel module, you will need to install the headers for your kernel.
```
 sudo apt-get install linux-headers-`uname -r` 
```
Now you can install VMware. Go into the VMware directory and run the install script.
```
 cd vmware-server-distrib  # or vmware-player-distrib sudo ./vmware-install.pl 
```
The install script will ask you a bunch of questions. Unless you have special needs, the default values should work fine.

Warning

Unlike Qemu's apt-get package, VMware does not have an uninstall option. Once you install it, it is installed. And VMware scatters files all over the system.

Unlike Qemu, VMware provides lots of options for networking virtual machines. You can easily create a virtual subnet with lots of virtual machines. This is really useful for testing network software.

Creating a VMware Disk

If you purchased the VMware server or workstation, then you already have the ability to create a VM. But if you are cheap, then you installed the VMware player. The player can only run existing images, it cannot create new images. Fortunately, you can still create a boot image for use with the VMware player; the secret is to use Qemu. In this example, we'll install Dapper onto a 2 GB drive.

Install Qemu. (You will need qemu-img.)
You will need to create a VMware disk image (VMDK). You can do this with qemu-img.
```
 qemu-img create -f vmdk disk.vmdk 2G 
```

Every VMware disk image also includes a configuration file. Listing 6-4 shows the configuration for booting off the Ubuntu server ISO. (Configurable portions appear in bold.) Call this file disk.vmx. If you plan to customize this example, then be sure to change the disk names.

Listing 6-4: Sample VMX Configuration File: disk.vmx

 #!/usr/local/bin/vmware config.version = "8" virtualHW.version = "3" memsize = "128" nvram = "ubuntu.nvram" ide0:0.present = "TRUE" ide0:0.fileName = "disk.vmdk" ide0:0.redo = "" ide1:0.present = "TRUE" ide1:0.fileName = "ubuntu-6.06-server-i386.iso" ide1:0.deviceType = "cdrom-image" ide1:0.autodetect = "TRUE" floppy0.fileName = "/dev/fd0" scsi0:0.redo = "" usb.present = "TRUE" sound.present = "TRUE" sound.virtualDev = "es1371" displayName = "Ubuntu" guestOS = "other24xlinux" uuid.location = "56 4d ad 0d c5 c8 b9 88-d2 f8 c0 01 12 5e 26 c4" uuid.bios = "56 4d f3 a5 03 8c cb b9-ed bb 8f 10 a3 de b0 10" ethernet0.addressType = "generated" ethernet0.present = "TRUE" ethernet0.connectionType = "nat" ethernet0.generatedAddress = "00:0c:29:de:b0:10" ethernet0.generatedAddressOffset = "0" checkpoint.vmState = "" tools.remindInstall = "TRUE"

Run the program vmplayer. This will ask you to select an image.
Select your disk.vmx file. Don't worry if some configuration items are not correct; vmplayer will correct them for you.

VMware is noticeably faster than Qemu, but installing Ubuntu still takes a very long time. After installing the operating system, you can boot the system off the virtual hard drive.

Note

This hack enables you to test VMware, but is not intended for long-term use. For full support, consider purchasing the VMware server or workstation.

Converting Partitions

VMware and Qemu use different disk image formats. Qemu supports a variety of formats, but is usually used with either a copy-on-write disk (cow or qcow) or a raw disk image. In contrast, VMware uses the vmdk format that includes metadata along with the disk. Fortunately, you can easily convert between formats using qemu-img. To convert a raw Qemu image (for example, disk.img) to VMware disk (disk.vmdk), use:

 qemu-img convert disk.img -O vmdk disk.vmdk

Similarly, you can convert a VMware disk to a Qemu disk using:

 qemu-img convert -f vmdk disk.vmdk disk.img

Using this approach, you can use any Qemu disk under VMware and vice versa. Even though VMplayer cannot be used to create a virtual disk, Qemu can! And the Qemu disk can be quickly converted for use with VMware.

Using Xen (Open Source)

Xen is an operating system virtualizer designed for speed. This application replaces the host operating systems. At the time of this writing, Xen was not available from the Ubuntu repositories and only supported Linux and BSD guest operating systems.

Warning

Installing Xen requires you to change your kernel and boot loader (Grub) configuration. If you screw up the installation then you can really hose your system. If you are not comfortable with a lot of manual configuration and troubleshooting, then don't try Xen. You might want to wait until the project matures a little more and you can use apt-get to install it.

Note

Before you begin, check your system requirements. Xen only supports i386 and IA64, architectures with optional SMP support. If you are using a PowerPC system, then you cannot use Xen.

While there are many different ways to install Xen, I believe that the instructions at http://www.howtoforge.com/xen_3.0_ubuntu_dapper_drake are very helpful and detailed.

You will also receive instructions when you download the Xen source code-follow those instructions closely.

Install necessary packages. Xen uses these for installation and when it runs.
```
 sudo apt-get install python-twisted bridge-utils debootstrap 
```
Edit /etc/mkinitramfs/modules and increase the number of loop-back devices.
```
 sudo bash -c 'echo "loop max_loop=64" >> /etc/mkinitramfs/modules' 
```
Download Xen from http://www.xensource.com/xen/downloads/. You should download the pre-build installations (listed as tarballs). To download the code, you may need to provide an e-mail address where you will receive download installation instructions. Wait for the e-mail and follow the instructions. Be sure to check the prerequisites and follow the post-install steps! This should allow you to install Xen.

Tip

When downloading Xen, look for the open source downloads, not the commercial software.

Note

Why are the instructions not in this book? Xen is undergoing a lot of development and the installation instructions frequently change. It is likely that any instructions listed here would become outdated quickly. The best instructions are found in the README file for the Xen installer.

After you install Xen and reboot your system to use the Xen kernel, you can use the xm command to create a new VM, launch a VM, and remove a running VM.

Wine or Vinegar?

When people talk about emulators, the topic almost always turns to Wine. (Wine is a recursive acronym: Wine Is Not an Emulator.) Wine was created before most hardware emulators. It provides converted Windows libraries, enabling Windows applications to run under Linux. This is different than an emulator: emulators provide an entire system, while Wine provides support for Windows applications under Linux. Wine's support is very good for most Windows programs. In fact, it is so complete that it even permitted some Windows viruses to spread! (In http://os.newsforge.com/print.pl?sid=05/01/25/1430222, Matt Moen reported that Klez, Sobig, and a few other viruses worked under Wine.)

While many applications can be used with Wine (for example, Microsoft PowerPoint and Word), other applications are hit-and-miss. Even newer versions of supported programs do not always work because of changes in the Microsoft libraries. Wine is also a mostly unsupported application-it had no updates between August 2005 and August 2006. Even though it was created in 1993, it is still listed as beta code 13 years later.

Although Wine was an acceptable solution a few years ago, true emulators and virtualizers such as Qemu, VMware, and Xen provide better compatibility.

Sharing Files with Emulators

Regardless of your choice of emulator, you will need some way to exchange files between the guest and host systems. There are a variety of choices-you should choose the one(s) that best fit your needs.

FTP-You can run an FTP server on the host OS and use the guest to connect and transfer files. This can also be done using Secure Shell's scp command. This option is almost universally supported by all guest operating systems.
NFS-The host OS can export a partition to the guest OS. This works well if the guest is running a version of Linux or Unix.
SAMBA-For a Windows guest OS, you can export a partition from the host OS using SAMBA.
Port forwarding-Qemu and VMware allow you to forward ports between the host and guest operating systems. For example, the Qemu parameter -redir tcp:10022:22 will redirect port 10022/tcp on the host to the SSH server (22/tcp) on the guest. Port forwarding allows the host to communicate with the guest OS using whatever server is you require.

Note

All of these options require network access. Although network access is supported by each of these hardware emulators, you may want to disable the emulator's network access if you are evaluating viruses, performing disk forensics, or installing questionable software. If the network is disabled then there is no easy way to get data off of the virtual system.