Using a USB Drive | Hacking Ubuntu: Serious Hacks Mods and Customizations (ExtremeTech)

The ubiquitous USB flash memory drives (also called thumb drives) have effectively replaced floppy disks. They are smaller, less fragile, and store much more data. Because they are convenient, they can be used to kick off an installation, repair a damaged system, run a stand-alone operating system, or simply share files.

Formatting a USB Drive

USB drives support two basic formats: floppy drive and hard drive. A USB floppy drive consists of one large formatted drive. In contrast USB hard drives contain partition tables and one or more formatted partitions. If you purchased a thumb drive and never formatted it, then it is most likely configured as a USB hard drive with one large partition.

Warning

Before formatting or partitioning any device, be sure the device is not mounted! Use the mount command (without any parameters) to see if it is mounted, and then use umount to unmount any partitions. For example, to unmount /dev/sdc1 mounted at /media/usbdrive, you can use sudo umount /dev/sdc1 or sudo umount /media/usbdrive.

Thumb drives are usually partitioned just like a regular hard drive. Commands such as fdisk and cfdisk can easily modify the drive partitions, and mkfs can be used to format a partition. Besides capacity, speed is a significant difference between thumb drives and hard drives. When you change the partition table on a flash drive or format a partition, wait a few seconds before removing the drive; otherwise, some data may be buffered and not yet transferred.

Tip

When writing to a thumb drive, I usually run the sync command (sudo sync). This flushes all cached data to the disk. When the command returns, it is safe to remove the drive.

When you use the fdisk or cfdisk command on a thumb drive, you configure it as a USB hard drive. However, you can also configure it as a USB floppy drive. This requires formatting the device without partitioning it. For example, to make an ext2-formatted USB floppy drive on my 64 MB USB thumb drive (/dev/sdc), I can use:

 $ sudo mkfs /dev/sdc mke2fs 1.38 (30-Jun-2005) /dev/sdc is entire device, not just one partition! Proceed anyway? (y,n) y Filesystem label= OS type: Linux Block size=1024 (log=0) Fragment size=1024 (log=0) 16128 inodes, 64512 blocks 3225 blocks (5.00%) reserved for the super user First data block=1 8 block groups 8192 blocks per group, 8192 fragments per group 2016 inodes per group Superblock backups stored on blocks:         8193, 24577, 40961, 57345 Writing inode tables: done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 22 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. $ sudo sync

Warning

When you first plug in a USB hard drive, all the partitions will appear and automatically mount. However, to create a USB floppy drive, be sure to unmount all partitions and then format the main device (for example, /dev/sda or /dev/sdc) and not a partition (for example, /dev/sda2 or /dev/sdc1). You will need to disconnect and reconnect the device after you format it in order to remove any stale device partition identifiers.

Sharing Files with a USB Drive

The simplest and most common use for a USB drive is to share files between systems. Dapper supports most USB drives. Simply plugging the drive into the USB port will automatically mount the drive. From there, you can access it as you would access any mounted partition.

Tip

Many thumb drives have a light to indicate that the drive is being accessed. Even if the drive is not mounted, do not unplug the drive until the light indicates all activity has stopped.

Linux, Windows, Mac, and most other systems support FAT file systems. In order to share files with other users, consider formatting the drive with mkdosfs. For example:

Install the dosfstools package if mkdosfs is not already installed.
```
 sudo apt-get install dosfstools 
```
Unmount the drive (for example, /dev/sda1) if it is currently mounted.
```
 sudo umount /dev/sda1 
```

Format the drive using either FAT16 or FAT32.

 mkdosfs -F 16 /dev/sda1 # format FAT16 mkdosfs -F 32 /dev/sda1 # format FAT32

Tip

If you want to create a FAT-formatted USB floppy drive, then use the -I option. For example: sudo mkdosfs -I -F 32 /dev/sda.

If you do not mind restricting file sharing to Linux-only systems, then you can format the drive using an ext2 or ext3 file system using any of the following commands:

 mkfs /dev/sda1          # default format is ext2 mkfs -t ext2 /dev/sda1  # explicitly format type as ext2 mkfs -t ext3 /dev/sda1  # explicitly format type as ext3 mkfs.ext2 /dev/sda1     # directly call format ext2 mkfs.ext3 /dev/sda1     # directly call format ext3

Booting from a USB Drive

Beyond file sharing, USB drives can be used as bootable devices. If your computer supports booting from a USB drive, then this is a great option for developing a portable operating system, emergency recovery disk, or installing the OS on other computers.

Although most systems today support USB drives, the ability to boot from a USB thumb drive is not consistent. Even if you create a bootable USB drive, your BIOS may still prevent you from booting from it. It seems like every computer has a different way to change BIOS settings. Generally, you power on the computer and press a key before the operating system boots. The key may be F1, F2, F10, Del, Esc, or some other key or combination of keys. It all depends on your computer's BIOS. When you get into the BIOS, there is usually a set of menus, including one for the boot order. If you can boot from a USB device, this is where you will set it. However, every computer is different, and you may need to have the USB drive plugged in when you power-on before seeing any options for booting from it.

Warning

Making changes to your BIOS can seriously screw up your computer. Be careful!

Different USB Devices

Even if your computer supports booting from a USB device, it may not support all of the different USB configurations. In general, thumb drives can be configured one of three ways:

Small USB floppy drives-Thumb drives configured as USB floppy devices (that is,, no partitions) with a capacity of 256 MB or less are widely supported. If your computer cannot boot this configuration, then the chance of your computer booting any configuration is very slim.
Large USB floppy drives-These are USB floppy devices with capacities greater than 256 MB. My own tests used two different 1 GB thumb drives and a 250 GB USB hard drive.
USB hard drives-In my experience, this is the least-supported bootable configuration. I only have one computer that was able to boot from a partitioned USB hard drive.

Changing between a USB hard drive or USB floppy drive is as simple as formatting the base device or using fdisk and formatting a partition. However, converting a large USB floppy device into a small USB floppy device is not direct.

Use dd to create a file that is as big as the drive you want to create. For example, to create a 32 MB USB drive, start with a 32 MB file:
```
 dd if=/dev/zero of=usbfloppy.img bs=32M count=1 
```

Treat this file as the base device. For example, you can format it and mount it.

 mkfs usbfloppy.img sudo mkdir /mnt/usb sudo mount -o loop usbfloppy.img /mnt/usb

When you are all done configuring the USB floppy drive image, unmount it and copy it to the real USB device (for example, /dev/sda). This will make the real USB device appear as a smaller USB floppy device.
```
 sudo umount /mnt/usb dd if=usbfloppy.img of=/dev/sda 
```

The 10-Step Boot Configuration

Creating a bootable USB thumb drive requires 10 basic steps:

Unmount the drive. When you plug a USB drive into the computer, Ubuntu immediately mounts it. You need to unmount it before you can partition or format it.

Use the mount command to list the current mount points and identify the USB thumb drive. Be aware that the device name will likely be different for you. In this example, the device is /dev/sda1 and the drive label is NEAL.

 $ mount /dev/hda1 on / type ext3 (rw,errors=remount-ro) proc on /proc type proc (rw) /sys on /sys type sysfs (rw) varrun on /var/run type tmpfs (rw) varlock on /var/lock type tmpfs (rw) procbususb on /proc/bus/usb type usbfs (rw) udev on /dev type tmpfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) devshm on /dev/shm type tmpfs (rw) lrm on /lib/modules/2.6.15-26-686/volatile type tmpfs (rw) /dev/sda1 on /media/NEAL type vfat (rw,nodev,quiet,umask=077)

Use the unmount command to free the device:

 sudo umount /dev/sda1

Blank, or zero, the USB device. This is needed because previous configurations could leave residues that will interfere with future configurations. The simplest way to zero a device is to use dd. Keep in mind, large drives (even 1 GB thumb drives) may take a long time to zero. Fortunately, you usually only need to zero the first few sectors.
```
 dd if=/dev/zero of=/dev/sda            # format all of /dev/sda dd if=/dev/zero of=/dev/sda count=2048 # format the first 2048 sectors 
```
Use the sync command (sudo sync) to make sure all data is written. After zeroing the device, unplug it and plug it back in. This will remove any stale device partitions. Ubuntu will not mount a blank device, but it will create a device handle for it.
If you are making a USB hard drive, then partition the device.
Format the partitions. If you are making a USB floppy drive, then format the base device. For USB hard drives, format each of the partitions.
Mount the partition.
Copy files to the partition.
Place the kernel and boot files on the partition.
Configure the boot menus and options.
Use the sync command (sudo sync) to make sure all data is written and then unmount the partition.
Install the boot manager.

Now the device should be bootable. The next few sections show different ways to do these 10 steps.

Kicking Off the Network Install with a USB Drive

USB drives can be used to simplify system installs. For example, if the computer can boot from a USB drive then you can use it to kick off a network install.

Note

The preconfigured network boot image, boot.img, is very small-only 8 MB. It should work on all USB drives.

Configuring the thumb drive for use as a network installation system requires some simple steps:

Plug in the USB drive. If it mounts, unmount it.
Download the boot image. There is a different boot image for every platform. Be sure to retrieve the correct one.
```
 wget http://archive.ubuntu.com/ubuntu/dists/\ dapper/main/installer-i386/current/images/netboot/boot.img.gz 
```
The boot image is pre-configured as a USB floppy drive. Copy the image onto the thumb drive. Be sure to specify the base device (for example, /dev/sda) and not any existing partitions (for example, /dev/sda1).
```
 zcat boot.img.gz > /dev/sda 
```

Now you are ready to boot off the thumb drive and the operating system will be installed over the network.

Every PC that I tested with Boot from USB support was able to run the default network installer: boot.img.gz. However, since USB support is not consistent, this may not necessarily work on your hardware. If you cannot get it to boot, then make sure your BIOS is configured to boot from the USB drive, that it boots from the USB before booting from other devices, and that the USB drive is connected to the system. If you have multiple USB devices connected, remove all but the bootable thumb drive.

Using the Boot Image with Files

The boot.img.gz image is a self-contained file system and only uses 8 MB of disk space. If you have a bigger thumb drive (for example, 64 MB or 512 MB), then you can copy diagnostic tools or other stuff onto the drive.

In order to create a bootable USB drive, you will need a boot loader. The choices are GRUB or SYSLINUX. There are significant tradeoffs here. GRUB is the default boot loader used when Ubuntu is installed. However, GRUB requires knowing the drive identifier (for example, /dev/sda1). Since you may plug-in and remove USB devices, the identifier may change, breaking the boot loader's configuration. SYSLINUX does not use a static drive identifier, but is limited to supporting FAT12 or FAT16 drives. Since USB devices are expected to be portable, use SYSLINUX:

 sudo apt-get install syslinux mtools

The main steps require you to format the drive as FAT16 and use syslinux to make it bootable.

Become root.
```
 sudo bash 
```
Unmount the USB drive, if it is already mounted.
Format the drive as a FAT16 USB floppy drive (in this example, /dev/sdc) and mount it.
```
 mkdosfs -I -F 16 /dev/sdc sync mkdir /mnt/usb mount -o loop /dev/sdc /mnt/usb 
```

Mount the boot.img file. You will use this to provide the boot files.

 zcat boot.img.gz > boot.img mkdir /mnt/img mount -o loop boot.img /mnt/img

Copy the files over to the USB drive. This can take a few minutes.

 sudo bash  # become root, run these commands as root (cd /mnt/img ; tar -cf - *) | (cd /mnt/usb ; tar -xvf -) sync

Set up the files for a bootable disk. This is done by copying over the SYSLINUX configuration files for an ISO image (isolinux.cfg) to a configuration file for a FAT16 system (syslinux.cfg).
```
 mv /mnt/usb/isolinux.cfg /mnt /usb/syslinux.cfg rm /mnt/usb/isolinux.bin sync 
```

Unmount the drive and make it bootable by installing the boot loader.

 umount /mnt/usb syslinux /dev/sdc sync eject /dev/sdc exit  # leave the root shell

Now you can boot from the USB drive in order to install the operating system.

Installing a Full File System

The Holy Grail for USB hacking is the ability to boot a standalone operating system from a thumb drive. It is not fast, it may not have much space, but it sure is cool. Given a large enough USB drive and a computer that can boot from the USB port, you can configure a thumb drive as a standalone operating system.

Warning

There are many different methods discussed in online forums for configuring a USB drive as a bootable system. Unfortunately, most of the instructions are either incomplete or very complicated. Even if you follow these steps exactly, you may still be unable to boot from the USB device. The instructions listed here worked consistently for me, but I spent days trying to make it work.

There are two configurations for making a bootable file system: a huge USB floppy drive or a large USB hard drive. In both of these examples, I will use the Ubuntu Desktop Live CD as the bootable device.

Using the Live CD from a USB Floppy Drive

Converting the Live CD to a bootable USB floppy drive requires at least a 1 GB thumb drive.

Become root. This is done for convenience since nearly every command must be done as root.
```
 sudo bash 
```
Unmount and blank the thumb drive. (See the section "The 10-Step Boot Configuration" for directions.)

Format the disk as one big FAT16 drive. The -I parameter to mkdosfs says to format the entire device. In this example, the USB drive is /dev/sdc.

 mkdosfs -I -F 16 /dev/sdc sync

Tip

FAT16 only supports drives up to 2 GB. If you have a larger USB drive, then you will need to use the hack found in the Different USB Devices section to convert a large USB drive into a smaller one.

Mount the Live CD and the USB drive.

 mkdir /mnt/usb mkdir /mnt/img mount -o loop ubuntu-6.06-desktop-i386.iso /mnt/img/ mount /dev/sdc /mnt/usb

Copy over the files. This can take 20 minutes or longer. Go watch TV or have lunch. Also, ignore the errors about symbolic links since FAT16 does not support them.
```
 (cd /mnt/img ; tar -cf - *) | (cd /mnt/usb ; tar -xvf -) sync 
```

Set up the files for a bootable disk. Since SYSLINUX does not support subdirectories for kernel files, you need to move these to the top directory on the USB drive.

 # move the kernel files and memory tester mv /mnt/usb/casper/vmlinuz /mnt/usb/vmlinuz mv /mnt/usb/casper/initrd.gz /mnt/usb/initrd.gz mv /mnt/usb/install/mt86plus /mnt/usb/mt86plus # move boot files to top of the drive mv /mnt/usb/isolinux/* /mnt/usb/ mv /mnt/usb/isolinux.cfg /mnt/usb/syslinux.cfg rm /mnt/usb/isolinux.bin # Optional: Delete Windows tools and ISO files to free space rm -rf /mnt/usb/start.* /mnt/usb/autorun.inf rm /mnt/usb/bin /mnt/usb/programs rm -rf /mnt/usb/isolinux # All done sync

Edit the /mnt/usb/syslinux.cfg file and correct the kernel paths. Remove the paths /casper/ and /install/ wherever you see them. This is because Step 6 moved the files to the root of the USB drive. There should be eight occurrences of /casper/ and one for /install/. After you write your changes, run sync.

Unmount the drive and make it bootable.

 umount /mnt/usb syslinux /dev/sdc sync eject /dev/sdc exit  # leave the root shell

This USB thumb drive should now be bootable! You can run the Ubuntu Live operating system or install from this USB thumb drive.

For customization, you can change the boot menu by editing the /mnt/usb/syslinux.cfg file and modify the kernels.

Using the Live CD from a USB Hard Drive

Converting the Live CD to a USB hard drive is much more complicated. First, many computers that support booting from USB devices do not support this configuration. Even if the basic configuration is supported, there may be BIOS restrictions on the disk's layout. Second, the boot loader needs to support partitions. Finally, the USB drive's identifier cannot change after installation. This final issue is the main reason that I do not use GRUB or LILO as the boot loader.

Hard drives are defined by a combination of heads, sectors, and cylinders. Although heads and cylinders used to match physical drive heads and platters, this is no longer the case. In general, each sector contains 512 bytes of data, and each cylinder contains one set of heads × sectors. However, when booting from a USB hard drive, many BIOS manufacturers assume 64 heads and 32 sectors per cylinder. This is the configuration used by USB ZIP drives. If you use a different configuration, it may not boot. In addition, the first partition must not be larger than 1023 cylinders.

Although the syslinux command only supports FAT12 and FAT16 file systems, the syslinux package includes extlinux, which support ext2 and ext3 file systems. For this example, we will use extlinux as the boot loader with an ext2 file system on the bootable partition.

There are 10 steps to configure a bootable operating system on the USB drive:

Become root.
```
 sudo bash 
```
Unmount the drive.

Use fdisk of cfdisk to partition the drive (for example, /dev/sdc). Be sure to specify 64 heads and 32 sectors. The last cylinder of the first partition must not be larger than 1023. If you have additional disk space after allocating the first partition, then you can allocate additional partitions.

 # fdisk -H 64 -S 32 /dev/sdc Command (m for help): d No partition is defined yet! Command (m for help): n Command action    e   extended    p   primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-983, default 1): 1 Last cylinder or +size or +sizeM or +sizeK (1-983, default 983): 983 Command (m for help): a Partition number (1-4): 1 Command (m for help): p Disk /dev/sdc: 1030 MB, 1030750208 bytes 64 heads, 32 sectors/track, 983 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes    Device Boot      Start         End      Blocks   Id  System /dev/sdc1   *           1         983     1006576   83  Linux Command (m for help): w The partition table has been altered!

Warning

The partition must be marked as "active," otherwise you will not be able to boot from it.

Format the partition as an ext2 file system.
```
 mkfs /dev/sdc1 
```

Mount the Live CD and the USB drive.

 mkdir /mnt/usb mkdir /mnt/img mount -o loop ubuntu-6.06-desktop-i386.iso /mnt/img/ mount /dev/sdc /mnt/usb

Copy over the files. As mentioned in the previous section, this can take 20 minutes or longer.
```
 (cd /mnt/img ; tar -cf - *) | (cd /mnt/usb ; tar -xvf -) sync 
```
Create the boot files. Unlike syslinux, the boot files for extlinux can be located in a directory. In this case, we will reuse the casper directory since it already contains the kernel files.
```
 cp /mnt/usb/isolinux/* /mnt/usb/casper/ rm /mnt/usb/casper/isolinux.bin mv /mnt/usb/casper/isolinux.cfg /mnt/usb/casper/extlinux.conf sync 
```
Note
The extension for the boot configuration file is .conf, and not .cfg.
Do not unmount the drive yet! Making it bootable with extlinux requires the mounted directory containing the extlinux.conf file.
```
 extlinux -z /mnt/usb/casper sync 
```

Copy over the boot loader. There is a file missing from the syslinux binary package but available in the source package. This file is called mbr.bin and is a master boot record containing the boot loader. Download the source package:

 apt-get source syslinux

This creates a directory, such as syslinux-3.11/. In this directory is the missing file. Install it on the drive using:

 cat mbr.bin > /dev/sdc sync

Warning

If you are configuring a file image instead of the actual drive, then this cat command will truncate your file. Instead, use dd if=mbr.bin of=usbdrive.img bs=1 notrunc to install the master boot record to your USB drive image file (in this case, usbdrive.img).

Now, unmount the drive and boot from it.

If all goes well, you should have a working, bootable USB thumb drive. This drive can also be used as a bare-bones recovery and repair system.

As an alternative configuration, you can format the drive with FAT16 and use syslinux to make the partition bootable. In this case, you will also need to copy the boot files to the top of the partition and edit the syslinux.cfg file as described in the previous section.

Booting Variations

I used a variety of computers for testing the USB boot process. Every computer acted differently to different boot configurations.

Every computer with Boot from USB support was able to boot the original boot.img file.
Most computers were able to boot the Ubuntu Live operating system when my 1 GB thumb drive was formatted as a USB floppy drive. However, one computer gave a generic boot error message.
Only one computer could boot the USB hard drive with the ext2 file system. Using a real USB hard drive or thumb drive did not make any difference. In addition, specifying the USB ZIP configuration was the only way to make the hard drive configuration work.

Depending on the configuration variation and hardware that you use, you may see some well- known errors.

Blank screen-If all you see is a blank screen with a blinking cursor, then something definitely did not work. This happens when the boot loader fails. It could be due to failing to install the boot loader properly, or it could be a BIOS problem. Try rebuilding the USB drive in case you missed a step. Also, try booting the USB drive on a different computer. If it works on one computer and not on another, then it is a BIOS problem. But if it fails everywhere, then it is probably the boot loader.
"PCI: Cannot allocate resource region…"-This indicates a BIOS problem. You may be able to boot using additional kernel parameters to bypass the PCI errors, for example: live noapic nolapic pci=noacpi acpi=off

However, you may not be able to get past this. Check if there is a BIOS upgrade available for your computer.

"cdrom: open failed"-This error is generated by the initrd.gz file because Ubuntu's installer wants to run from a CD-ROM. You can either choose a different installer image, or edit this initrd.gz file to disable the failure. To edit initrd.gz:

Extract the file. The initrd.gz file is actually a compressed archive containing all of the executables, libraries, and configuration files needed during boot.

 mkdir extract cd extract zcat /mnt/usb/casper/initrd.gz | cpio -i

Note

If you have ever pressed Alt+F2 from the installation menu and noticed that there was a very minimal operating system with few commands available, this is it. The minimal operating system files are stored in the initrd.gz archive.

Edit the file scripts/casper. In the mountroot() function, comment out the following lines:

 # for i in 0 1 2 3 4 5 6 7 8 9 a b c d e f 10 11 12 13; do #     live_image=$(find_cd) #     if [ "${live_image}" ]; then #         break #     fi #     sleep 1 # done # if [ "$?" -gt 0 ]; then #    panic "Unable to find a CD-ROM containing a live file system" # fi

Right below the section you commented out, insert these lines:

 mount -t vfat -o ro /dev/sda $mountpoint live_image=$mountpoint/casper/filesystem.squashfs

This example assumes that you are using a USB floppy drive configuration (/dev/sda). If you are using a USB hard drive configuration, then replace /dev/sda with /dev/sda1.

Warning

The device identifier must match the identifier found by the booted thumb drive. This is not necessarily the same device identifier that you are currently using to configure the thumb drive.

Repackage the initrd.gz file and put it on the USB drive:

 find . | cpio -o --format='newc' | \   gzip -9 > /mnt/usb/casper/initrd.gz

Root not found-There are a variety of errors to indicate that the root partition was not available during boot. This is usually caused when the USB drive is still initializing or transferring data and is not ready for the root partition to be mounted. You can fix this by extracting the initrd.gz file and editing the conf/initramfs.conf file. Add in a WAIT line to delay mounting by 15 seconds, giving the USB time to initialize, configure, and transfer data.
```
 WAIT=15 
```
Now you can repackage the initrd.gz file and boot with it.

The hack for editing the initrd.gz file can also be used to add commands to the basic boot image. For example, you can add an editor or diagnostic tools to the /bin directory. However, be sure that the commands that you add do not use shared libraries. For example, file /usr/bin/vim.basic shows a dynamically linked executable, so you cannot use it unless you also include all of the dependent libraries. In contrast, any executable identified as statically linked is good to go! If you are compiling programs (see Chapter 4 for installing the compilation environment), then you can use gcc -static to generate statically linked executables.