Troubleshooting and Using Rescue Disks

As a system administrator, you ll need to examine and edit a number of configuration files. When changes are made, mistakes are possible. For example, if you make a mistake in editing the GRUB configuration file, you might see the following message the next time you boot Linux:

 root (hd0,1) Filesystem type is ext2fs, partition type 0x83 kernel /vmlinuz-2.4.20-8 ro root=LABEL=/ Error 15: File not found Press any key to continue 

This is just one of many possible boot problems. Sometimes the boot disk floppy that you created during the installation process can help. If you ve misplaced this disk, the mkbootdisk command can help.

However, the boot disk may not help you in every case. And if you don t have a boot disk, Red Hat Linux has other automated recovery options. You can use any standard installation boot disk, even the first installation CD, to rescue your Linux system. Depending on the problem, you could select the automated recovery process, or start Linux in single- user mode.

Tip  

There s an optimized rescue CD that Red Hat sometimes makes available for download from ftp.redhat.com . It includes text editors such as emacs and pico , as well as network utilities such as Telnet and FTP. Look for a file named rescue-cd.iso on the Red Hat FTP site. Use the techniques described in Chapter 14 to set up your CD.

The Specialized Boot Disk

The easiest way to get around the specified problem is with a boot disk. The boot disk that you should have created during Red Hat Linux installation is customized for this purpose. As long as you haven t changed the way partitions are organized, the custom boot disk should start your Linux system.

It s easy to create a new boot disk with the mkbootdisk versionnumber command, where you use the version number associated with your Linux kernel. For example, if the kernel shown in the /boot directory is vmlinuz-2.4.21-13 , the following command creates a customized boot disk on a 1.44MB floppy:

 # mkbootdisk 2.4.21-13 Insert a disk in /dev/fd0. Any information on the disk will ? be lost. Press <Enter> to continue or ^C to abort:_ 

Just remember to test your customized boot disk as soon as possible. You don t want problems with this disk when you re trying to rescue your Linux system.

Rescue Mode

Customized boot disks don t solve all possible Linux boot problems. Fortunately, you re not out of luck. Even if you ve lost your customized boot disk, Red Hat s linux rescue mode will normally get you into your Linux system. Once you ve started Linux, you can restore or repair any damaged files that you have.

Tip  

To use linux rescue mode, you need access to the Red Hat Linux installation files. If you re starting from a network boot disk, you need the address and location of the /RedHat directory. See Chapter 04 for examples.

You can start linux rescue mode from any Red Hat Linux installation boot disk. If you don t have one available, you can download it from ftp.redhat.com or associated mirror sites. You can even create installation boot disks on a Microsoft Windows computer using the RAWRITE.EXE utility discussed in Chapter 03 . Just type linux rescue at the boot: prompt for installing Red Hat Linux:

 boot:  linux rescue  

At this point, you may wonder if you did the right thing, because Red Hat Linux takes you through the first two steps of a standard installation: language and keyboard type. If you used a Red Hat Linux installation floppy, you re taken to Figure 11.7, where you ll also need to enter the location (local or network) of the Red Hat Linux installation files. Refer to Chapter 04 if you need more information.

Figure 11.7: Enter the location of the Red Hat Linux installation files here

Now linux rescue mode presents a menu with three different options:

Continue If you select Continue, Red Hat Linux searches your hard disk for your installation. All located filesystems are mounted as subdirectories of /mnt/sysimage . I think of this as automatic rescue mode.

Read-Only The Read-Only option is almost identical, except that located filesystems are mounted in read-only mode. You can think of this as read-only rescue mode.

Skip The Skip option proceeds directly to a root shell prompt in single-user mode. No attempt is made to look through available filesystems. I view this as a manual rescue mode.

Once you ve made the necessary changes, type the exit command. Repeat as needed until you see messages regarding termination signals. Linux should unmount all filesystems and then automatically reboot your computer.

Automatic Rescue Mode

If automatic rescue mode is successful, Red Hat Linux mounts all appropriate filesystems from /etc/fstab on /mnt/sysimage . In this case, the df command reflects the mounted directories, as shown in Figure 11.8.

Figure 11.8: Rescue mode mounts

In Figure 11.8, the CD is mounted on /mnt/source , /dev/sda2 is mounted on /mnt/sysimage , and /dev/sda1 is mounted on /mnt/sysimage/boot . While it s easy to see that /dev/sda1 is associated with /boot and /dev/sda2 is associated with root ( / ), you can confirm this with the e2label partitiondevice command:

 #  e2label /dev/sda1  /boot 

But what if automatic rescue mode can t mount all of your filesystems? In this case, you might see an error message such as the following:

 Error mounting filesystem on sdc1: Invalid argument 

Simply continue with automatic rescue mode. Linux mounts as many filesystems as it can. In this case, you can work on any damage to an unmounted filesystem such as /dev/sdc1 .

If you have one or more unmounted filesystems, the first two things to check are the fstab configuration file and the integrity of the format itself. At this point, you can use the vi editor to check fstab , but since the root ( / ) directory is actually mounted on /mnt/sysimage , you ll need the following command to open up fstab :

 # vi /mnt/sysimage/etc/fstab 

Alternatively, to clean up a damaged, unmounted filesystem, use the fsck devicename command. For example, to check /dev/sdc1 , run the following command:

 # fsck /dev/sdc1 

Tip  

If you want to access the Linux man pages in linux rescue mode, run the chroot /mnt/sysimage command. This restores your top-level root ( / ) directory to the top of the hierarchy, activating the standard paths to the Linux man pages.

Read-Only Rescue Mode

The only difference between read-only and automatic rescue mode is that all filesystems are mounted in read-only mode. This may be the best choice if you have a large number of filesystems, such as with a typical server installation of Red Hat Linux.

You can remount any desired filesystem in read-write mode. For example, the following command remounts partition device /dev/sda2 on the root ( / ) directory in read-write mode:

 # mount -w -o remount /dev/sda2 / 

Note  

This command is equivalent to the mount -o remount,rw / command described in Chapter 07 .

Manual Rescue Mode

Sometimes linux rescue mode can t find any of your filesystems. Don t panic; the problem could be as simple as an error in the name of /etc/fstab . Manual rescue mode is most appropriate here.

This mode loads a minimal root image and the kernel to a RAM disk, and then sends you to a root shell prompt ( # ). No filesystems are mounted; you have access only to a basic set of commands, such as mount , mkdir , mv , cp , rm , fdisk , and fsck . Once you ve mounted a directory, you can also use the vi editor to change the files that you need.

But remember, this is a minimalist version of Linux. You don t have all the commands that you might be used to at this level.

In manual rescue mode, the first step is to mount the partition associated with your root ( / ) directory in a temporary location such as /mnt/sysimage . This should allow you access to additional commands from directories such as /bin , /sbin , and /usr/sbin .

Single-User Mode

There s one other method, known as single-user mode , that you can use to log into a damaged Linux system. If Linux can find your root ( / ) directory filesystem, it can start Linux in this mode. As described earlier, single-user mode, also known as runlevel 1, requires only two services.

Once you ve made any required changes, you don t have to reboot. The exit command automatically moves you to the default runlevel as defined in /etc/inittab . Alternatively, the init 3 or init 5 commands can immediately start those respective runlevels. Single-user mode is also useful for changing the root password. If you forgot the password, run the passwd command in single-user mode. The password that you enter becomes the new root password.

Sometimes you ll encounter a problem such as a bad /etc/fstab file or an unmountable filesystem during the boot process. In this case, you ll see a prompt similar to that shown in Figure 11.9. When you enter the root password at the prompt, Linux starts in single-user mode.

Figure 11.9: Dropping to single-user mode

You can also start single-user mode from the GRUB menu. As described earlier, it s easy to protect GRUB with a password. If you don t see GRUB editing options as shown in Figure 11.10, enter the p command, and then enter the GRUB password.

Figure 11.10: GRUB editing commands

Highlight the Linux operating system of your choice, and then press the a command to modify the kernel arguments. GRUB should take you to a line such as the following:

 grub append> ro root=LABEL=/ 

At this point, you can add a command to the end of this line, such as single , 1 , or init=/bin/sh , as shown in Figure 11.11.

Figure 11.11: Modifying GRUB for single-user mode

When you press Enter, Red Hat Linux proceeds to boot in single-user mode, runlevel 1. At this point, you can fsck unmounted drives , edit configuration files, check the status of LVM partitions, and more.