Troubleshooting and System Maintenance | Linux Patch Management: Keeping Linux Systems Up To Date

For this sample section, you need a computer that you're willing to dedicate for experimental purposes. Actual troubleshooting questions require the installation of the latest version of Red Hat Enterprise Linux, configured with a specific problem. The exam conditions would delete any and all data that you have on that computer. One option is a virtual machine solution such as that provided by VMware; as of this writing, no-cost subscriptions for VMware Server are available at www.vmware.com. I used VMware, using the tips that I describe in "Studying with a Virtual Machine" in the Online Learning Center (http://highered.mhhe.com/sites/0072264543), to write much of this book. Another alternative is Xen, also described in "Studying with a Virtual Machine."

If possible, get a friend, fellow student, or colleague to help set up the exercises for this exam. That's the best way to simulate real-world conditions. As shown in the RHCE and RHCT Exam Prep guide at www.redhat.com/training/rhce/examprep.html, you may have to boot into different runlevels, solve networking and host name problems, configure the GUI, add partitions of various types, and use standard command line tools to analyze and configure your system. You can use any documentation that you can find on your Red Hat Enterprise Linux computer; however, you're not allowed to reinstall Linux to address these problems.

You can't pass either exam unless you solve all of the RHCT-level troubleshooting problems. On the other hand, you need to budget your time judiciously; if you can't solve one RHCE-level problem, you may want to give up and move on to the next problem. But you may not be able to go back. You may be able to debug the next problem in just a few minutes. Even if you have time left over at the end of the section, you may not be able to go back and will not get any credit for any problems that you abandon.

These are not actual questions, but exercises consistent with the guidelines in the Red Hat Exam Prep guide. As exercises, they have no answers per se; however, they include a lot of information that can help you as a Linux administrator. However, I've set them up in a format that can allow someone else to set up exercises similar to what I'm guessing you might see on the Red Hat exams.

Even for these exercises, do not use a production computer. Some or all of these exercises are designed to make Linux unbootable. If you're unable to recover from the steps documented in these exercises, you may need to reinstall Red Hat Enterprise Linux. Saving any data that you have on your computer at that point may not be possible.

Troubleshooting and System Maintenance Exercise: RHCT Components

There are five problems on the RHCT part of this section. You have to reconfigure your computer to address all five problems appropriately, within the first hour of your exam. In this exercise, you'll set up five different problems on the same computer or virtual machine.

Ideally, you'll have a friend or classmate who can help you prepare your computer. Assume that you have a network server with the RHEL installation files. When your friend or classmate installs RHEL on this text computer, assume that you're on a network with an IP address of 10.20.30.0 (you can substitute a different network IP address if you want). Include the GNOME Desktop Environment in the installation. Then have your friend or classmate take the following steps:

Configure an RHEL system with some empty space on a hard drive. For the purpose of this section, I'll assume that you've configured a system with at least 5000MB of free space.
If you have a DHCP server on your network, disconnect or deactivate it.
Configure an IP address for your network card of 10.20.30.40, with a network mask of 255.255.255.0. You can use a tool such as system-config-network. Once saved, assign an IP address in /etc/hosts of 10.20.30.50 to the local host name; for example, on my /etc/hosts, I've configured:
```
 127.0.0.1   localhost.localdomain  localhost 10.20.30.50 enterprise5 
```
If necessary, configure this system to start in runlevel 3 by changing the appropriate command in /etc/inittab to:
```
 id:3:initdefault 
```
Make the system start with the KDE Display Manager for a login screen; open /etc/X11/prefdm, and a few lines inside the file, set:
```
 preferred=kdm 
```
Deactivate the named (DNS) services, and make sure it isn't active on reboot. The simplest way is with the following command:
```
 # chkconfig named off 
```
Back up and then edit /etc/X11/xorg.conf; delete the first EndSection directive.
Make sure SELinux is disabled the next time this system is booted. The quickest way is to set SELINUX=disabled in /etc/sysconfig/selinux.
Run the poweroff command. It is now ready for this part of the exam.
Now have the person taking the exam start the computer. Do not tell the candidate about the steps you took to prepare this computer. Tell him or her that there are problems with the configuration of the local network card (suggest that the candidate try pinging the local host name on the LAN) and an error associated with logging into the GUI. This computer should be configured to start the GUI automatically when Linux is booted, with a GDM login screen. Finally, the candidate should dedicate the remaining free space on the hard drive as a swap partition. Tell the candidate that you'll check his or her work after a reboot.

Troubleshooting and System Maintenance: RHCE Components

You have to solve three of these five simulated RHCE-level problems correctly. Your time starts after you've answered all of the simulated RHCT problems correctly; thus, you'll have at least 1.5 hours for this part of the exam. It's best if a friend or classmate helps you prepare your computer. Assume that you have access to a network server with the RHEL installation files.

You may want to have your friend or classmate prepare additional questions. Some boot exercises can be easily created from the Scenario & Solution list in Chapter 16. Other exercises on network services can be created based on what you've learned in Chapters 7 and 9–15. Exercises related to adding, removing, and resizing logical volumes can be developed from the information in Chapter 8.

In these exercises, you'll be working with an RHEL computer with an error in a key file created by a friend or classmate. Assume that you have a network server with the RHEL installation files. This should be on a system with some free unpartitioned space on the hard drive. Assuming you've run the RHCT exercise, you can use the swap partition created then; delete the associated entry from your /etc/fstab, and delete the partition using fdisk or parted. (Just don't remove both swap partitions.)

You'll also need a computer on which you can boot directly from your CD drive, so you can use the first Red Hat installation CD. (If you can boot from the USB, you can substitute a specially prepared boot USB drive described in Chapter 2.) Then have the person preparing your exam take the following steps:

If there are problems with the solutions in the RHCT troubleshooting section, you may need to reinstall or restore from a VMware snapshot.
Make sure to enable SELinux. Do not make any changes to the SELinux defaults.
Enable the firewall, and do not enable SMTP as a trusted service.
Open the /boot/grub/grub.conf configuration file, and create a typo in the boot stanza. For this exercise, change the initrd directive by removing the .img from the end of the file name; in other words for example, you might change
```
 initrd /initrd-2.6.18-8.el5.img 
```
to
```
 initrd /initrd-2.6.18-8.el5 
```
Install sendmail if needed; make sure the sendmail.mc file is in the original configuration, including the following directive:
```
 DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl 
```
In /etc/fstab, change the directive associated with the top-level root directory. Introduce a typo. For example, you might change
```
 /dev/VolGroup00/LogVol00 /  ext3 defaults  1 1 
```
to
```
 /dev/VolGroup00/LogVol0 /  ext3 defaults  1 1 
```
Define a standard root password, which you'll tell your partner or candidate.
Make sure you have a boot disc or USB key that can serve as a rescue disc. As the Red Hat Exam Prep guide requires the use of the first RHEL installation CD as a rescue disc, make sure that your computer can boot from this CD.
Have the person taking the exam start the computer. Do not tell the person about the steps you took to prepare this computer. Make sure the first RHEL installation CD is available. Advise that there may be a boot problem, that a mail server needs to be configured for more than just the local system, and provide the root user password that you created. Note that there is a firewall that can't be disabled for most services. Finally, advise the candidate to create a new swap partition of 100MB based on a logical volume created from existing free space.

Troubleshooting and System Maintenance Discussion

You shouldn't read this discussion until you've had a chance to try out the problems. I describe one possible solution to each problem. The solutions that you come up with can vary. The method you use doesn't matter; the results are what count. As of this writing, in the Troubleshooting and System Maintenance section, you can't reinstall RHEL on the target computer, and you can't go to the Internet for help during the exam.

Troubleshooting and System Maintenance Exam: RHCT-Level Problems

To pass either exam, you need to solve all five RHCT-level Troubleshooting and System Maintenance problems. In this simulation, you'll need to:

Find any problems with the configuration of the local network card; try pinging the local host name.
Make sure the computer is configured to start the GUI automatically when Linux is booted.
Solve the errors that keep you from logging into the GUI.
When booting into the GUI, make sure it boots into a GDM login screen.
Dedicate the remaining free space on the hard drive as a swap partition.

To accomplish this, take the following steps (there may be other ways of solving these problems):

When you first start this computer, watch the start messages carefully. Make a note of anything unusual.
Try pinging the local host name. What happens? What are sources for local host names? Is DNS active? If not, where does the ping command search?

Make any needed changes to /etc/hosts, and make sure it corresponds to what you see when you run ifconfig on the local network card.
To make sure the system boots directly into the GUI, you need to make sure the default runlevel in /etc/inittab is appropriate (runlevel 5). The system may offer you the chance to run the Display Configuration tool. It might work. If there are still problems, it's possible that some needed services aren't active in runlevel 5, or perhaps there are problems with a key configuration file, such as /etc/X11/xorg.conf.
When you move into runlevel 5, perhaps by reboot or with the init 5 command, what happens? Based on what you know from the book, what could be the cause? Check the space available in partitions associated with the /home and /tmp directories, or even some sort of error related to the X Window configuration files, including xorg.conf and xinitrc. If it's service related, make sure that service starts the next time you boot this system.
If that all works, you'll see an X login screen when you start or boot in runlevel 5. But remember that you need to make sure you're booted into the GDM screen.
Now to dedicate the remaining free space as a swap partition, open fdisk or parted, and create a new partition with the available free space. Make sure it's associated with the Linux swap file type, and format it with the mkswap command.

Once created, you'll then want to add it to /etc/fstab, as a swap partition. For example, if the new freshly formatted partition is /dev/hda10, you could add the following directive to /etc/fstab:
```
 /dev/hda10   swap   swap    defaults   0 0 
```
Reboot your computer to make sure your changes work.
Show your partner or instructor what you did.

Troubleshooting and System Maintenance: RHCE-Level Problems

You'll see what's wrong when you boot RHEL. Be prepared to use the first RHEL installation CD to rescue this system. If you get instructions for a network source for the RHEL installation files, take careful notes. You've been told that:

There may be a boot problem.
A mail server needs to be configured for more than just the local system.
You know the root user password created by your partner or instructor.
You need to create a new swap partition of 100MB based on a logical volume created from existing free space.
There may also be related issues associated with a firewall.

When you see your boot loader, select your current version of RHEL.
Watch the messages as they scroll across the screen. You'll probably see a message similar to:
```
 Error 15: File Not Found 
```
Chances are, this is related to the last message you've seen, probably something like this:
```
 initrd /initrd-2.6.18-8.el5 
```
If you're paying attention, you'll realize that this file name isn't right. You can confirm this from the GRUB command line; reboot and press e (unless there's password protection) to see the commands associated with the stanza. Make a note of the directives, then press c for a grub> prompt.
At the grub> prompt, type the root directive, probably something like:
```
 grub> root (hd0,0) 
```
Assuming you get a proper response, you should be able to follow up with the other commands associated with the stanza, as you saw in step 4. The error was "file not found." Use command completion to make sure the files cited actually exist.
After entering the noted commands, enter the boot command. If it's correct, Linux will boot normally and you won't even need the first Red Hat installation CD.
Well, Linux isn't booting normally yet. You'll see an error message like:
```
 fsck.ext3: No such file or directory while trying to open /dev/VolGroup00/LogVol0 
```
followed by the following prompt; enter the root password (as defined by your proctor or instructor).
```
 Give root password for maintenance (or type Control-D to continue): 
```
Run the mount command to see mounted filesystems. You should see a message like this:
```
 /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw) 
```
This should give you a hint to the problem. You should be able to confirm that there is no /dev/VolGroup00/LogVol0 device, but /dev/VolGroup00/LogVol00 and /dev/VolGroup00/LogVol01.
But even though the output suggests that files in this volume are writable, it's not so. To make it writable, run the following command:
```
 # mount -o remount /dev/VolGroup00/LogVol00 / 
```
Now you should be able to modify your /etc/fstab. You should see the typo now. Change
```
 /dev/VolGroup00/LogVol0 on / type ext3 (rw) 
```
to (change is bolded)
```
 /dev/VolGroup00/LogVol00 on / type ext3 (rw) 
```
You should also be able to mount the /boot directory; if /etc/fstab is in sufficient order, simply run
```
 # mount /boot 
```
You can now open the GRUB configuration file, /boot/grub/grub.conf, where you can correct the typo that caused the original "file not found" problem.
Reboot to make sure you've solved the boot problems.
Install the mail server of your choice, sendmail or postfix. If it's sendmail, comment out the following directive by adding a dnl in front:
```
 DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl 
```
Make sure to activate the sendmail service and use chkconfig to activate it the next time you boot:
```
 # service sendmail start # chkconfig sendmail on 
```
Of course, similar actions are possible if you install postfix, but that's another option left for the candidate.
Don't forget to make sure that mail (SMTP) services are allowed through any existing firewall; it's easiest with the Security Level Configuration tool.
Check the current space allocated to swap with the top command.
Finally, you can create the new swap volume of 100MB from existing free space. All you need for a logical volume is one partition, created to the Linux LVM file type. You can use fdisk or parted for this purpose. Remember, if you use fdisk, run partprobe to reread the partition table. Then create a PV, assign the space to a VG, and allocate it to an LV. You can use commands such as pvcreate, vgcreate, and lvcreate, or the Logical Volume Management GUI tool. For example, if you've created an LVM partition on /dev/hda10, run the following commands:
```
 # pvcreate /dev/hda10 # vgcreate vgrp1 /dev/hda10 # lvcreate --size=100M vgrp1 /dev/hda10 
```
You'll see the following message from the lvcreate command:
```
 Logical volume "lvol0" created 
```
This should create a new device, /dev/vgrp1/lvol0; you can confirm the new volume with the lvs command.
Now format the new device:
```
 # mkswap /dev/vgrp1/lvol0 
```
Add the new swap device to your /etc/fstab, to make sure it's read the next time you boot Linux. On my system, I did that with the following directive:
```
 /dev/vgrp1/lvol0   swap   swap defaults  0 0 
```
You can activate the new device with the swapon /dev/vgrp1/lvol0 command, check active LVs with the lvs command, and see the new space allocated to swap with the top command.
If there are SELinux-related issues, review the Setroubleshooter, which can be started in the GUI with the sealert -b command. Follow any recent recommendations, especially since your test system was last booted.
Reboot the system, and make sure the changes you've made are implemented.