Hack 67 Fix Broken Init Services

Errors in init scripts may prevent a system from completely booting. Use Knoppix to disable the troublesome scripts .

Before you log in to a Linux system, a number of programs are automatically run to start various daemons and load services that the system needs. If your system uses a graphical login, then that program is also started at boot. If one of these programs stalls, then you might not be able to completely boot to fix it. Depending on the service, it might be tricky to even bypass it using single- user mode. Knoppix can mount all major Linux filesystems, so it is a great rescue disk to use to temporarily disable a broken startup service so you can boot the system. This hack covers a few different scenarios that might stop a system from booting and how to use Knoppix to disable bad services.

6.17.1 Some Init-ial Background

Before learning how to disable services, it's important to understand Linux's startup process and how Linux determines which programs to run when it starts. For most Linux distributions, System V init scripts govern which programs start at boot and which programs don't. All System V init scripts that could potentially be run at boot are typically located in the /etc/init.d/ directory, although some distributions place them elsewhere within /etc . Not every script in /etc/init.d is executed at boot however. Linux organizes which scripts to run for different circumstances into runlevels ; most Linux systems have seven runlevels, ranging from zero to six. Think of a runlevel as a checklist of programs to start before it presents a login.

A few of these runlevels are set aside for special states in a Linux system:

Runlevel 0

Halts the system.

Runlevel 1

Sets up single-user mode.

Runlevels 2-5

Sets up different multiuser modes. Although, typically, only one or two are used by a distribution.

Runlevel 6

Reboots the system.

Each runlevel has a directory that stores symlinks to the init scripts in /etc/init.d , which are started when that runlevel is selected and stopped when it is exited. While the location of these directories differs a bit across Linux distributions, they are often found at /etc/rc runlevel .d for example, all runlevel 2 scripts are located in /etc/rc2.d/ .

If you look in one of these runlevel directories, you should notice that many of the symlinks to scripts in /etc/init.d have odd names that begin with an S, K, or D, then a number, and finally the name of the script. The letter at the beginning of each filename tells init when to execute this script. If the script begins with an S, then init starts the script when it goes through the runlevel. If the script begins with an K, then init stops (or kills ) the script when it changes to a different runlevel. If the script begins with an D, then that script is disabled for the time being and init ignores it. Init runs the scripts in alphabetical order, so the numbers in each script are just there to arrange the scripts in the order they are to be run. This is useful to ensure that dependent services start after the service they are dependent on.

When Linux boots and starts the init process, it reads its configuration from /etc/inittab , which configures each available runlevel, the default runlevel to use, as well as some other settings. Next, init loads any system scripts from a special system runlevel directory often at /etc/rcS.d . These scripts load daemons and services that are vital to the boot process. Lastly, init runs any startup scripts for the default runlevel in alphabetical order.

You can change the runlevel yourself on the command line with the init command. To switch to single-user mode from the command line, type:

 knoppix@ttyp0[knoppix]$  sudo init 1  

This command runs all of the shutdown scripts for your current runlevel and then any startup scripts for single-user mode. You can also start and stop scripts manually by running the script with the start or stop argument. For example, to start samba from the command line, run:

 knoppix@ttyp0[knoppix]$  sudo /etc/init.d/samba start  

6.17.2 Disable Broken Init Scripts

Now that you understand the basics of how services are started in Linux, it's time to learn how to disable a problem script. To disable a script, you must know your default runlevel. On Knoppix and Debian systems, the default runlevel is 2, but this varies across distributionsfor example, Red Hat defaults to runlevel 5. All runlevels are defined in /etc/inittab . To read this file, mount your root partition (for this example, /mnt/hda1 ) and run:

 knoppix@tty0[knoppix]  grep initdefault  /mnt/hda1    /etc/inittab  id:2:initdefault: 

Once you know your runlevel, you can disable the correct scripts. In general, to disable a script, change to the directory for that runlevel, identify the script name starting with an S, and then rename the file by changing the S to a D.

One common scenario that might prevent you from booting is a broken X configuration. Most Linux distributions today default to a desktop manager that lets you log in graphically. This desktop manager is typically set to run continuously so that even if X is stopped, it automatically restarts itself and presents you with another login screen. If X is broken, the desktop manager resets every five seconds or so, which makes it rather tricky to quickly switch to a virtual terminal, log in as root, and disable the program. Some desktop managers now set a maximum number of restarts before disabling the program for you, and you can also disable this script in a nongraphical or single-user runlevel, but for the purposes of demonstration, I explain how to disable the desktop manager.

In the previous example, the default runlevel for Knoppix is 2. Change to the directory for that runlevel on your root partition (in this example, /mnt/hda1/etc/rc2.d ). This directory has a number of scripts, but you want to disable the desktop manager, so you are interested only in scripts for xdm , gdm , or kdm the three common desktop managers. In your system, you must determine which desktop manager is actually used, or otherwise , simply disable any of the three that you see for that runlevel. Typically, the desktop manager is one of the last scripts to be run, so it is often numbered 99. This is a script that is started at boot, so the symlink starts with an S; to disable gdm , the desktop manager for Gnome, look for a file named S99gdm or something similar. To disable this script, rename it by changing the S to a D:

 knoppix@tty0[rc2.d]  sudo mv  S99gdm     D99gdm   

Of course, change the name of the script to xdm or kdm if you are disabling those services. After renaming the file, you should be able to reboot without the desktop manager starting, which gives you an opportunity to debug your X configuration.

Another scenario that might prevent you from booting is a stalled service that does not time out. Init runs each script in sequence, so if a script does not exit, any scripts after it do not run. I've had cases where an init script mounting networked filesystems stalled out and sat there for minutes without exiting, and I had to boot with a rescue disc to disable the service. In this case, the script was not part of my runlevel but was instead a system script, so I had to go to /etc/rcS.d/ to find and disable it. Avoid making changes to scripts in rcS.d unless you know what you are doing; these scripts are considered by the system to be important, if not essential, for booting.

Once you have disabled a service and booted the system, track down the cause of the problem. Remember that you can run a startup script after booting by running it from the command line with the start option. This allows you to test a broken script while still having it disabled should you need to reboot.