rc Scripts

The rc script is not a big, monolithic script that starts all the processes needed for Linux services such as sshd, syslog, xinetd, and so on. Rather, rc runs a small script in /etc/init.d for each required Linux service. Each service script both starts and stops the service. Here is an example of the cron service script:

-rwxr-xr-x    1 root    root      1297 Mar 3 2005 /etc/rc.d/init.d/crond

The rc script knows which service scripts to start or stop for each runlevel by using directories populated with links to the /etc/init.d startup and shutdown service scripts. These directories could be /etc/init.d/rc#.d or /etc/rc.d/rc#.d, where # is the runlevel to execute. Runlevels are defined by the service scripts that start or stop services at that level. This example shows all the cron startup and shutdown links:

lrwxr-xr-x    1 root     root           15 Feb 9 2005 /etc/rc.d/rc0.d/K60crond -> ../init.d/crond lrwxr-xr-x    1 root     root           15 Feb 9 2005 /etc/rc.d/rc1.d/K60crond -> ../init.d/crond lrwxr-xr-x    1 root     root           15 Feb 9 2005 /etc/rc.d/rc2.d/S90crond -> ../init.d/crond lrwxr-xr-x    1 root     root           15 Feb 9 2005 /etc/rc.d/rc3.d/S90crond -> ../init.d/crond lrwxr-xr-x    1 root     root           15 Feb 9 2005 /etc/rc.d/rc4.d/S90crond -> ../init.d/crond lrwxr-xr-x    1 root     root           15 Feb 9 2005 /etc/rc.d/rc5.d/S90crond -> ../init.d/crond lrwxr-xr-x    1 root     root           15 Feb 9 2005 /etc/rc.d/rc6.d/K60crond -> ../init.d/crond

The rc script (/etc/rc.d/rc or /etc/init.d/rc) links to service scripts starting with S in either /etc/init.d/rc#.d or /etc/rc.d/rc#.d. For the syntax /etc/rc.d/rc 5, all the /etc/init.d/rc5.d/S* or /etc/rc.d/rc5.d/S* service scripts are executed. The following is from a Red Hat 9.0 system^[4]:

#ls -al /etc/rc.d total 76 drwxr-xr-x   10 root     root        4096 Dec 12 15:18 . drwxr-xr-x   70 root     root        8192 Dec 16 04:08 .. drwxr-xr-x    2 root     root        4096 Dec 12 15:52 init.d -rwxr-xr-x    1 root     root        2338 Feb 18 2003 rc drwxr-xr-x    2 root     root        4096 May 18 2004 rc0.d drwxr-xr-x    2 root     root        4096 May 18 2004 rc1.d drwxr-xr-x    2 root     root        4096 May 18 2004 rc2.d drwxr-xr-x    2 root     root        4096 Aug 20 08:53 rc3.d drwxr-xr-x    2 root     root        4096 May 18 2004 rc4.d drwxr-xr-x    2 root     root        4096 Aug 20 08:53 rc5.d drwxr-xr-x    2 root     root        4096 May 18 2004 rc6.d -rwxr-xr-x    1 root     root         220 Jul 10 2001 rc.local -rwxr-xr-x    1 root     root       23299 Feb 24 2003 rc.sysinit

The entries in rc#.d are symbolic links. The actual scripts are in /etc/init.d. These links begin with either K or S. The S links are for startup scripts, and the K links are for shutdown scripts. The numbers following the S or K are used to order the execution of the scripts. When moving to a new runlevel, the shutdown scripts are run, followed by the startup scripts. Let's look more closely at the startup scripts in rc5.d:

# ls -al /etc/rc.d/rc5.d/S* lrwxrwxrwx    1 root     root           15 May 18 2004 /etc/rc.d/rc5.d/S05kudzu  -> ../init.d/kudzu lrwxrwxrwx    1 root     root           18 May 18 2004 /etc/rc.d/rc5.d/S08iptables -> ../init.d/iptables lrwxrwxrwx    1 root     root           17 May 18 2004 /etc/rc.d/rc5.d/S10network -> ../init.d/network lrwxrwxrwx    1 root     root           16 May 18 2004 /etc/rc.d/rc5.d/S12syslog -> ../init.d/syslog lrwxrwxrwx    1 root     root           17 May 18 2004 /etc/rc.d/rc5.d/S13portmap -> ../init.d/portmap lrwxrwxrwx    1 root     root           17 May 18 2004 /etc/rc.d/rc5.d/S14nfslock -> ../init.d/nfslock ... (rest omitted)

The script name is the same as the link name without the leading S or K and numbers.

#ls -al /etc/init.d lrwxrwxrwx    1 root     root           11 Nov  6  2003 /etc/init.d -> rc.d/init.d #ls /etc/rc.d/init.d aep1000  firstboot  isdn       network     random      squid      xinetd anacron  FreeWnn    kdcrotate  nfs         rawdevices  sshd       ypbind apmd     functions  keytable   nfslock     rhnsd       syslog     yppasswdd atd      gpm        killall    nscd        saslauthd   tux        ypserv autofs   halt       kudzu      ntpd        sendmail    vncserver  ypxfrd bcm5820  httpd      linuxcoe   pcmcia      single      vsftpd canna    innd       lisa       portmap     smb         webmin crond    iptables   named      postgresql  snmpd       winbind cups     irda       netfs      pxe         snmptrapd   xfs

For Red Hat, the initlog command is called to run the individual service startup scripts and log the output using syslogd. The /etc/initlog.conf file defines local7 as the syslog facility for the messages. Looking at this /etc/syslog.conf excerpt, we can see that the boot messages are sent to /var/log/messages and boot.log:

# Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;news.none;authpriv.none;cron.none     /var/log/messages ... (lines omitted) # Save boot messages also to boot.log local7.*                                               /var/log/boot.log

The initlog(8) and syslogd(8) man pages have further details.

For SUSE, the blogger command sends messages to /var/log/boot.msg. See the blogger(8) man page for details.

It would be cumbersome to manage all the symbolic links needed for an rc start or stop script. Let's look at crond as an example. It runs at runlevels 2 through 5, so it has a start script for each runlevel. It does not run at levels 0, 1, and 6, so it has kill scripts for these levels. That makes seven symbolic links for just crond.

#find /etc/rc.d -name *crond /etc/rc.d/init.d/crond /etc/rc.d/rc0.d/K60crond /etc/rc.d/rc1.d/K60crond /etc/rc.d/rc2.d/S90crond /etc/rc.d/rc3.d/S90crond /etc/rc.d/rc4.d/S90crond /etc/rc.d/rc5.d/S90crond /etc/rc.d/rc6.d/K60crond

Fortunately, the chkconfig command is provided to add and remove the links as needed. The chkconfig(8) man page lists all the options, but the most useful options are provided in Table 1-4.

Table 1-4. chkconfig Syntax

Option	Purpose
chkconfig --list	Lists all rc scripts and their runlevels
chkconfig --list <script name>	Lists a single script and its runlevel
chkconfig --level <levels> <script name> <on or off>	Turns a script on or off at specified levels
chkconfig <script> reset	Sets the configuration for a script to defaults

As an example, let's manipulate crond. First we determine what the current settings are:

#chkconfig --list crond crond           0:off    1:off    2:on    3:on    4:on    5:on    6:off

Next we turn off crond at runlevel 2:

#chkconfig --level 2 crond off #chkconfig --list crond crond           0:off     1:off   2:off   3:on    4:on    5:on    6:off

Now we look at the symbolic links to see whether anything changed:

#find /etc/rc.d -name *crond /etc/rc.d/init.d/crond /etc/rc.d/rc0.d/K60crond /etc/rc.d/rc1.d/K60crond /etc/rc.d/rc2.d/K60crond /etc/rc.d/rc3.d/S90crond /etc/rc.d/rc4.d/S90crond /etc/rc.d/rc5.d/S90crond /etc/rc.d/rc6.d/K60crond

We can see that rc2.d has the K60crond stop script instead of the S90crond start script. We can return the crond configuration to the default values:

#chkconfig crond reset #chkconfig --list crond crond           0:off     1:off   2:on    3:on    4:on    5:on    6:off

The chkconfig command uses the following line in /etc/rc.d/init.d/crond to determine the default values and link names:

# chkconfig: 2345 90 60

Linux distributions use different methods to encode the default runlevel values in startup and shutdown scripts. The previous example was from a Red Hat 9.0 crond script. A SUSE 9.0 cron script has the following:

### BEGIN INIT INFO # Provides:       cron # Required-Start: $remote_fs $syslog $time # X-UnitedLinux-Should-Start: sendmail postfix # Required-Stop:  $remote_fs $syslog # Default-Start:  2 3 5 # Default-Stop:   0 1 6 # Description:    Cron job service ### END INIT INFO

We must mention one more directory. The /etc/sysconfig directory contains configuration files for the rc scripts. Here is a typical listing:

ls /etc/sysconfig apmd         grub              mouse               redhat-config-                                                    securitylevel apm-scripts  harddisks         named               redhat-config-users authconfig   hwconf            netdump             redhat-logviewer autofs       i18n              netdump_id_dsa      rhn clock        init              netdump_id_dsa.pub  samba console      installinfo       network             sendmail desktop      ip6tables-config  networking          squid devlabel     iptables-config   network-scripts     syslog dhcpd        irda              ntpd                tux dhcrelay     irqbalance        pcmcia              vncservers firstboot    keyboard          prelink             xinetd gpm          kudzu             rawdevices          yppasswdd

These configuration files contain variables for the rc scripts. The configuration files are sourced by the rc scripts from which they get their name. As we can see, /etc/sysconfig includes some directories, such as the network directory. The /etc/sysconfig files are small. The sendmail script, for example, consists of only two lines:

# cat sendmail DAEMON=yes QUEUE=1h

Red Hat provides the ntsysv command to manipulate startup/shutdown script configuration. It is not as powerful as chkconfig, but it is easier to use.

Linux gives us a way to control what scripts run at boot time. Sure, a system administrator can use chkconfig to configure which scripts run and which don't, but wouldn't it be nice to pick and choose during boot up? The Linux confirm mode provides this feature.

Confirm Mode

You can use the rc script to prompt whether each script should be run during startup. This feature is useful when one or more scripts need to be skipped for whatever reason.

To run rc in confirm mode, add the keyword confirm to the kernel command line, just as you would add the keyword single to boot to single user mode. This can be done from the bootloader, as Figure 1-15 shows.

Figure 1-15. Booting confirm mode with LILO

Figure 1-16 shows how the Ethernet rc script hangs if the LAN card is configured to get an IP address through DHCP but is not connected to the network. The user must sit and wait for the timeout from DHCP.

Figure 1-16. Boot hanging at eth1 configuration

If Linux is started in confirm mode, the script can be skipped. Press y to run the script, or press n to skip it. Press c to run the script and all the following scripts. This is a nice way to skip a script and not have to change the configuration. Figure 1-17 shows how this looks.

Figure 1-17. Skipping eth1 configuration in confirm mode

Startup Problems in rc Scripts

Problems with startup scripts can be difficult to troubleshoot because many files are involved. The problem could be with the rc script, one of the scripts rc is trying to run, or any command or script rc relies on. Figure 1-18 demonstrates a typical example, in which the Linux kernel boots and then displays some errors.

Figure 1-18. Boot error from rc

It is a good idea to stop and write down all the messages and errors before they scroll off the screen. We have a limited opportunity to fix problems with Linux in this state because / is mounted as read-only. However, we can troubleshoot and hopefully find the problem.

The following command not found errors from Figure 1-18 stand out:

/etc/rc.d/rc.sysinit: line 81: action: command not found grep: /proc/mounts: No such file or directory /etc/rc.d/rc.sysinit: line 93: action: command not found /etc/rc.d/rc.sysinit: line 140: action: command not found /etc/rc.d/rc.sysinit: line 169: action: command not found

We edit the file with vi just to check the line numbers:

*** An error occurred during the file system check. *** Dropping you to a shell; the system will reboot *** when you leave the shell. Give root password for maintenance (or type Control-D to continue): (Repair filesystem) 1 # vi /etc/rc.d/rc.sysinit

Enter :se nu to turn on line numbers. Enter :81G to go to line 81. In Figure 1-19, you can see that the rc.sysinit script is calling the subroutine named action. The other line numbers call action as well.

Figure 1-19. Editing /etc/rc.d/rc.sysinit

We could look around to see where the action subroutine is located, but it might have been removed. Let's verify that the startup script files are all in place and the correct size. We can use rpm, but we need to determine what package to verify. We use the rpm command to learn what delivered the rc script:

(Repair filesystem) 2 # rpm -q -f /etc/rc.d/rc initscripts-7.14-1

Now we verify that the initscripts-7.14-1 rpm is intact:

(Repair filesystem) 3 # rpm -V initscripts-7.14-1 S.5....T c /etc/inittab S.5....T c /etc/rc.d/init.d/functions

The output looks pretty cryptic, but the rpm(8) man page^[5] gives a good explanation:

The format of the output is a string of 8 characters, a possible "c" denoting a configuration file, and then the file name. Each of the 8 characters denotes the result of a comparison of attribute(s) of the file to the value of those attribute(s) recorded in the database. A single "." (period) means the test passed, while a single "?" indicates the test could not be performed (e.g. file permissions prevent reading). Otherwise, the (mnemonically emboldened) character denotes failure of the corresponding --verify test:        S file Size differs        M Mode differs (includes permissions and file type)        5 MD5 sum differs        D Device major/minor number mis-match        L readLink(2) path mis-match        U User ownership differs        G Group ownership differs        T mTime differs

So, the output means the size and timestamp of the inittab and functions files have changed since the files were delivered. This can be expected for inittab, but why for functions? Let's look:

(Repair filesystem) 5 # ls -al /etc/inittab -rw-r--r--    1 root     root       1807 Dec 17 15:52 /etc/inittab (Repair filesystem) 6 # ls -al /etc/rc.d/init.d/functions -rwxr-xr-x    1 root     root          0 Dec 21 14:39 /etc/rc.d/init.d/functions

It looks like function was zeroed out. We need to restore this file. Because / is mounted as read-only, the box should be booted from a different source. We can use a rescue CD, rescue disk, or second boot disk. When Linux is up, we just mount / to a temporary mount point and restore the functions file. Rescue disks are explained in the next section.