The rc script is not a big, monolithic script that starts all the processes needed for Linux services such as sshd, syslog, xinetd, and so on. Rather, rc runs a small script in /etc/init.d for each required Linux service. Each service script both starts and stops the service. Here is an example of the cron service script:
-rwxr-xr-x 1 root root 1297 Mar 3 2005 /etc/rc.d/init.d/crond
The rc script knows which service scripts to start or stop for each runlevel by using directories populated with links to the /etc/init.d startup and shutdown service scripts. These directories could be /etc/init.d/rc#.d or /etc/rc.d/rc#.d, where # is the runlevel to execute. Runlevels are defined by the service scripts that start or stop services at that level. This example shows all the cron startup and shutdown links:
lrwxr-xr-x 1 root root 15 Feb 9 2005 /etc/rc.d/rc0.d/K60crond -> ../init.d/crond lrwxr-xr-x 1 root root 15 Feb 9 2005 /etc/rc.d/rc1.d/K60crond -> ../init.d/crond lrwxr-xr-x 1 root root 15 Feb 9 2005 /etc/rc.d/rc2.d/S90crond -> ../init.d/crond lrwxr-xr-x 1 root root 15 Feb 9 2005 /etc/rc.d/rc3.d/S90crond -> ../init.d/crond lrwxr-xr-x 1 root root 15 Feb 9 2005 /etc/rc.d/rc4.d/S90crond -> ../init.d/crond lrwxr-xr-x 1 root root 15 Feb 9 2005 /etc/rc.d/rc5.d/S90crond -> ../init.d/crond lrwxr-xr-x 1 root root 15 Feb 9 2005 /etc/rc.d/rc6.d/K60crond -> ../init.d/crond
The rc script (/etc/rc.d/rc or /etc/init.d/rc) links to service scripts starting with S in either /etc/init.d/rc#.d or /etc/rc.d/rc#.d. For the syntax /etc/rc.d/rc 5, all the /etc/init.d/rc5.d/S* or /etc/rc.d/rc5.d/S* service scripts are executed. The following is from a Red Hat 9.0 system:
#ls -al /etc/rc.d total 76 drwxr-xr-x 10 root root 4096 Dec 12 15:18 . drwxr-xr-x 70 root root 8192 Dec 16 04:08 .. drwxr-xr-x 2 root root 4096 Dec 12 15:52 init.d -rwxr-xr-x 1 root root 2338 Feb 18 2003 rc drwxr-xr-x 2 root root 4096 May 18 2004 rc0.d drwxr-xr-x 2 root root 4096 May 18 2004 rc1.d drwxr-xr-x 2 root root 4096 May 18 2004 rc2.d drwxr-xr-x 2 root root 4096 Aug 20 08:53 rc3.d drwxr-xr-x 2 root root 4096 May 18 2004 rc4.d drwxr-xr-x 2 root root 4096 Aug 20 08:53 rc5.d drwxr-xr-x 2 root root 4096 May 18 2004 rc6.d -rwxr-xr-x 1 root root 220 Jul 10 2001 rc.local -rwxr-xr-x 1 root root 23299 Feb 24 2003 rc.sysinit
The entries in rc#.d are symbolic links. The actual scripts are in /etc/init.d. These links begin with either K or S. The S links are for startup scripts, and the K links are for shutdown scripts. The numbers following the S or K are used to order the execution of the scripts. When moving to a new runlevel, the shutdown scripts are run, followed by the startup scripts. Let's look more closely at the startup scripts in rc5.d:
# ls -al /etc/rc.d/rc5.d/S* lrwxrwxrwx 1 root root 15 May 18 2004 /etc/rc.d/rc5.d/S05kudzu -> ../init.d/kudzu lrwxrwxrwx 1 root root 18 May 18 2004 /etc/rc.d/rc5.d/S08iptables -> ../init.d/iptables lrwxrwxrwx 1 root root 17 May 18 2004 /etc/rc.d/rc5.d/S10network -> ../init.d/network lrwxrwxrwx 1 root root 16 May 18 2004 /etc/rc.d/rc5.d/S12syslog -> ../init.d/syslog lrwxrwxrwx 1 root root 17 May 18 2004 /etc/rc.d/rc5.d/S13portmap -> ../init.d/portmap lrwxrwxrwx 1 root root 17 May 18 2004 /etc/rc.d/rc5.d/S14nfslock -> ../init.d/nfslock ... (rest omitted)
The script name is the same as the link name without the leading S or K and numbers.
#ls -al /etc/init.d lrwxrwxrwx 1 root root 11 Nov 6 2003 /etc/init.d -> rc.d/init.d #ls /etc/rc.d/init.d aep1000 firstboot isdn network random squid xinetd anacron FreeWnn kdcrotate nfs rawdevices sshd ypbind apmd functions keytable nfslock rhnsd syslog yppasswdd atd gpm killall nscd saslauthd tux ypserv autofs halt kudzu ntpd sendmail vncserver ypxfrd bcm5820 httpd linuxcoe pcmcia single vsftpd canna innd lisa portmap smb webmin crond iptables named postgresql snmpd winbind cups irda netfs pxe snmptrapd xfs
For Red Hat, the initlog command is called to run the individual service startup scripts and log the output using syslogd. The /etc/initlog.conf file defines local7 as the syslog facility for the messages. Looking at this /etc/syslog.conf excerpt, we can see that the boot messages are sent to /var/log/messages and boot.log:
# Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;news.none;authpriv.none;cron.none /var/log/messages ... (lines omitted) # Save boot messages also to boot.log local7.* /var/log/boot.log
The initlog(8) and syslogd(8) man pages have further details.
For SUSE, the blogger command sends messages to /var/log/boot.msg. See the blogger(8) man page for details.
It would be cumbersome to manage all the symbolic links needed for an rc start or stop script. Let's look at crond as an example. It runs at runlevels 2 through 5, so it has a start script for each runlevel. It does not run at levels 0, 1, and 6, so it has kill scripts for these levels. That makes seven symbolic links for just crond.
#find /etc/rc.d -name *crond /etc/rc.d/init.d/crond /etc/rc.d/rc0.d/K60crond /etc/rc.d/rc1.d/K60crond /etc/rc.d/rc2.d/S90crond /etc/rc.d/rc3.d/S90crond /etc/rc.d/rc4.d/S90crond /etc/rc.d/rc5.d/S90crond /etc/rc.d/rc6.d/K60crond
Fortunately, the chkconfig command is provided to add and remove the links as needed. The chkconfig(8) man page lists all the options, but the most useful options are provided in Table 1-4.
As an example, let's manipulate crond. First we determine what the current settings are:
#chkconfig --list crond crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Next we turn off crond at runlevel 2:
#chkconfig --level 2 crond off #chkconfig --list crond crond 0:off 1:off 2:off 3:on 4:on 5:on 6:off
Now we look at the symbolic links to see whether anything changed:
#find /etc/rc.d -name *crond /etc/rc.d/init.d/crond /etc/rc.d/rc0.d/K60crond /etc/rc.d/rc1.d/K60crond /etc/rc.d/rc2.d/K60crond /etc/rc.d/rc3.d/S90crond /etc/rc.d/rc4.d/S90crond /etc/rc.d/rc5.d/S90crond /etc/rc.d/rc6.d/K60crond
We can see that rc2.d has the K60crond stop script instead of the S90crond start script. We can return the crond configuration to the default values:
#chkconfig crond reset #chkconfig --list crond crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
The chkconfig command uses the following line in /etc/rc.d/init.d/crond to determine the default values and link names:
# chkconfig: 2345 90 60
Linux distributions use different methods to encode the default runlevel values in startup and shutdown scripts. The previous example was from a Red Hat 9.0 crond script. A SUSE 9.0 cron script has the following:
### BEGIN INIT INFO # Provides: cron # Required-Start: $remote_fs $syslog $time # X-UnitedLinux-Should-Start: sendmail postfix # Required-Stop: $remote_fs $syslog # Default-Start: 2 3 5 # Default-Stop: 0 1 6 # Description: Cron job service ### END INIT INFO
We must mention one more directory. The /etc/sysconfig directory contains configuration files for the rc scripts. Here is a typical listing:
ls /etc/sysconfig apmd grub mouse redhat-config- securitylevel apm-scripts harddisks named redhat-config-users authconfig hwconf netdump redhat-logviewer autofs i18n netdump_id_dsa rhn clock init netdump_id_dsa.pub samba console installinfo network sendmail desktop ip6tables-config networking squid devlabel iptables-config network-scripts syslog dhcpd irda ntpd tux dhcrelay irqbalance pcmcia vncservers firstboot keyboard prelink xinetd gpm kudzu rawdevices yppasswdd
These configuration files contain variables for the rc scripts. The configuration files are sourced by the rc scripts from which they get their name. As we can see, /etc/sysconfig includes some directories, such as the network directory. The /etc/sysconfig files are small. The sendmail script, for example, consists of only two lines:
# cat sendmail DAEMON=yes QUEUE=1h
Red Hat provides the ntsysv command to manipulate startup/shutdown script configuration. It is not as powerful as chkconfig, but it is easier to use.
Linux gives us a way to control what scripts run at boot time. Sure, a system administrator can use chkconfig to configure which scripts run and which don't, but wouldn't it be nice to pick and choose during boot up? The Linux confirm mode provides this feature.
You can use the rc script to prompt whether each script should be run during startup. This feature is useful when one or more scripts need to be skipped for whatever reason.
To run rc in confirm mode, add the keyword confirm to the kernel command line, just as you would add the keyword single to boot to single user mode. This can be done from the bootloader, as Figure 1-15 shows.
Figure 1-15. Booting confirm mode with LILO
Figure 1-16 shows how the Ethernet rc script hangs if the LAN card is configured to get an IP address through DHCP but is not connected to the network. The user must sit and wait for the timeout from DHCP.
Figure 1-16. Boot hanging at eth1 configuration
If Linux is started in confirm mode, the script can be skipped. Press y to run the script, or press n to skip it. Press c to run the script and all the following scripts. This is a nice way to skip a script and not have to change the configuration. Figure 1-17 shows how this looks.
Figure 1-17. Skipping eth1 configuration in confirm mode
Startup Problems in rc Scripts
Problems with startup scripts can be difficult to troubleshoot because many files are involved. The problem could be with the rc script, one of the scripts rc is trying to run, or any command or script rc relies on. Figure 1-18 demonstrates a typical example, in which the Linux kernel boots and then displays some errors.
Figure 1-18. Boot error from rc
It is a good idea to stop and write down all the messages and errors before they scroll off the screen. We have a limited opportunity to fix problems with Linux in this state because / is mounted as read-only. However, we can troubleshoot and hopefully find the problem.
The following command not found errors from Figure 1-18 stand out:
/etc/rc.d/rc.sysinit: line 81: action: command not found grep: /proc/mounts: No such file or directory /etc/rc.d/rc.sysinit: line 93: action: command not found /etc/rc.d/rc.sysinit: line 140: action: command not found /etc/rc.d/rc.sysinit: line 169: action: command not found
We edit the file with vi just to check the line numbers:
*** An error occurred during the file system check. *** Dropping you to a shell; the system will reboot *** when you leave the shell. Give root password for maintenance (or type Control-D to continue): (Repair filesystem) 1 # vi /etc/rc.d/rc.sysinit
Enter :se nu to turn on line numbers. Enter :81G to go to line 81. In Figure 1-19, you can see that the rc.sysinit script is calling the subroutine named action. The other line numbers call action as well.
Figure 1-19. Editing /etc/rc.d/rc.sysinit
We could look around to see where the action subroutine is located, but it might have been removed. Let's verify that the startup script files are all in place and the correct size. We can use rpm, but we need to determine what package to verify. We use the rpm command to learn what delivered the rc script:
(Repair filesystem) 2 # rpm -q -f /etc/rc.d/rc initscripts-7.14-1
Now we verify that the initscripts-7.14-1 rpm is intact:
(Repair filesystem) 3 # rpm -V initscripts-7.14-1 S.5....T c /etc/inittab S.5....T c /etc/rc.d/init.d/functions
The output looks pretty cryptic, but the rpm(8) man page gives a good explanation:
The format of the output is a string of 8 characters, a possible "c" denoting a configuration file, and then the file name. Each of the 8 characters denotes the result of a comparison of attribute(s) of the file to the value of those attribute(s) recorded in the database. A single "." (period) means the test passed, while a single "?" indicates the test could not be performed (e.g. file permissions prevent reading). Otherwise, the (mnemonically emboldened) character denotes failure of the corresponding --verify test: S file Size differs M Mode differs (includes permissions and file type) 5 MD5 sum differs D Device major/minor number mis-match L readLink(2) path mis-match U User ownership differs G Group ownership differs T mTime differs
So, the output means the size and timestamp of the inittab and functions files have changed since the files were delivered. This can be expected for inittab, but why for functions? Let's look:
(Repair filesystem) 5 # ls -al /etc/inittab -rw-r--r-- 1 root root 1807 Dec 17 15:52 /etc/inittab (Repair filesystem) 6 # ls -al /etc/rc.d/init.d/functions -rwxr-xr-x 1 root root 0 Dec 21 14:39 /etc/rc.d/init.d/functions
It looks like function was zeroed out. We need to restore this file. Because / is mounted as read-only, the box should be booted from a different source. We can use a rescue CD, rescue disk, or second boot disk. When Linux is up, we just mount / to a temporary mount point and restore the functions file. Rescue disks are explained in the next section.