17.5 The Boot Sequence

17.5 The Boot Sequence

To see how a cluster is formed (or for that matter how a node becomes a member), let's see what happens when you boot a node in a cluster.

When the "boot" command is issued from the console of a node that's part of a cluster configuration that node will boot off of its member boot disk. Contained on the boot disk at logical block number (LBN) zero is the boot block.

 CPU 0 booting (boot dkb100.1.0.3.2 -flags A) block 0 of dkb100.1.0.3.2 is a valid boot block 

17.5.1 From the Boot Block to the Kernel

The boot block code points to the primary bootstrap code at LBN 64.

 reading 19 blocks from dkb100.1.0.3.2 bootstrap code read in Building FRU table base = 200000, image_start = 0, image_bytes = 2600 initializing HWRPB at 2000 initializing page table at 1f2000 initializing machine state setting affinity to the primary CPU jumping to bootstrap code 

The primary bootstrap code loads the osf_boot program, located on the boot_partition(partition "a") of the boot disk. It is the osf_boot program that handles prompting for the kernel and subsystem attributes during an interactive boot (i.e., boot -fl i). The osf_boot program is also responsible for loading the kernel.

 UNIX boot - Wednesday August 01, 2001 Loading vmunix ... Loading at 0xfffffc0000230000 Sizes: text = 8624320 data = 1778320 bss = 3908272 Starting at 0xfffffc0000242360 

Once the kernel is loaded and starts to execute, many things happen prior to single-user mode including cluster formation! So, please turn your hymnals to "The Ballad of V. M. Unix" and follow along. Please feel free to shout out or sing along if the muse strikes you.

17.5.2 Loading the Kernel to Subsystem Configuration

While we don't intend to walk through every step in the kernel boot sequence, we want to point out a few steps to help in understanding how a cluster is formed.

One of the kernel's earliest tasks is to read the sysconfigtab file. This is the reason why the sysconfigtab file is located in the etc directory on the boot_partition. Once the sysconfigtab file is read, the kernel loads the attributes into the configuration management database (cfgdb), loads the Configuration Manager (cm) subsystem, and sets up the subsystem callback mechanism. After the callback mechanism is loaded, the kernel configures all the static kernel subsystems.

If you are curious about how the kernel knows which subsystems to configure, take a look at the conf.c file in any member's /sys/ HOSTNAME directory.

 # cd /usr/sys/MOLARI ; grep -p static_subsys_list conf.c \ > | grep CFG_STATE_LOADED | cut -d, -f2 | cut -d\" –f2 | tee /tmp/conf_c ... rm rmvm rdg clubase kevm_clu drd cnx kch dlm mcs clua ics_hl ics_ll_tcp ics_ll_mct cfs cms token icsnet clsm ... 

The conf.c file is created by the config(8) program which is run by the doconfig(8) script.

Of course, once a system is up and running it is usually much more straightforward to query the configuration manager using the sysconfig program to see which subsystems are statically loaded.

 # sysconfig -m | grep static | cut -d: -f1 | tee /tmp/sysconfig 

Using the diff(1) command on the two files we created in the previous two examples, we can see that there are no differences.

 # diff /tmp/conf_c /tmp/sysconfig <no output> 

There are approximately 97 static subsystems loaded into the kernel. Of course this number can vary depending on your configuration and the options chosen.

 # wc -l /tmp/sysconfig /tmp/conf_c          97 /tmp/sysconfig          97 /tmp/conf_c          194 total 

17.5.3 Interactive Boot Subsystem Attribute Overrides

Getting back to our boot discussion…

 Loading vmunix symbol table ... [2188944 bytes] Alpha boot: available memory from 0x10c8000 to 0xff3e000 

Between "Loading vmunix symbol table ... [2188944 bytes]" and the very next line ("Alpha boot: available memory from 0x10c8000 to 0xff3e000"), the sysconfigtab file is read, the static subsystems are configured, and any subsystem attribute values that were entered on the command-line via an interactive boot are retrieved and used to override the values read from sysconfigtab.

By booting interactively, attribute values in sysconfigtab can be superseded. Remember this little piece of information because we'll take advantage of this feature in section 17.8.5.

Caution

Unlike a standalone system, a cluster member cannot be booted without a sysconfigtab file (i.e., boot -fl c) as there are some very important cluster base (clubase) subsystem attributes that are needed in order for the system to form or join a cluster.

Here is an example of what happens when you boot without a sysconfigtab file.

 >>> boot –fl c ... TruCluster Server V5.1A (Rev. 1312); 08/14/01 10:35 panic (cpu 0): clubase_cfg: no cluster name in /etc/sysconfigtab trap: invalid memory ifetch access from kernel mode     faulting virtual address: 0x0000000000000000     pc of faulting instruction: 0x0000000000000000     ra contents at time of fault: 0xfffffc0000799414     sp contents at time of fault: 0xfffffe0413427920 DUMP: A dump found in memory will tie up 2605056 bytes until released. ... 

17.5.4 Subsystem Callback Dispatch Points

When a static subsystem is configured, it may register callback routines to complete its configuration at a later time in the boot process. Think of this as a wakeup call for the subsystem. In fact, since a subsystem can register more than one callback and the callbacks can be registered for different points along the boot process, think of it as both a wakeup call and a snooze button. To illustrate, here's how one author's wife gets up for work in the morning:

She sets the alarm for 5:30 AM. When the alarm goes off, she hits the snooze button deciding to cuddle for 15 minutes (callback number 1). At 5:45 AM, callback number 2 is reached; she rolls over and hits the snooze button again for a 15-minute pre-shower-power-nap. Finally, at 6:00 AM, the last callback is triggered as the alarm sounds to indicate it's time to get on with the day.

Callback routines are triggered when particular dispatch points are reached. Additionally, some subsystems must be configured before or after other subsystems at the same dispatch point, so callback routines can also be registered at a specified priority to insure that callbacks are called in the proper order. When a dispatch point is reached, all subsystem callbacks, registered for that dispatch point, will be executed in order of priority. Callback dispatch points are defined in the sysconfig.h file located in the /sys/include/sys directory.

We will not discuss what happens at each and every dispatch point but instead, in Table 17-5 we will show you the boot messages you will likely see (if any) as the various cluster and cluster-related subsystems are configured. These messages will appear on the system's console as well as the messages file in the /var/adm directory.

For more information regarding a particular subsystem, you can consult the sys_attrs_ subsystem reference page.