| This problem is not as serious as the problem in Section 14.1. However, it demonstrates other uses of the Install and Recovery media. This problem utilizes one of the automated tasks on the Install and Recovery media; in this case, it will download a minimal kernel onto our system. Why are we doing this? The reason is that we do not have a valid bootable kernel. We previously manually modified some kernel parameters. The syntax we used was valid, but our logic was somewhat amiss (we set nproc = 5). This produced a kernel that, when booted , would cause a system to PANIC every time. The other problem was that we neglected to keep a backup of our original kernel. We will boot the system to the Recovery Shell as before. I assume that you have read through Section 14.1 and managed to navigate your way to the Recovery Shell screen you see below. I will continue my discussions from that point.
HP-UX NETWORK SYSTEM RECOVERY MAIN MENU s. Search for a file b. Reboot l. Load a file r. Recover an unbootable HP-UX system x. Exit to shell c. Instructions on chrooting to an LVM /(root). This menu is for listing and loading the tools contained on the core media. Once a tool is loaded, it may be run from the shell. Some tools require other files to be present in order to successfully execute. Select one of the above:
Again, I don't have a bootable system, so the option I will choose is r. Recover an unbootable HP-UX system .
Select one of the above: r HP-UX Recovery MENU Select one of the following: a. Rebuild the bootlif (ISL, HPUX, and the AUTO file) and install all files  required to boot and recover HP-UX on the root file system. b. Do not rebuild the bootlif, but install files required to boot and recover HP-UX on the root file system. c. Rebuild only the bootlif. d. Replace only the kernel on the root file system. m. Return to 'HP-UX Recovery Media Main Menu'. x. Exit to the shell. Use this menu to select the level of recovery desired. Selection:
As you can see, there is an option here simply to replace the kernel. As far I know, the only problem I have is that when I boot the system it keeps giving a PANIC error. You could try booting in single-user mode and even maintenance mode, but if we think about it, maintenance mode is for corrupt boot/disk configuration problems, and that's not going to help. Single- user mode might help because it starts only a handful of processes. In this case and when the kernel is really sick , even a single-user mode boot will not help. The only option when we don't have a backup kernel is to try to recover a minimal kernel and effect the necessary changes in order to put a real kernel in place. I am going to choose option d. Replace only the kernel on the root file system to perform that very task:
Selection: d DEVICE FILE VERIFICATION MENU This menu is used to specify the path of the root file system. When the information is correct, select 'a'. INFORMATION to verify: Device file used for '/'(ROOT) is c1t15d0s1lvm The path to disk is 0/0/1/1.15.0 Select one of the following: a. The above information is correct. b. WRONG!! The device file used for '/'(ROOT) is incorrect. m. Return to the 'HP-UX Recovery MENU.' x. Exit to the shell. NOTE: If '/' is an LVM, use an 's1lvm' suffix (e.g.,c0t1d0s1lvm). Selection:
As before, we need to confirm the location of the boot volume. This looks okay to me:
Selection: a FILE SYSTEM CHECK MENU The file system check '/sbin/fs/hfs/fsck -y /dev/rdsk/c1t15d0s1lvm' will now be run. Select one of the following: a. Run fsck -y . b. Prompt for the fsck run string on c1t15d0s1lvm. m. Return to the 'HP-UX Recovery MENU.' Selection:
Here we can see that the Recovery Shell is ensuring that the boot volume is consistent in itself by wanting to run fsck on the filesystem. I think this is a good idea, so I choose option a :
Selection: ** /dev/rdsk/c1t15d0s1lvm ** Last Mounted on /stand ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 63 files, 0 icont, 53713 used, 57924 free (124 frags, 7225 blocks) Mounting c1t15d0s1lvm to the /ROOT directory... /sbin/fs/vxfs/fsck -y /dev/rdsk/c1t15d0s2lvm file system is clean - log replay is not required /sbin/fs/vxfs/mount /dev/dsk/c1t15d0s2lvm /ROOT /sbin/fs/hfs/mount /dev/dsk/c1t15d0s1lvm /ROOT/stand Filesystem kbytes used avail %cap iused ifree iused Mounted on /ROOT 1335296 1054579 263202 80% 7081 70179 9% ? Filesystem kbytes used avail %cap iused ifree iused Mounted on /ROOT/stand 111637 53713 46760 53% 65 17983 0% ? Should the existing kernel be 'left', 'overwritten', or 'moved'?[overwritten]
As we can see, the Recovery Shell has detected my non-bootable kernel. It is up to you what you do with it. If you have enough room in the /stand filesystem, you might want to keep it for future reference, i.e., why did it fail? I will choose to keep the kernel and just move it to a new location:
Should the existing kernel be 'left', 'overwritten', or 'moved'?[overwritten] m -- '/stand/vmunix' has been renamed '/stand/vmunixBK' -- downloading WINSTALL to /stand/vmunix RECOVERY COMPLETION MENU Use this menu after the recovery process has installed all Requested files on your system. Select one of the following: a. REBOOT the customer's system and continue with recovery. b. Return to the HP-UX Recovery Media Main Menu. Selection:
We are now in a position to reboot this system. I don't want the system to boot into multi-user mode because the kernel that has just been downloaded is a minimal kernel and does not support all the functionality of a multi-user operating system. I will interact with the boot process to boot from the primary boot path, but then I will boot into single-user mode:
Selection: a NOTE: System rebooting... NOTE: run_cmd: Process: 65 (/sbin/sh): killed by signal: 9. sync'ing disks (0 buffers to flush): 0 buffers not flushed 0 buffers still dirty Closing open logical volumes... Done
Once it's rebooted, I can interact with the boot process to get to the ISL prompt:
Processor is booting from first available device. To discontinue, press any key within 10 seconds. Boot terminated. ---- Main Menu -------------------------------------------------------------- Command Description ------- ----------- BOot [PRIALT<path>] Boot from specified path PAth [PRIALT] [<path>] Display or modify a path SEArch [DIsplayIPL] [<path>] Search for boot devices COnfiguration menu Displays or sets boot values INformation menu Displays hardware information SERvice menu Displays service commands DIsplay Redisplay the current menu HElp [<menu><command>] Display help for menu or command RESET Restart the system ---- Main Menu: Enter command or menu > bo pri Interact with IPL (Y, N, or Cancel)?> y Booting... Boot IO Dependent Code (IODC) revision 1 HARD Booted. ISL Revision A.00.43 Apr 12, 2000 ISL> hpux ll Ls : disk(0/0/1/1.15.0.0.0.0.0;0)/. drwxrwxrwx 10 2 2 1024 ./ drwxrwxrwx 10 2 2 1024 ../ drwxr-xr-x 2 0 0 8192 lost+found/ -rw-r--r-- 1 0 3 3440 ioconfig -rw-r--r-- 1 0 3 1127 system -rw-r--r-- 1 0 3 20 bootconf drwxr-xr-x 2 0 3 1024 krs/ drwxr-xr-x 2 0 3 1024 system.d/ drwxr-xr-x 4 0 3 2048 build/ -r--r--r-- 1 0 3 82 kernrel -rw------- 1 0 0 12 rootconf drwxr-xr-x 2 0 0 1024 krs_tmp/ drwxr-xr-x 2 0 0 1024 krs_lkg/ -rw------- 1 0 0 0 .kminstall_lock -rw-r--r-- 1 0 3 12342712 vmunix -r--r--r-- 1 0 3 1104 system.prev drwxr-xr-x 5 0 3 1024 dlkm/ -rwxr-xr-x 1 0 3 25927256 vmunixBK* ISL>
As you can see, the old kernel ( vmunixBK ) is much larger than the kernel downloaded by the Recovery Shell ( vmunix ). I will now boot to single-user mode to repair the real kernel.
ISL> hpux -is Boot : disk(0/0/1/1.15.0.0.0.0.0;0)/stand/vmunix 9777152 + 1687552 + 2602664 start 0x2012e8 alloc_pdc_pages: Relocating PDC from 0xf0f0000000 to 0x3fb01000. ... . INITSH: /sbin/init.d/vxvm-startup2: not found INIT: Overriding default level with level 's' INIT: SINGLE-USER MODE INIT: Running /sbin/sh # cd /stand # ls -al total 125578 drwxrwxrwx 10 bin bin 1024 Sep 24 02:17 . drwxr-xr-x 24 root root 1024 Sep 24 02:01 .. -rw------- 1 root root 0 Mar 10 2003 .kminstall_lock -rw-r--r-- 1 root sys 20 Sep 23 07:44 bootconf drwxr-xr-x 4 root sys 2048 Sep 24 01:22 build drwxr-xr-x 5 root sys 1024 Sep 24 01:22 dlkm -rw-r--r-- 1 root sys 3440 Sep 24 02:28 ioconfig -r--r--r-- 1 root sys 82 Feb 18 2003 kernrel drwxr-xr-x 2 root sys 1024 Sep 24 01:22 krs drwxr-xr-x 2 root root 1024 Sep 24 02:28 krs_lkg drwxr-xr-x 2 root root 1024 Sep 24 01:22 krs_tmp drwxr-xr-x 2 root root 8192 Feb 18 2003 lost+found -rw------- 1 root root 12 Sep 24 02:28 rootconf -rw-r--r-- 1 root sys 1127 Sep 24 01:20 system drwxr-xr-x 2 root sys 1024 Sep 23 07:49 system.d -r--r--r-- 1 root sys 1104 Sep 24 01:20 system.prev -rw-r--r-- 1 root sys 12342712 Sep 24 02:17 vmunix -rwxr-xr-x 1 root sys 25927256 Sep 24 01:21 vmunixBK # #
I am going to extract the system file from the old kernel. I will use the command /usr/lbin/sysadm/system_prep to do this. First, I need to mount the /usr filesystem:
# # mount /usr # # /usr/lbin/sysadm/system_prep -k /stand/vmunixBK -s /stand/systemBK # grep nproc systemBK nproc 5 #
In this instance, we can see the offending kernel parameter quite clearly. We now need to rectify this problem by putting in place a real kernel with the offending kernel parameter(s) being given proper values. In this instance, I am making a quick change to return nproc to its default value:
# cp system system.prev # mv systemBK system # kmtune -r nproc # mk_kernel Generating module: krm... Compiling /stand/build/conf.c... Loading the kernel... Generating kernel symbol table... # kmupdate Kernel update request is scheduled. Default kernel /stand/vmunix will be updated by newly built kernel /stand/build/vmunix_test at next system shutdown or startup time. # cd / # shutdown ry now
Once the server is back up and running, I should make a coherent backup of the good kernel to avoid having this problem happen again. | 