9.6 Problem determination

< Day Day Up >

When there are problems on the Domino server on Linux, the Domino Administrator and the Linux Administrator will need to work closely to determine the root cause of the problem and resolve it.

9.6.1 Domino server problems

If the Domino server has crashed, there may have been a core dump generated. You will need to plan with the Domino Administrator in which directory to place the core dumps. By default, it will be generated in the Domino data directory, but the Domino Administrator can change it to a different location by a parameter in the notes.ini file (this is described in Chapter 8, "Domino administration" on page 169 of this book).

You will need to allocate sufficient space for the dumps. It is also advisable to compress core dumps using gzip to save disk space. You should also clean up this filesystem on a regular basis.

At times, the Domino administrator will need to contact the Linux administrator to check on the status of resources. For example, in 8.6, "Problem determination" on page 204, the Domino Administrator has been alerted to a mail backlog condition and has checked and ruled out any Domino server issues (server availability, corrupt file, and so on). So the Domino administrator would contact the Linux administrator.

At this point, one of the items that the Linux administrator can check is the DASD I/O. This can be done by using the iostat command. The iostat command reports the CPU utilization and the device utilization.

We ran iostat with the -x option to display extended statistics, as shown in Figure 9-8 on page 243. The first section reports CPU utilization on the user and system level, as well as percentage of time that the CPU or CPUs were idle. In the device section, the following stats are important to check:

await	Average time (in milliseconds) for I/O requests issued to the device
svctm	Average service time (in milliseconds) for I/O requests issued to the device
%util	Percentage of CPU time during which I/O requests were issued to the device

  domserva@linuxa:/domserva/notesdata > iostat -x 40 2  Linux 2.4.19-4suse-SMP (linuxa)          08/28/03 avg-cpu:   %user    %nice    %sys    %idle             1.97     0.00    2.23    95.80    Device:   rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s   rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm    %util    /dev/dasda  0.90   2.38  0.48  0.57   11.00   23.55    5.50    11.78    33.19     0.17  165.58  46.85    0.49    /dev/dasda1                0.90   2.38  0.48  0.57   11.00   23.55    5.50    11.78    33.19     0.17  165.59  46.85    0.49    /dev/dasdb  1.77   0.76  0.45  0.23   17.78    7.91    8.89     3.96    37.97     0.09  136.48  30.11    0.20    /dev/dasdb1                1.77   0.76  0.45  0.23   17.78    7.91    8.89     3.96    37.97     0.09  136.50  30.11    0.20    /dev/dasdc  1.53   0.00  0.40  0.01   15.41    0.09    7.71     0.04    37.85     0.02   45.02  34.91    0.14    /dev/dasdc1                1.53   0.00  0.40  0.01   15.41    0.09    7.71     0.04    37.86     0.02   45.02  34.91    0.14    /dev/dasdd  1.50   0.00  0.41  0.01   15.26    0.06    7.63     0.03    37.00     0.02   45.43  35.42    0.15    /dev/dasdd1                1.50   0.00  0.41  0.01   15.26    0.06    7.63     0.03    37.00     0.02   45.43  35.42    0.15  /dev/dasde 1.54  0.00  0.41  0.01   15.63    0.05    7.82     0.02    37.42     0.01   22.24  17.98   0.08

Figure 9-8: iostat -x command

Domino and Linux administrators working on a problem

The Domino Administrator has the ability to collect platform statistics with the Domino command Show Stat Platform, and the Linux Administrator can use OS commands to obtain information from the system using cat /proc/meminfo and iostat.

Example 9-15 checks the CPU using iostat -c command and system output from platform statistics.

Example 9-15: Checking CPU

Linux

 domserva@linuxa:/domserva/notesdata > iostat -c Linux 2.4.19-4suse-SMP (linuxa)     09/02/03 avg-cpu:  %user   %nice    %sys   %idle            2.32    0.03    2.73   94.92Domino:

Domino

 Platform.System.PctCombinedCpuUtil = 0   Platform.System.PctCombinedCpuUtil.Avg = 11.26   Platform.System.PctCombinedCpuUtil.Peak = 97   Platform.System.PctTotalPrivilegedCpuUtil = 0   Platform.System.PctTotalPrivilegedCpuUtil.Avg = 6.1   Platform.System.PctTotalPrivilegedCpuUtil.Peak = 64   Platform.System.PctTotalUserCpuUtil = 0   Platform.System.PctTotalUserCpuUtil.Avg = 5.02   Platform.System.PctTotalUserCpuUtil.Peak = 52   Platform.Time.LastSample = 09/02/2003 20:29:40 EDT   Platform.Time.SampleRateInMins = 1

Example 9-16 checks memory using cat /proc/meminfo and memory output from platform statistics. (Notice that what Domino reports as paging statistics, Linux reports as swapping statistics, but it is the same information.)

Example 9-16: Checking memory

Linux

domserva@linuxa:/domserva/notesdata > cat /proc/meminfo

         total:    used:    free:  shared: buffers:  cached: Mem:  195035136 192409600  2625536        0 32403456 110645248 Swap: 507617280 139149312 368467968 MemTotal:       190464 kB MemFree:          2564 kB MemShared:           0 kB Buffers:         31644 kB Cached:          96028 kB SwapCached:      12024 kB Active:          26488 kB Inactive:       127504 kB HighTotal:           0 kB HighFree:            0 kB LowTotal:       190464 kB LowFree:          2564 kB SwapTotal:      495720 kB SwapFree:       359832 kB

Domino

 Platform.Memory.PagesPerSec = 0   Platform.Memory.RAM.AvailMBytes = 4   Platform.Memory.RAM.AvailMBytes.Avg = 1   Platform.Memory.RAM.AvailMBytes.Min = 1   Platform.Memory.RAM.AvailMBytes.Peak = 11   Platform.Memory.RAM.PctUtil = 97   Platform.Memory.RAM.TotalMBytes = 186   Platform.PagingFile.Free.SizeMBytes = 351   Platform.PagingFile.Total.PctUtil = 27   Platform.PagingFile.Total.PctUtil.Avg = 19   Platform.PagingFile.Total.PctUtil.Peak = 34   Platform.PagingFile.Total.SizeMBytes = 484

Example 9-17 on page 245 checks iostats using the iostat -d command and logical output from platform statistics.

Example 9-17: Checking iostats

Linux

 domserva@linuxa:/domserva/notesdata > iostat -d -x dasdl Linux 2.4.19-4suse-SMP (linuxa)     09/02/03 Device:    rrqm/s wrqm/s   r/s   w/s   rsec/s wsec/s     rkB/s     wkB/s avgrq-sz avgqu-sz   await  svctm  %util /dev/dasdl   0.98   0.01  0.24  0.01     9.78   0.16      4.89      0.08    39.84     0.01   29.71  21.37   0.05

Domino

 Platform.LogicalDisk.10.AssignedName = dasdl   Platform.LogicalDisk.10.AvgQueueLen = 0   Platform.LogicalDisk.10.AvgQueueLen.Avg = 0.05   Platform.LogicalDisk.10.AvgQueueLen.Peak = 0.91   Platform.LogicalDisk.10.PctUtil = 0   Platform.LogicalDisk.10.PctUtil.Avg = 0.13   Platform.LogicalDisk.10.PctUtil.Peak = 7.55   Platform.LogicalDisk.10.ServiceTime = 0   Platform.LogicalDisk.10.ServiceTime.Avg = 11.18   Platform.LogicalDisk.10.ServiceTime.Peak = 100

Using the two output commands will show both Domino and Linux Administrators where bottlenecks may be occurring and what means are needed to correct the situation.

< Day Day Up >