8.4 GPFS logs and traces

< Day Day Up >

GPFS has the ability to generate very detailed logs and traces so problems can be quickly found in order to have GPFS up and running again as quickly as possible. In this section, we describe the logs, tracing, and some of the most common problems you may face when managing a GPFS cluster.

8.4.1 GPFS logs

The mmfsd daemon generates detailed logs about GPFS operations and errors in the /var/adm/ras directory. Within this directory, we can find many log files named mmfs.log.date, where date is the date when the GPFS daemon was started. The current log file will also have a symbolic link to it named mmfs.log.latest so you can easily find it. Example 8-15 is an example of the mmfs.log file showing the GPFS initialization process.

Example 8-15: mmfs.log file

 Tue Nov 26 11:34:27 CST 2002 runmmfs starting Removing old /var/adm/ras/mmfs.log.* files: Unloading modules from /usr/lpp/mmfs/bin mmfslinux configuration: I386 SMP 4G Loading modules from /usr/lpp/mmfs/bin mmfslinux configuration: I386 SMP 4G Warning: loading /usr/lpp/mmfs/bin/tracedev will taint the kernel: non-GPL licen se - BSD without advertisement clause Warning: loading /usr/lpp/mmfs/bin/mmfslinux will taint the kernel: non-GPL lice nse - BSD without advertisement clause Warning: loading /usr/lpp/mmfs/bin/mmfs will taint the kernel: non-GPL license -  International Business Machines Warning: loading /usr/lpp/mmfs/bin/mmfs will taint the kernel: forced load Module                  Size  Used by    Tainted: PF mmfs                  642368   0  (unused) mmfslinux             125632   0  [mmfs] tracedev                9152   0  [mmfs mmfslinux] Removing old /var/mmfs/tmp files: ./complete.map.2002.10.23.15.30.06.node002 ./loadsyms ./complete.map.2002.10.23.17.04.16.node002 ./complete.map.2002.10.23.17.47.57.node002 ./complete.map.2002.10.23.17.50.42.node002 ./complete.map.2002.10.23.18.24.27.node002 ./complete.map.2002.10.24.12.22.40.node002 Tue Nov 26 11:34:29 2002: mmfsd initializing. {Version: 1.3.0.0   Built: Oct 7 2002 14:11:57} ... Tue Nov 26 11:34:31 2002: Waiting for myri0 adapter group subscription Tue Nov 26 11:34:31 2002: Successfully subscribed to myri0 group Tue Nov 26 11:34:31 2002: Cluster type: 'LC' Tue Nov 26 11:34:31 2002: Using TCP communication protocol Tue Nov 26 11:34:31 2002: Connecting to 10.2.1.141 Tue Nov 26 11:34:31 2002: Connected to 10.2.1.141 Tue Nov 26 11:34:31 CST 2002 /var/mmfs/etc/gpfsready invoked Tue Nov 26 11:34:31 2002: mmfsd ready Tue Nov 26 11:34:31 CST 2002: mounting /dev/gpfs0 Tue Nov 26 11:34:31 2002: Command: mount /dev/gpfs0 12006 Tue Nov 26 11:34:33 CST 2002: finished mounting /dev/gpfs0

This log file will show any significant GPFS events and is extremely important when diagnosing and solving GPFS problems. If problems arise, look at the log on the nodes where the problems occurred and on the File System Manager.

To find out which node is the File System Manager, you can issue the mmlsmgr command from any node in the cluster. The output will be similar to that shown in Example 8-16.

Example 8-16: mmlsmgr command

 [root@storage001 root]# mmlsmgr file system      manager node ---------------- ------------------ gpfs0            1 (storage001-myri0) [root@storage001 root]#

8.4.2 Trace facility

The GPFS code is equipped with a large number of trace points to allow rapid problem determination of failures. In case you need the trace facility for problem determination, IBM support personnel will give you instructions on how the trace must be done, specifying trace levels and format.

You can use the mmtrace command to generate the trace for your system. To generate a trace using this script, follow these steps:

Start the trace by issuing:
```
 # /usr/lpp/mmfs/bin/mmtrace 
```
Recreate the problem.
Stop the trace after the problem-related events have already happened:
```
 # /usr/lpp/mmfs/bin/mmtrace stop 
```

If the problem occurs in a shutdown and restart of the GPFS daemons, specify -T in the mmstartup command. By default, traces are written to the /tmp/mmfs directory.

< Day Day Up >