What to Look for When Something Goes Wrong

There are several things to look for when something goes wrong in MySQL Cluster. The log files on each storage node are stored in DataDir, which is typically in /var/lib/mysql-cluster. The typical directory listing for such a folder on a storage node looks like this:

[user@storage] ls
ndb_3_error.log ndb_3_out.log ndb_3_trace.log.1
ndb_3_fs ndb_3.pid ndb_3_trace.log.next

For a management node the directory listing looks like this:

[user@mgm] ls
config.ini ndb_1_cluster.log ndb_1_out.log ndb_1.pid

These files all take the form ndb_x_filename where x is the ID of the node (as either specified in config.ini or assigned when the node connects to the management daemon automatically).

The file that ends in out.log contains a log of the actions. Where there is .log.1 and so on, this means that the node crashed and recovered, and rather than replace the older files, the node simply creates new ones and adds a number to the end. This makes it fairly easy for you to go in and discover what caused the node to crash because the old log files are frozen in time at the time of the crash.

For example, this is what the log looks like after you start a storage node:

[user@storage] tail ndb_3_out.log
2005-08-31 11:43:53 [NDB] INFO -- Angel pid: 2508 ndb pid: 2509
2005-08-31 11:43:53 [NDB] INFO -- NDB Cluster -- DB node 3 
2005-08-31 11:43:53 [NDB] INFO -- Version 5.0.11 (beta) --
2005-08-31 11:43:53 [NDB] INFO -- Configuration fetched at 10.0.0.1 port 1186

After a complete start of the cluster, the management server log contains far more information about the status of each node, and it records many other pieces of information, such as nodes missing heartbeats. You should monitor this proactively because, for example, regular missed heartbeats indicates a problem with load or networking in the cluster. If you spot such a problem early, you won't face a cluster shutdown, but if you fail to notice it until a shutdown, you have to fix the cause while your cluster is down, which is not a good idea.