| One of the most common debugging techniques is the output of certain messages at strategic program positions meaningful screen outputs are simply inserted (ideally before and after) at potential sources of error. This helps us to track the kernel's behavior and how it progresses. Of course, we could also output variable values and similar useful things. Though this rather simple but useful fault-tracing variant often helps, there can be cases where an error in the operating-system kernel causes the entire computer to crash, leaving no way to store or read log information. These cases often occur when function pointers are wrongly initialized or are due to accesses beyond array boundaries. Some of these errors can be caught with the well-known "kernel Ooops," but some lead inevitably to a crash. The following sections introduce several helpful methods to create outputs from the kernel and make them visible to the programmer or user. B.1.1 Using printk() to Create Log Outputs The printf() function is normally used in conventional writing of C programs and the Linux kernel is actually not different to output messages at a text console. printf() is a function of the standard input/output library (<stdio.h>), which is not available in the Linux kernel. For this reason, the Linux developers simply simulated printf() and integrated it into the kernel as the printk() function. Other functions borrowed from the standard libraries help the handling of character strings (lib/string.c):
printk() offers almost the same functionality as printf() and has a similar syntax. A special property of printk() is the classification of messages to be output, by different debugging levels. The syslogd and klogd daemons can be used to store and output kernel messages or send them to specific addresses. Altogether, there are eight debugging levels, from KERN_DEBUG, which is the lowest level (normal debugging messages) to KERN_EMERG the highest level (system unusable). These debugging levels are defined in <linux/kernel.h>. Depending on the DEFAULT_CONSOLE_LOGLEVEL variable, messages are output on the current console. The administrator can use the command syslogd -c to modify the value appropriately. printk() itself uses the printf() function to generate the output string. This is the reason why the syntax of the two functions and parametrizing of the variables to be output are identical. sprintf() will be introduced in Section B.2. B.1.2 The syslogd Daemon While the operating system is running, there are often situations where programs have to log error messages or specific information. An application in text mode outputs these messages simply at the console (stdout or stderr). A popup window is normally created for window-based applications (X11). The operating system kernel and processes working directly for it, such as daemons or child processes of the init process, have no direct allocation to a console. Now, where should error and log messages be output? The standard output (stdout) of these processes uses the /dev/console console. In the X-Windows system, this is the xconsole window. This approach can cause problems in a multiuser environment. On the one hand, messages can be read by anyone; on the other hand, the person in charge (normally the administrator) might be looking at something else and not pay attention to the message window. Another problem is that these screen outputs cannot easily be stored, which means that they are very volatile. To solve these problems, we use the syslogd daemon. Daemons or other system programs outside the kernel have the possibility to generate messages for syslogd, which receives these messages over the /dev/log device and processes them. Within the kernel, log messages are simply created by the printk() function. If you additionally have started the klogd daemon, then the messages are also forwarded to syslogd. Section B.1.1 described how printk() works. The messages mentioned above are accepted automatically after syslogd has started. The file /etc/syslog.conf defines how these messages should be further handled (i.e., in what files and by what criteria they should be stored). The syslogd Configuration File /etc/syslog.conf This file defines the rules for the processing of log messages by the syslogd daemon. If this file needs to be modified, then we first have to terminate the daemon and restart it afterwards. syslog messages are distinguished by two criteria: by their creator and by their priority. Priorities for syslog messages: There are eight different priorities. These are identical to the priorities defined for printk in linux/kernel.h (i.e., emerg corresponds to KERN_EMERG). The following priorities are sorted in descending order by urgency: emerg: System unusable crash. alert: Serious error that has to be dealt with immediately. crit: Critical situation. err: Error notification. warning: Warning. notice: Special-situation notification. info: Regular messages within routine operation. debug: Fault-tracing messages.
Origin of syslog messages: Messages for syslogd can be generated by the following areas: kern: Messages from the kernel (printk()). auth: Messages from the area of security, authentication (login, ...). mail: Messages from the mail system. news: Messages from the news system. lpr: Messages from the print daemon. cron: Messages from the cron daemon. syslog: Messages from the syslog daemon. daemon: Messages from any daemon. user: Messages from application programs. local0-7: Message areas that can be freely assigned.
Processing messages: There are various ways to store and to deliver messages, which can be configured depending on the priority and origin of messages. The file /etc/syslog.conf always defines actions to be done in the following syntax:
origin.priority{;origin.priority}* action*
Each line outputs a processing rule, consisting of a sequence of message types (origin.priority), separated by a semicolon, and an action. Both the origin and the priority of a message type can be replaced by a wildcard (*). The following actions are possible: The message can be written to a file. To write a message to a file, we state the file name with the absolute path (starts always with a "/"). The message can be sent to one or several users. The user names (login names) have to be stated in a list (separated by commas). The message is displayed only for those users who are logged in at the time of the event. You can use a wildcard (*) to send a message to all users currently logged in. A message can be sent to the syslog daemon of another computer, which is specified with a leading "at" character (e.g., @tux.icsi.Berkeley.edu).
Example for a syslogd configuration file comments begin with #: # /etc/syslog.conf - Configuration file for syslogd(8) # # print most on tty10 kern.warn;*.err;authpriv.none /dev/tty10 *.emerg * # # all email messages in one file # mail.* -/var/log/mail # # all news messages news.crit -/var/log/news/news.crit news.err -/var/log/news/news.err news.notice -/var/log/news/news.notice # enable this, if you want to keep all news messages # in one file #news.* -/var/log/news.all # # Warnings in one file *.warn /var/log/warn # # save the rest in one file # *.*;mail.none;news.none /var/log/messages # # All messages with priority emerg and higher will be sent to the # users, who are logged in. They will also be sent to syslogd on # host tux.icsi.Berkeley.edu. *.emerg * *.emerg @ tux.icsi.Berkeley.edu # All messages with priority alert and higher will be sent to the # administrator, if he/she is logged in. *.alert root
B.1.3 Using console_print() for direct outputs There can always be situations where the kernel gets stuck and no more output is possible. This is usually the case when printk() has written a message to the buffer, but klogd hasn't been invoked by the scheduler yet. In such cases, it is recommended to use the console_print() function for outputs. It outputs a specific string immediately at the current console. console_print() | kernel/printk.c |
In contrast to printk(), console_print() merely outputs a string, but does not convert variables into characters. In such cases, the output string should have been created previously by sprintf(). Naturally, we first have to reserve a byte array. The use of console_print() is extremely helpful, but only provided that it is absolutely required to always output to the current console. This is conceivable only for temporary debugging or really important exceptions. Otherwise, unexpected outputs at the console can be disturbing and confusing for the regular user. | 