10.2. Logging via syslogd

The syslog daemon, syslogd, is installed with the base distributions of FreeBSD and OpenBSD. syslogd accepts logs from the kernel (from /dev/klog) and from applications running on the system via the logging socket it creates (/var/run/log on FreeBSD and /dev/log on OpenBSD by default). It can also accept log messages from other systems over UDP port 514 (by default) when configured to do so. A machine running syslogd that accepts remote logs in this way is typically called a loghost. The configuration file for syslogd, /etc/syslog.conf, controls what happens to log entries once they are received by the daemon.

10.2.1. syslog.conf Configuration

Several configuration parameters control how incoming logs are handled. The most basic elements are facility and level, which make up the selector field in the syslog.conf file. These facilities describe the part of the system generating the message, and the level describes the message severity. Syslog can then perform a variety of actions on the selected log messages. When an asterisk (*) is used in the place of either facility and/or level, it's understood to mean all facilities and/or levels.

The asterisk doesn't always mean all levels in other operating systems. In some operating systems, for example, an asterisk not only matches the levels from debug to emerg, it also matches the keyword none. As a result, syslog.conf files with an asterisk in the level field on this platform will cause the daemon to ignore all logs for the associated facility. If you have a secure file distribution scheme in a heterogeneous environment, take note.

Facilities and levels may also be specified for a subset of incoming logs through program and hostname specifications. We describe this in some detail later in this chapter. The action, the last important configuration element, is covered at the end of the section.

10.2.2. Syslog Facilities

OpenBSD includes the facilities auth, authpriv, cron, daemon, ftp, kern, lpr, mail, mark, news, syslog, user, uucp, and local0 tHRough local7. FreeBSD adds the facilities console and security. Other operating systems have a slightly different set of facilities. The specific variations between operating systems are not important. What is important is to remember that these differences exist and result in two issues: building software and remote logging.

When building software on BSD systems, bear in mind that although you may expect the software to use a given facility, the OS on which the software was designed may not have had that facility. Therefore the application programmer may log, by default, to another arbitrary facility. Software built out of ports generally "does the right thing." When building a loghost as described later in this chapter, you may find yourself accepting log messages from various operating systems. As a result, facility specification cannot be used as a means of capturing remote logs when the source operating system does not support a given facility.

Caveats aside, most applications allow for a user-configurable facility so you can choose something that makes sense in your environment. Let's look closely at the facilities present in the OpenBSD and FreeBSD operating systems.

auth

For messages pertaining to authentication like those produced by login(1), su(1), getty(8), and sshd(8).

authpriv

For messages like auth that should only be read by privileged individuals. For example, failed logins are logged to both auth and authpriv, but the attempted username is only logged to authpriv, as shown:

authpriv: Mar 28 18:31:56 dreadnaught login: 1 LOGIN FAILURE ON ttyp0, nobody     auth: Mar 28 18:31:56 dreadnaught login: 1 LOGIN FAILURE ON ttyp0

console (FreeBSD only)

This facility allows syslogd to capture log messages written to /dev/console by the kernel console output driver.

cron

For messages generated by the cron(8) daemon, and for when users run crontab(1). If you write administrative scripts that are meant to run out of cron, you might want to send messages to this facility.

daemon

For messages generated by system daemons like routed(8) that are not sent to other facilities.

ftp

As the name implies, this facility is for the file transfer protocol daemon, ftpd(8). You can expect the FTP daemon included in the base operating system to log to this facility. While most FTP daemons you use automatically configure themselves to log to this facility (especially when they are installed through the ports tree), make sure you check the application's logging behavior and ensure FTP logs don't show up under the daemon facility.

kern

Kernel messages read from /dev/klog are sent to this facility. User processes are not permitted to log kernel messages any attempts to do so are quietly redirected to the user facility. Likewise, kernel messages from remote systems will end up in the user facility. FreeBSD administrators may force the daemon to accept events submitted to the kern facility by users by using the -k argument to syslogd.

local0 through local7

The manual page for syslog.conf(5) specifies that these facilities are for local use. This means that they may be defined locally by the organization in which they are used, not that they are somehow not usable by logs received from remote systems in a loghost context. You, the administrator, may define these seven facilities to provide consistent facilities per local policies. For instance, you might choose to use these levels in conjunction with EventReporter (formerly EventSLog), which sends Windows NT event logs to Unix systems' syslogd.

lpr

For messages generated by lpr(1), lpc(8), lpd(8). In other words, this facility is for messages relating to the line printing and spooling daemon.

mail

As intuitively named as many of the other facilities, this facility is for messages pertaining to mail. Expect messages to show up here from Sendmail, Postfix, and other mail-related programs like the popular Courier IMAP and POP mail server software.

mark

The mark facility is a pseudofacility and used only by syslogd. When specified, mark messages are generated. The frequency at which this occurs depends on the parameter given to the -m argument of syslogd. For logfiles that grow very slowly, it can be useful to add mark messages so that you have confidence that syslogd is still writing to logfiles. Mark messages follow the following form:

date/timestamp hostname -- MARK --

news

Messages generated by the programs responsible for providing the network news system (nntp, for example) be assigned to this facility.

security (FreeBSD only)

Security related subsystems like FreeBSD's ipfw(4) firewall will log to this facility.

syslog

Syslog-related messages are sent to this facility. Generally this includes messages pertaining to syslogd's startup and problems with its configuration.

user

When no facility is specified, messages received by syslogd are sent to this facility. As mentioned earlier, messages from user processes with the kernel facility and messages from the kernel on remote systems are automatically assigned the user facility, though FreeBSD administrators may use the -k argument to syslogd to alter this behavior.

uucp

For messages pertaining to the uucp system generated by uucp-related binaries.

Understanding Syslog facilities will get you well on the way to managing your logfiles. Before really understanding how logging works using Syslog, however, we need to examine logging levels and actions.

10.2.3. Syslog Levels

Syslog levels are a great deal simpler than facilities and consistent across operating systems. For one, there are only eight of them: debug, info, notice, warning, err, crit, alert, and emerg. There is little to contribute to the definition of these levels, so a slightly modified excerpt from the syslog(3) manpage is reproduced here:

emerg

A panic condition. This is normally broadcast to all users.

alert

A condition that should be corrected immediately, such as a corrupted system database.

crit

Critical conditions, e.g., hard device errors.

err

Errors.

warning

Warning messages.

notice

Conditions that are not error conditions, but should possibly be handled specially.

info

Informational messages.

debug

Messages that contain information normally of use only when debugging a program.

none

The none pseudo-priority causes messages of the attached level to not be logged.

Syslog levels form the only other component required for complete specification of the selector field used in the syslog.conf file. This selector by default will apply to all messages handled by syslogd, but this behavior may be modified by using a program or hostname specification.

10.2.4. Program and Hostname Matching

The syslog.conf file may be broken into configuration blocks that apply to specific programs or hostnames. As discussed earlier, when dealing with Syslog messages arriving from other hosts on the network, a local facility may not be available on all remote systems. In this case, using a program specification, you can capture logs from this host even when the message originated from a remote system with a different facility. Example 10-1 shows a typical configuration for the capture of FTP logs on a system running syslogd.

Example 10-1. Basic syslog.conf to capture FTP logs

ftp.info               action1

This will work fine as long as the FTP program on this host and any other hosts that use this host as a loghost are configured to use the ftp facility. Some non-BSD operating systems, however, lack the ftp facility and instead log FTP messages to the daemon facility. If your BSD system is accepting logs from other systems, it may be useful to consolidate FTP logs into one file although if they are not distinguishable by facility, the problem is a little trickier.

To capture the FTP logs sent to the daemon facility along with your other FTP logs, specify the program ftpd after an exclamation mark (!) on a blank line as shown in Example 10-2. This matches all log entries with the ftpd[pid]: string.

Example 10-2. Capture FTP logs when facility may not be FTP

!ftpd *.info                action1

I n FreeBSD, hostnames may be specified by using a plus sign (+). This specification is not supported in OpenBSD. Example 10-3 demonstrates how it's possible to segregate log entries on a host-by-host or host-group basis.

Example 10-3. Split logfiles based on hostname

+mailhost mail.*                action-A auth.*                action-B +webserv1,webserv2 cron.notice           action-C

10.2.5. Syslog Actions

Now that we know a bit about log specification, it's time to examine how to configure syslogd to do something with log messages. On both FreeBSD and OpenBSD platforms, syslogd supports five log actions. Four of them are the same and likely to be familiar to even the novice administrator. They are as follows:

Log to a file/path

When a pathname is specified as the log action, messages are appended to the file specified. When syslogd starts, these logfiles must already exist; syslogd does not create logfiles.

Log to the loghost

To configure syslogd to send messages to another system, specify a hostname preceded by an at (@) sign. Typically all systems other than the loghost will log to @loghost. A canonical name (CNAME) entry is added to the local name servers to point requests for the hostname loghost to the appropriate system.

Logging to another host significantly improves your ability to build a successful audit trail for a variety of reasons. We go into more detail in the Section 10.3 section, later in this chapter.

Log to specific users

By specifying user accounts, separated by commas, log messages can be sent to specific users when they are logged in. This is useful for important administrative alerts.

Log to all logged-in users

Specify an asterisk (*) and all logged-in users will receive the log message on their pseudo-terminal. This can be useful for critical system failures, disk space usage alerts, and so on.

Although the aforementioned four actions are familiar, there are a variety of non-file locations to which log messages may be sent using the pathname specification. These are as follows:

Log to the console

It's possible to cause specific messages to be directed straight to the console by specifying the pathname as /dev/console.

Log to the printer

In high-security environments, it may be necessary to maintain physical paper logs. Adept intruders are often able to clean up any traces that they were on your system by modifying the system logs and other key files. Logging to the printer ensures that a log entry, once captured, is not lost. To capture log entries line by line, it might be useful to use a dot-matrix printer instead of a laser or inkjet printer. Bear in mind that printers can run out of paper and ink, ensure you also record your logs elsewhere. Logging to the printer may be accomplished by either specifying the pathname of the printer device or piping log output to lp(1) when lpd(8) is running.

Log to virtual tty

FreeBSD and OpenBSD systems support virtual ttys accessible from the console by pressing Alt-Fn (where n corresponds to the device /dev/ttyv(n-1)). It's possible to send syslog messages to these virtual ttys by merely specifying the specific device. It may also be useful to turn the tty "off" so that nobody can log in as described in the manpage for ttys(5).

OpenBSD offers a fifth action that writes log messages to an in-memory buffer. This will provide logged-in users access to logs even on systems with no local storage. This configuration must be used in conjunction with the syslogc(8) program that reads messages from the created buffer. For more information about this, consult the manpages for syslogd and syslogc.

FreeBSD's fifth option is to pipe selected messages to an external program. By specifying a vertical bar or pipe (|) symbol, the log message from syslogd is piped to the specified external program, much like the pipe symbol works on the command line. For more information about syslog actions, consult the manpage for syslog.conf(5).

10.2.5.1 Debugging syslogd

Remember that Syslog facilities vary by operating system. This makes it tricky to reuse syslog.conf files between operating systems, and you will need to be extra diligent about testing your configuration to make sure it does what you want. As you are building your Syslog configuration, you may notice that by default, syslogd offers little in the way of debugging information, even when you have horribly misconfigured syslog.conf.

Fortunately there are ways to get useful debugging information from syslogd. FreeBSD administrators may run the daemon with the -v option, which will cause syslogd to log the numeric facility and priority of every locally generated message. Specify the -v flag twice to also log the name of the facility and level. OpenBSD administrators may run syslogd with the -d option to run syslogd in the foreground and send debugging output to the attached terminal. Of course, when debugging, you may also want to capture the output of syslog.debug on either platform. Finally, to inject messages into Syslog with arbitrary levels and facilities, use the logger(1) tool.

10.2.6. Running syslogd

The syslogd binaries on FreeBSD and OpenBSD behave differently. While the functionality they provide is equivalent, they have different command-line arguments and defaults. We will briefly examine the most important command-line arguments for syslogd.

10.2.6.1 Additional sockets

Don't forget to configure syslogd to receive messages from jailed and chrooted programs. To accomplish this, you will first need to configure syslogd to create additional log sockets. In FreeBSD, the path to this socket is specified as the parameter to the -l argument, and in OpenBSD, as the parameter to -a. For example, BIND chrooted in /var/named will write system logs to /dev/log within the chroot. Therefore, syslogd must be run with /var/named/dev/log as the parameter to -l or -a. Unlike files specified in the action column in syslog.conf, these log sockets are created automatically when syslogd is run.

10.2.6.2 syslogd on FreeBSD

By default, syslogd on a FreeBSD system will run with the -s option. This tells it to operate in "secure mode," which causes the daemon to not listen for incoming UDP datagrams on port 514. This argument can be specified twice in the form -ss, which causes syslogd to not open a network socket at all. Doing this also disables syslogd's ability to log to other systems.

These defaults are specified in /etc/defauts/rc.conf. To override these defaults, use the syslogd_flags variable in /etc/rc.conf. Client syslogd systems should always have the -s argument specified. Loghost systems must not have the -s argument specified.

Finally, syslogd may also be configured to accept log messages from only certain systems specified either by IP address and mask or domain name. To activate this "allowed peer" functionality, use the -a option as described in the manpage for syslogd. This will prevent syslogd from performing actions on messages received from non-peers; -a does not act as a replacement for a local firewall, which will block the datagram at the host level preventing it from being received by the daemon.

10.2.6.3 syslogd on OpenBSD

syslogd on OpenBSD starts on boot. There is no good way to disable syslogd besides specifying invalid options in /etc/rc.conf.local, like syslogd_flags="NO". The default OpenBSD configuration allows logging to remote systems but does not accept input from the UDP port. To enable loghost functionality, the -u (unsecure) argument must be specified as an argument to syslogd.

In OpenBSD 3.4, privilege separation was added to syslogd. This creates two instances of syslogd: the child runs as user _syslogd and listens for log requests from log sockets while the parent process gives the child access to write to logfiles and so on. Privilege separation ensures that when running as a loghost, a root-owned process is not exposed to network traffic.

Finally, the section in /etc/rc that starts syslogd on OpenBSD systems will open up to two additional logging sockets. The first, /var/named/dev/log, will be opened if named has been configured to run in /etc/rc.conf. Since named can run chrooted, an additional logging socket is necessary. The second, /var/empty/dev/log, is created if /var/empty exists. This directory is used by a variety of system daemons that, like syslogd, fork a child process that listens on a network port and chroots itself in /var/empty.

10.2.7. syslogd Drawbacks

syslogd has a long history as the system logger for Unix systems. Despite its prevalence, there are a number of inherent security concerns, which are as follows.

10.2.7.1 Lack of access control

Without the ability to limit at a granular level which users or applications may use syslogd, everyone has access to the local Syslog daemon (which may forward messages onto a loghost, in certain circumstances). In addition, unless protected by a firewall or network topology, anyone can send logs directly to the loghost. This may lead to denial of service attacks that consume disk space or throughput (CPU or network).

There are unfortunately no easy ways to prevent users from logging to a local system logger using the logger command or the syslog system call. In general, loghost systems should be protected by a local firewall. FreeBSD loghost servers may instead use the -a (allowed peer) argument to syslogd.

Denial of service attacks should be mitigated wherever possible by using a separate filesystem for logs and by monitoring CPU and network activity closely. See the Section 4.6 in Chapter 4 for options to monitor activity on FreeBSD and OpenBSD systems.

10.2.7.2 Lack of reliability

The use of UDP for message transport minimizes network overhead due to logging but makes a variety of problems possible. Messages are easily spoofed, injected, silently dropped, or significantly delayed. The UDP protocol has no provisions for ensuring datagrams reach the destination; this is the responsibility of the application, if the application cares. In this case, syslogd does not. Without a guarantee that messages that were sent are delivered, logs on a loghost system may be incomplete or incorrect without anyone's knowledge.

As with mitigating the risks associated with a lack of access control, it's important to monitor systems closely to ensure adequate CPU cycles and network bandwidth are available for the transmission of these messages.

10.2.7.3 Lack of integrity or confidentiality

Without cryptographically verifiable signatures, there is no way to guarantee that messages have not been modified before being received by syslogd. Without encryption, messages are easily observable while in transit to a loghost system.

One mitigation strategy is to use netcat in conjunction with cryptcat to send UDP log messages over an encrypted TCP stream. netcat is installed by default on OpenBSD and may be installed via ports on FreeBSD. cryptcat is available in the FreeBSD ports tree and may be installed on OpenBSD by hand. After installation, non-loghost servers can be configured to log via UDP to localhost with the following line in the file /etc/syslog.conf:

*.info                                @localhost

netcat must then collect UDP datagrams from local port 514 and send these over an encrypted tunnel created by cryptcat as follows:

% sudo nc -l -u -p 514 | sudo cryptcat  loghost 12345

On the loghost server, cryptcat receives encrypted data from remote systems, converts it back into UDP traffic and sends this content to the local 514 port on the loghost. This is accomplished on the loghost side by a command similar to the following:

% sudo cryptcat -l -p  12345  | sudo nc -u localhost 514

While this approach works, it requires that all systems (not just the loghost) be configured to receive UDP datagrams. On FreeBSD systems, use the -b argument to syslogd to specify a bind address of 127.0.0.1, or localhost, thus preventing receipt of datagrams over the network. OpenBSD systems will need to be configured to use a local firewall. Since cryptcat uses twofish encryption, which relies on a symmetric key (both parties encrypt using the same key), you will have to modify the source code of cryptcat and replace the default key of "metallica" with something of your choosing.

FreeBSD administrators should make a point to adjust the Makefile and remove the flag GAPING_SECURITY_HOLE. This will disable unneeded inetd-like functionality in netcat.

After this configuration is complete for each client, the loghost server accepts log messages over an encrypted connection. Because messages are only received over encrypted tunnels with a symmetric key, there is higher message integrity. There is still no guarantee that all messages that were sent were actually received. There is also no guarantee of in-order delivery or timeliness. Users who can log in on the clients can still send arbitrary log entries to the loghost.

As you can see, this mitigation strategy requires a great deal of upfront administration and does not scale well. Every change to the symmetric key (and this should change regularly) will require recompilation and/or automated distribution of a recompiled binary of cryptcat. It demonstrates, however, the extent to which some individuals will go to mitigate the risks associated with the lack of confidentiality in message transport.

Unfortunately other options are limited to transmitting log messages over a secure network (perhaps only for administrative data like log messages), or replacing syslogd with another daemon altogether. We consider replacing syslogd in next section of this chapter.

10.2.7.4 Monolithic

The syslogd binary is monolithic. It accepts messages over the network and locally, filters based on facility and level, and also can perform a variety of actions all from within a single binary. From a separation-of-duties perspective, this is not great design. Compare this with a program like Postfix, which has separate binaries to handle distinct tasks some of these components need to run as root, others don't.

OpenBSD's support of privilege separation and chrooted behavior for syslogd mitigates most of the risk associated with this issue. Although still a single binary, the instance that communicates with external programs through sockets does not run as the root user and is carefully restricted from the rest of the filesystem.

There is little more that can be done about this security concern other than keeping an eye open for better-designed syslogd replacements.

10.2.8. syslogd Replacements

syslogd 's known security issues relating to lack of authentication, encryption, reliable delivery and message integrity have spawned several parallel efforts that attempt to solve this problem. For instance, an Internet Engineering Task Force (IETF) working group was formed to "document and address the security and integrity problems of the existing Syslog mechanism."^[1] While this group has made some progress like the creation of RFC 3195, which describes reliable syslog transmission, it will be a little while until the standards they create can be implemented. In the meantime, other attempts have been made to solve the problems associated with syslogd.

^[1] Chris Lonvick in May 2004. Available on the IETF Syslog Working Group home page (available at: http://www.employees.org/~lonvick/index.shtml).

10.2.8.1 syslog-ng

BalaBit, a Hungarian company, developed a "new generation" syslogd replacement in 1998, which was adopted by the Debian Linux distribution in 1999. syslog-ng boasts a variety of improvements over syslogd:

• Additional, more flexible, filtering options

• Communications over the more reliable, but less efficient, TCP protocol

• Support for TCP wrappers

• Logging of the complete chain of loghost relays to reach the destination loghost

Although this represents useful functionality, most of the security issues associated with syslogd as noted previously have not been addressed. Denial of service attacks are still possible without local access restrictions. syslog-ng's integration with TCP wrappers helps restrict hosts that can send logs to the loghost. However, the form of access control this provides can be accomplished, to a lesser extent, natively by syslogd on FreeBSD using the -a argument or by a firewall in either FreeBSD or OpenBSD. Finally, syslog-ng is an even larger daemon, which runs as root and relies on even more complex libraries to provide this additional functionality.

The major advantage provided by syslog-ng is its ability to use TCP instead of UDP. TCP support in itself increases reliability of log transport and unfortunately also decreases its ability to deal with large volumes of log messages. More importantly, TCP support means you can use syslog-ng in conjunction with a program like stunnel, which allows non-SSL aware TCP-based daemons to use SSL-based encrypted tunnels. This combination provides an effective solution to the confidentiality problem associated with sending logs in the clear over a potentially insecure network. It's not without tradeoffs, however, as throughput is further decreased with the overhead of encryption.

For more information about setting up syslog-ng, see the documentation at balabit.com. For detailed information about integrating syslog-ng and stunnel, see the examples page on http://www.stunnel.org.

10.2.8.2 minirsyslogd

The minirsyslogd project was driven by paranoia over the bloat associated with syslogd and the even larger syslog-ng. The daemon is small, advertised as "minimalistic," and low in flexibility and functionality; but it provides one key service that it does well.

minirsyslogd is a loghost daemon. It receives inbound log messages over a UDP port (by default 514) and performs as little processing as necessary to store messages in logfiles structured around the messages' source IP addresses. Because minirsyslogd refuses to create new directories, the existence of an IP-address named directory forms a rudimentary form of access control. This protects against arbitrary systems filling up disk space, but the daemon is still vulnerable to attack. This approach is less effective than using TCP wrappers, the -a argument to syslogd in FreeBSD, or a local firewall.

Nevertheless, the size and simplicity of the daemon makes a compelling case for a loghost or redundant high-security loghost system.

10.2.8.3 msyslog

The modular Syslog daemon (sometimes called ssyslog or Secure Syslog) is based on OpenBSD's Syslog daemon and is available in the FreeBSD ports tree (ports/sysutils/msyslog) but must be compiled by hand on OpenBSD. This daemon replaces the existing Syslog daemon and is designed as the name implies: separate modules are written to handle input, output, and file protection through encryption and hashing.

Installation of msyslog will result in the replacement of the syslogd binary, installation of the peochk utility to generate keys for file encryption and check the integrity of encrypted logfiles, and the installation of support for modular input/output specification (contained within /usr/local/lib/alat/libmsyslog.so.1). Input modules provide support for reading messages from the BSD kernel, files and named pipes, UDP, and TCP. Likewise output modular support is available for typical syslogd functionality, a more flexible regular-expression based output specification, and the ability to send logs elsewhere via UDP or TCP.

Modular Syslog has even more flexibility and enhanced functionality than syslog-ng provides. One of the most important differentiators is the ability to encrypt logfiles and maintain a hash record of logfiles, which can be used to verify their integrity. While the modular design is good, the daemon and modular code will nevertheless always run as root.

There are, of course, more alternatives to syslogd available with new options appearing regularly. Visit the library at http://www.loganalysis.org for a current list of syslogd alternatives and additional information and resources.

10.2.9. Capturing Logs

What is captured varies from system to system, but in general there should be a log of all interactions with other systems. This certainly includes connection attempts (successes and failures), mail flow, FTP connections, and so on depending on the role of the system. Remember that syslogd discards any messages that do not match a known facility/level specification. A good rule of thumb is to ensure all messages in all facilities of higher importance than debug are logged (*.info in Syslog terms). This helps ensure that when the time comes to answer difficult questions, you have the information you need to reconstruct events and build an audit trail.

How precisely you capture all the logs in which you are interested will depend on the system in question. You may be very interested in ftpd logs access on an externally accessible FTP server or you may want to know whenever someone tries to transfer zones from your primary DNS server. In order to capture logs to build a successful audit trail, you need to carefully configure logging in every application. Look closely at the relevant documentation for more information on getting this done. If your applications are not producing appropriate logs, your job of building an audit trail will be much more difficult. Once this is done, make sure your applications are logging to an appropriate Syslog facility or level, if they use Syslog at all.

After having determined which facilities and levels interest you, it's straightforward to configure Syslog to capture these log messages and write them into appropriate logfiles. In most cases, it makes sense to write logs locally. This allows you to grant various administrators on different systems access to logs in which they are interested. Although this helps you answer questions about a given system, following a trail of logs across multiple systems can be very time consuming without also sending log messages to a central loghost.