Using a Central syslog Server


Using a Central syslog Server

One of the important aspects of maintaining network security is keeping and checking various system and application log files. Other than being useful in determining and debugging network problems, the log files can also serve as early-warning indicators of an impending attack: Often you will see "probes" over a period of days or even weeks before an actual penetration attempt takes place.

NOTE

You can find discussions about log file maintenance, log parsing automation, and alerts in Chapter 12.


After an incident has occurred, good logs can help you determine how the attack was launched and what systems may have been compromised. In addition, the logs may even indicate the source (or the "alleged" source) of the intrusion. Because hackers know the values of these log files, they often wipe out or alter the logs, as well as install a Trojan logging daemon, to cover their tracks. To counter this compromise to your system, centralized logging could save the day. After all, if a machine is compromised but the log evidence isn't kept on that machine, it is going to be much more difficult for the intruder to cover his tracks. In addition to the extra level of protection, monitoring the logs for the whole network is also much easier when they are all located in one place.

TIP

By default, syslog messages are stored in various files in /var/log. However, you can change the filenames and locations by modifying /etc/syslog.conf.


syslog is the tried-and-true workhorse of Unix logging utilities, and it is not surprising that it is also used on Linux. syslogd accepts log data from the kernel (by way of klogd), from any and all local processes, and even from processes on remote systems. As part of the SLES base system installation, syslogd is preconfigured to use the local host as the receiver.

To set up a central syslog server, simply start syslogd with the -r switch, which tells it to listen for messages from remote machines on UDP port 514. On an SLES 9 server, syslogd is set up to automatically start at system boot, and the script used is /etc/init.d/syslog. This startup references /etc/sysconfig/syslog for a few variable settings, including SYSLOGD_PARAMS. Therefore, you should modify /etc/sysconfig/syslog and add -r to the SYSLOGD_PARAMS line, as follows:

 SYSLOGD_PARAMS="-r" 

Then restart syslogd using the /etc/init.d/syslog restart or /sbin/rcsyslog restart command.

TIP

Do you see a bunch of lines like this in /var/log/messages, about every 20 minutes?

 Jan  8 00:19:35 Athena -- MARK -- Jan  8 00:39:36 Athena -- MARK -- Jan  8 00:59:35 Athena -- MARK -- Jan  8 01:19:36 Athena -- MARK -- Jan  8 01:39:35 Athena -- MARK -- Jan  8 01:59:36 Athena -- MARK -- 

These lines are generated by the "mark" function (or subsystem) of syslog as a way of "touching base" with the system so that you can (theoretically) tell if syslog has unexpectedly died and stopped logging. Most times, this feature only serves to take up your disk space. Unless you are having trouble with syslog, you probably don't want this feature active. To turn off the mark function, add -m 0 to the SYSLOGD_PARAMS setting in /etc/sysconfig/syslog and then restart syslogd. (When -m isn't specified, the default value is 20 minutes.)


The next step is to instruct the clients to forward their syslog traffic to the central log host. You accomplish this easily by adding a single line in each client's /etc/syslog.conf file:

 *.*     @loghostname 

If this is the only line in the configuration file, all messages will be sent to the remote host, loghostname. Otherwise, the log messages will be stored both locally (as per the settings in the rest of the configuration file) and remotely for safekeeping. If you do this, it would be fairly easy to develop a script that uses grep or a similar tool to extract particular lines from the remote copy and compare to those from the local copy. Any data mismatch would indicate possible tampering.

Alternatively, you can arrange for selected message categories (known as facilities, in syslog-speak) to be sent to a remote server while keeping other message types local. Consider the following two entries:

 auth,authpriv.warning     @loghostname auth,authpriv.!warning    /var/log/messages 

The first line instructs syslog to send any auth (mainly security events) and authpriv (access-controlrelated events) messages whose priority is warning or higher to the remote logging host, loghostname. The second line tells syslog to store the rest of the auth and authpriv messages locally in /var/log/messages.

Hardening the Central Syslog Host

One drawback of remote logging is that the stock syslogd does not provide any means of authentication or access control with regard to which remote hosts a receiving syslog server should accept the logs from. An attacker can determine, with relative ease, whether you have a central syslog server. For example, the attacker can read the /etc/syslog.conf file on the system he compromised or simply sniff for UDP traffic that is using port 514. After the attacker finds out that information, he may also choose to compromise your central syslog server to modify the log files and cover his tracks. Or he may simply flood it with bogus syslog messages until it runs out of disk space and stops logging.

TIP

You may want to consider running a dedicated syslog server that has no other active services except for the syslog daemon running. Furthermore, you can block UDP port 514 on your Internet-facing firewall so your syslog host cannot be easily flooded from the outside; of course, this wouldn't help much if it is done inside your network, after the intruder has exploited one of your servers.


TIP

A more paranoid setup is to disable all logins to the central logging host except for the console. That way, to access the logs stored on this server, the person needs to have physical access.


As a precautionary measure, at the very least, you should consider running syslogd via xinetd. This way, you can use xinetd's access control feature (such as the only_from list) so syslogd will log messages only from trusted hosts. Alternatively, you can set up some local firewall filtering rules using iptables.

Ideally, you would want to use a syslog daemon that uses TCP instead of UDP communication so that data delivery is reliable and you don't risk losing logging information. Furthermore, with TCP traffic, you can use something like ssh or stunnel (http://www.stunnel.org) to secure the traffic between your syslog daemons. You can do this using a replacement syslog daemon called syslog-ng (syslog new generation).

NOTE

syslog-ng v1.6 is included on your SLES 9 media. By default, the traditional syslog is installed instead of syslog-ng, but you can easily install syslog-ng via YaST. You can download the latest version from http://www.balabit.com/downloads/syslog-ng.


NOTE

Also included with SLES 9 is a package called evlog (Enterprise Event Logging). evlog implements a syslogd-compatible logging service that is in compliance with draft POSIX Standard 1003.25, "System APIServices for Reliable, Available, and Serviceable Systems." Visit http://evlog.sourceforge.net for more information.


syslog-ng uses /etc/syslog-ng/syslog-ng.conf for configuration information. The structure of this file is vastly different from that used by syslog. You may find that configuring syslog-ng takes a little more work than configuring syslog, but that is a reflection of its flexibility. The configuration file consists of five sections: options{}, source{}, destination{}, filter{}, and log{}. Syntactically, syslog-nf.conf is very similar to C and other structured programming languages: Semicolons terminate statements; whitespaces are ignored and may therefore be used to enhance readability. The following is a sample syslog-ng.conf file:

 options {use_dns(yes); chain_hostnames(yes);}; source s_location {internal();}; destination d_loghostname {tcp("myloghost" port(514));}; destination d_local {file("/var/log/messages");}; filter f_auth_messages {facility(auth, authpriv)      and level(warning .. emerg);}; log {source(s_location); destination(d_local);}; log {      source(s_location);      filter(f_auth_messages);      destination(d_loghostname);     }; 

This file instructs syslog-ng to perform the following tasks and sets up this information:

  • The file resolves the IP address of the sender to the hostname via DNS and shows names of all hosts that have handled the log message.

  • The source of the logs is internal to the server (that is, from syslog-ng itself).

  • Two destinations have been defined. One is a remote logging host, myloghost, and the other is a local file.

  • A filter is set up to log only auth and authpriv messages whose priority is warning or higher.

  • All messages (of any type or priority) are stored locally in the /var/log/messages file.

  • Messages of the type auth and authpriv and whose priority level is warning or higher (because of the filter called f_auth_messages) are sent to the remote logging host.

When you gain an understanding of the various keywords, you will find that it's actually much easier and simpler to configure syslog-ng than syslog due to the more readable syntax.

TIP

You can use the logger command-line tool to test your syslog/syslog-ng configuration. The syntax of logger is simple:

 logger -p facility.priority "message" 

You can use the following shell script to send test messages to the kernel facility for each of the eight priorities:

 #!/bin/sh for i in {debug,info,notice,warning,err,crit,alert,emerg} do         logger -p kern.$i "Test kernel message, prio $i" done 

The following messages are logged to /var/log/messages:

 Mar  5 16:07:33 Athena logger: Test kernel message, prio debug Mar  5 16:07:33 Athena logger: Test kernel message, prio info Mar  5 16:07:33 Athena logger: Test kernel message, prio notice Mar  5 16:07:33 Athena logger: Test kernel message, prio warning Mar  5 16:07:33 Athena logger: Test kernel message, prio err Mar  5 16:07:33 Athena logger: Test kernel message, prio crit Mar  5 16:07:33 Athena logger: Test kernel message, prio alert Mar  5 16:07:33 Athena logger: Test kernel message, prio emerg 

In addition to being a diagnostic tool, logger is especially useful for adding logging functionality to shell scripts.


Although syslog-ng is much more secure than syslog, an intruder can still find out that there is a separate logging server because of the configuration files and take it out of commission using some form of DoS attack or perhaps hijack or divert the log traffic by forging addresses. Lance Spitzner of the Honeynet Project (http://www.honeynet.org) suggested an interestingnot hack-proof, but conceptually very interestingstealth logging method. Using this method, you hide the real logging destination from an intruder's view by using a host that does not have any network protocols (such as TCP/IP) bound to its network card.

A Stealth Logging Host

The operational theory behind a stealth logging host is based on the fact that syslogd receives messages passively as syslog messages are sent using the connectionless (one-way) UDP protocol; the sending hosts don't require an acknowledgment or any form of reply from syslogd when they send a syslog message. Therefore, you configure the syslog.conf files on the sending hosts to use a bogus but valid IP address for the remote logging host. That is, the IP address must be valid for the network numbering scheme you're using, but it should not be in use by any host on your network. For the host that is acting as the central logging host, you set up a network sniffer, such as Snort, to passively capture data packets destined for UDP port 514 (the syslog protocol port) off the wire.

NOTE

You cannot use tcpdump for stealth logging because it filters data packets based on packet headers, and not the data payload instead.


Mick Bauer wrote an excellent article titled "Stealthful Sniffing, Intrusion Detection and Logging" in the September 2002 issue of Linux Journal detailing how to use Snort, running in stealth mode, to act as syslogd. You can find an online version at http://www.linuxjournal.com/article/6222. As you can see in the comments at the end of the article, stealth logging is not a hack-proof solution, but it does introduce some possibilities.

If you are intrigued with the idea of a stealth syslog host but don't want to go through the trouble of setting up Snort or a packet sniffer, you may want to give passlogd a try. passlogd is a purpose-built syslog message sniffer that runs on a host without any open ports. You can find the latest version at http://www.morphine.com/void/src.html.

NOTE

The stealth logging technique works for both syslog and syslog-ng. However, the method does not work with syslog-ng if you secure the syslog message endpoints with ssh or stunnel and the like because the traffic that would be captured by the packet sniffer would be encrypted. passlogd does not work with syslog-ng if syslog-ng is configured to use TCP instead of UDP; however, you can easily set up Snort to look for TCP port 514 instead of UDP port 514.


Regardless of whether you are using stealth logging, if you use a central logging server, you should set up a monitoring system so that you are immediately alerted if this logging server is down for any reason because it could be a prelude to an intrusion.



    SUSE LINUX Enterprise Server 9 Administrator's Handbook
    SUSE LINUX Enterprise Server 9 Administrators Handbook
    ISBN: 067232735X
    EAN: 2147483647
    Year: 2003
    Pages: 134

    flylib.com © 2008-2017.
    If you may any questions please contact us: flylib@qtcs.net