Create a Centralized Server with syslog-ng and stunnel

Create a Centralized Server with syslog-ng and stunnel

To take your security to the next level, you should consider installing a more powerful version of syslog called syslog-ng . When the syslog daemon sends its log messages to a centralized server, it does so using the UDP protocol and in plaintext. This is not a good idea, as logs can contain sensitive information such as passwords. Also, the UDP protocol doesn t ensure that the information makes it to its destination. In the next section, there will be detailed instructions about adding additional software to your system.

SUSE: Download and Install stunnel 4.04

SUSE ships with stunnel 3.14. It is recommended but not essential that you download a newer version of stunnel such as 4.04 or newer. One quick way to get newer source code for stunnel is with wget :

 wget http://www.stunnel.org/download/stunnel/src/stunnel-4.04.tar.gz 

The next steps are the same as installing most source code, and are the same steps as the ones for compiling syslog-ng in the next section:

 tar -xzvf stunnel-4.04.tar.gz cd  stunnel-4.04/ ./configure make make install 

Download and Install syslog-ng

These instructions have been tested for setting up a centralized server on Red Hat Enterprise Linux AS 3.0, Red Hat 9.0, and SUSE SLES8. The first step in installing syslog-ng is to get the distribution. At the time of this printing, there are no widely disseminated RPM packages, so we will step you through building syslog-ng and its dependencies from source. Acquire the archive for syslog-ng and the dependency that is outside the standard distribution for the operating systems listed above. We will be working with syslog-ng-1.6.tar.gz and libol-0.3.13.tar.gz since they both are easy to build in these environments.

First, go through the steps for libol and then repeat the steps with syslog-ng .

1. Unpack the archive and cd into the directory that was created:

    # tar -zxf  filename  # cd  directory  

2. Run configure , make , and make check . You should not see any errors produced by make check . If you do, you need to fix them.

    # ./configure; make; make check 

3. Now install the program:

    # make install 

Great! You have installed the software you need, now all that is left is to configure it.

Heads Up  

Adding software that is not provided by your vendor may void support. You should make your decision regarding addition of software after discussing your needs with your vendor.

Create Certificates for Your Machines.

In Red Hat, you can do the following:

 # cd /usr/share/ssl/certs # make syslog-ng-server.pem # make syslog-ng-client.pem 

The SUSE distribution does not include the same makefile to create certificates that Red Hat does. You can make a script with the following commands and execute it to make your server and client certificates:

 #!/bin/bash ### ### A quick script for making certificates for use with STUNNEL ### umask 77 ; \ PEM1='/bin/mktemp /tmp/openssl.XXXXXX' ; \ PEM2='/bin/mktemp /tmp/openssl.XXXXXX' ; \ PEM3='/bin/mktemp /tmp/openssl.XXXXXX' ; \ PEM4='/bin/mktemp /tmp/openssl.XXXXXX' ; \ /usr/bin/openssl req -newkey rsa:1024 -keyout $$PEM1 -nodes -x509 -days 365 -out $$PEM2 ; \ cat $$PEM1 >  syslog-ng-server.pem ; \ echo ""    >> syslog-ng-server.pem; \ cat $$PEM2 >> syslog-ng-server.pem ; \ echo "Done Making Server Certificate."; /usr/bin/openssl req -newkey rsa:1024 -keyout $$PEM3 -nodes -x509 -days 365 -out $$PEM4 ; \ cat $$PEM3 >  syslog-ng-client.pem ; \ echo ""    >> syslog-ng-client.pem; \ cat $$PEM4 >> syslog-ng-client.pem ; \ rm  $$PEM1 $$PEM2 $$PEM3 $$PEM4 rm /tmp/openssl.* echo "Done Making Client Certificate."; 

Copy Certificates to /etc/stunnel

The server and client machines have different certificate requirements. The clients only need the certificate section of syslog-ng-server.pem. Remove the private key section from syslog-ng-server.pem. Next, copy the file syslog-ng-server.pem to each client machine and put it in the /etc/stunnel directory. You will know the private key section because it starts with

 -----BEGIN RSA PRIVATE KEY----- 

and ends with

 -----END RSA PRIVATE KEY----- 

On the server, copy the entire file into /etc/stunnel. Also, place the client s syslog-ng-client.pem in /etc/stunnel. For the server, create a special syslog-ng-client.pem containing the certificate sections for all the clients. You should remove the private key sections from all syslog-ng-client.pem files and concatenate what is left to create the server s special syslog-ng-client.pem.

Check Certificate Permissions

Make sure the server certificates are owned by root and have read and write permissions. These commands will ensure that:

 chown  root:root  syslog-ng-server.pem chown  root:root  syslog-ng-client.pem chmod 700 syslog-ng-server.pem chmod 700 syslog-ng-client.pem 

Create stunnel Configuration on the Server

This code listing shows a possible server configuration for stunnel that would belong in file /etc/stunnel/stunnel.conf:

 cert = /etc/stunnel/syslog-ng-server.pem CAfile = /etc/stunnel/syslog-ng-client.pem [5140]        accept = server IP address:5140        connect = 127.0.0.1:514 

Create stunnel Configuration on the Client

This code listing shows a possible client configuration for stunnel that would belong in file /etc/stunnel/stunnel.conf:

 client = yes    cert = /etc/stunnel/syslog-ng-client.pem    CAfile = /etc/stunnel/syslog-ng-server.pem    verify = 3    [5140]        accept = 127.0.0.1:514        connect = server IP address:5140 

Create syslog-ng Configuration on the Server

This code listing shows a possible server configuration for syslog-ng that would belong in file /etc/syslog-ng.conf:

 options {  long_hostnames(off);               sync(0);               keep_hostname(yes);               chain_hostnames(no);  };    source src {unix-stream("/dev/log");                pipe("/proc/kmsg");                internal();};    source stunnel {tcp(ip("127.0.0.1")                    port(514)                    max-connections(1));};    destination remoteclient {file("/var/log/remoteclient");};    destination dest {file("/var/log/messages");};    log {source(src); destination(dest);};    log {source(stunnel); destination(remoteclient);}; 

Create syslog-ng Configuration File on the Client Machines

This code listing shows a possible client configuration for syslog-ng that would belong in file /etc/syslog-ng.conf:

 options {long_hostnames(off);             sync(0);};    source src {unix-stream("/dev/log"); pipe("/proc/kmsg");                internal();};    destination dest {file("/var/log/messages");};    destination stunnel {tcp("127.0.0.1" port(514));};    log {source(src);destination(dest);};    log {source(src);destination(stunnel);}; 

Start stunnel and syslog-ng Manually

These are two commands that need to be executed to start stunnel and syslog-ng manually:

 # stunnel # syslog-ng -f /etc/syslog-ng.conf 

Check for Activity on the Server

The command tail -f filename is an excellent way to watch logfile activity. Log into the server and watch the file for activity in real time. To create activity, type su - on a client machine and press RETURN a couple of times to simulate a failed password.

 # tail -f  /var/log/remoteclient 

If you should see some activity like this, everything is working:

 Apr 1 12:34:56 chim su(pam_unix)[8451]: authentication failure; logname=rreck uid=2112 euid=0 tty= ruser=rreck rhost=  user=root 

If everything is not working as you expect, you can try running stunnel with increased debugging. In SUSE ( stunnel version 3.14) you can type:

 # stunnel -D 7 

In stunnel version 4.04, which is default with Red Hat, you can add a line like this to /etc/stunnel/stunnel.conf for maximum verbosity :

 debug = 7 

Carefully read the messages and try to determine what is failing. It can be something as simple as a typo in one of the configuration files. Once things are working you can use these scripts to start and stop stunnel and syslog-ng and put them in /etc/init.d (Red Hat) or /etc/rc.d/ (SUSE). This code listing shows a sample startup script for stunnel , complete with LSB-compliant comment conventions:

 #!/bin/sh   ### BEGIN INIT INFO   # Provides: stunnel   # Description: A stunnel version 4.04 startup script    # with  LSB Comment Conventions.   # Adapted from an example script by [e-mail]   # Further Adapted by Ronald P. Reck <[e-mail]> ### BEGIN INIT INFO   # Provides: stunnel   # Required-Start: $network    # Default-Start: 3 4 5 # Default-Stop: 0 1 2 6   # Short-Description: Start / Stop stunnel    ### END INIT INFO   # Source function library.   . /lib/lsb/init-functions   [ -f /usr/sbin/stunnel ]  exit 0   # Where is the stunnel program  STUNNEL="/usr/sbin/stunnel" case "" in      start)         ${STUNNEL} /etc/stunnel/stunnel.conf              ;;     stop)         killall 'basename ${STUNNEL}'         ;;     *)         echo ""         echo "Usage: basename 
 #!/bin/sh ### BEGIN INIT INFO # Provides: stunnel # Description: A stunnel version 4.04 startup script # with LSB Comment Conventions. # Adapted from an example script by [e-mail] # Further Adapted by Ronald P. Reck <[e-mail]> ### BEGIN INIT INFO # Provides: stunnel # Required-Start: $network # Default-Start: 3 4 5 # Default-Stop: 0 1 2 6 # Short-Description: Start / Stop stunnel ### END INIT INFO # Source function library. . /lib/lsb/init-functions [ -f /usr/sbin/stunnel ]  exit 0 # Where is the stunnel program STUNNEL="/usr/sbin/stunnel" case "$1" in start) ${STUNNEL} /etc/stunnel/stunnel.conf ;; stop) killall 'basename ${STUNNEL}' ;; *) echo "" echo "Usage: basename $0 { start  stop }" echo "" ;; esac exit 0; 
 { start  stop }"         echo ""         ;; esac    exit 0; 

This code listing shows a sample startup script for syslog-ng :

 #!/bin/sh   # # syslog-ng        Starts/Stops syslog-ng   # # chkconfig: 345 11 70   # description: syslog-ng is a enhanced system and kernel logging daemon   # Original Author:      Georg Funke, <[e-mail]> # Modifed by         Ronald P. Reck <[e-mail]> # /etc/init.d/syslog-ng     # LSB 1.1.0 header information   ### BEGIN INIT INFO   # Provides: syslog-ng   # Required-Start: network   # Required-Stop:  network   # Default-Start:  2 3 5   # Default-Stop:   # Description:    Start/Stop the syslog-ng logging daemon   ### END INIT INFO   # Source LSB initscript functions.   . /lib/lsb/init-functions # this is default place syslog-ng is installed to   [ -f /usr/local/sbin/syslog-ng ]  exit 0   # See how we were called.   case "" in     start)     echo -n "Starting system logger syslog-ng: "     daemon syslog-ng -f /etc/syslog-ng/syslog-ng.conf        echo      touch /var/lock/subsys/syslog-ng      ;;   stop)     echo -n "Shutting down system logger syslog-ng: "     killproc syslog-ng           echo        rm -f /var/lock/subsys/syslog-ng      ;;   restart) 
 #!/bin/sh # # syslog-ng Starts/Stops syslog-ng # # chkconfig: 345 11 70 # description: syslog-ng is a enhanced system and kernel logging daemon # Original Author: Georg Funke, <[e-mail]> # Modifed by Ronald P. Reck <[e-mail]> # /etc/init.d/syslog-ng # LSB 1.1.0 header information ### BEGIN INIT INFO # Provides: syslog-ng # Required-Start: network # Required-Stop: network # Default-Start: 2 3 5 # Default-Stop: # Description: Start/Stop the syslog-ng logging daemon ### END INIT INFO # Source LSB initscript functions. . /lib/lsb/init-functions # this is default place syslog-ng is installed to [ -f /usr/local/sbin/syslog-ng ]  exit 0 # See how we were called. case "$1" in start) echo -n "Starting system logger syslog-ng: " daemon syslog-ng -f /etc/syslog-ng/syslog-ng.conf echo touch /var/lock/subsys/syslog-ng ;; stop) echo -n "Shutting down system logger syslog-ng: " killproc syslog-ng echo rm -f /var/lock/subsys/syslog-ng ;; restart) $0 stop $0 start ;; reload) echo -n "Reloading syslog-ng: " killproc syslog-ng -HUP echo ;; *) echo "Usage: syslog-ng {startstoprestartreload}" exit 1 esac exit 0 
 stop 
 #!/bin/sh # # syslog-ng Starts/Stops syslog-ng # # chkconfig: 345 11 70 # description: syslog-ng is a enhanced system and kernel logging daemon # Original Author: Georg Funke, <[e-mail]> # Modifed by Ronald P. Reck <[e-mail]> # /etc/init.d/syslog-ng # LSB 1.1.0 header information ### BEGIN INIT INFO # Provides: syslog-ng # Required-Start: network # Required-Stop: network # Default-Start: 2 3 5 # Default-Stop: # Description: Start/Stop the syslog-ng logging daemon ### END INIT INFO # Source LSB initscript functions. . /lib/lsb/init-functions # this is default place syslog-ng is installed to [ -f /usr/local/sbin/syslog-ng ]  exit 0 # See how we were called. case "$1" in start) echo -n "Starting system logger syslog-ng: " daemon syslog-ng -f /etc/syslog-ng/syslog-ng.conf echo touch /var/lock/subsys/syslog-ng ;; stop) echo -n "Shutting down system logger syslog-ng: " killproc syslog-ng echo rm -f /var/lock/subsys/syslog-ng ;; restart) $0 stop $0 start ;; reload) echo -n "Reloading syslog-ng: " killproc syslog-ng -HUP echo ;; *) echo "Usage: syslog-ng {startstoprestartreload}" exit 1 esac exit 0 
 start      ;;   reload)     echo -n "Reloading syslog-ng: "     killproc syslog-ng -HUP       echo      ;;   *)     echo "Usage: syslog-ng {startstoprestartreload}"     exit 1   esac   exit 0 

Use the logger Command to Send Messages Directly to the syslog Daemon

You might want to log some piece of activity directly to the syslog daemon. The logger command can help you do it and it is easy to use. If you wanted to keep track of what the load was or how many users were logged in when someone new logged in, you could gather this information by editing the /etc/bashrc file and adding a line like

 logger -p info "user $USER starting a shell at 'w  head -1'" 

This would make entries into /var/log/messages like

 Apr  1 06:34:46 chim rreck: user rreck starting a shell at  06:34:46   up 53 min,  3 users,  load average: 0.06, 0.03, 0: 

Use Perl s Sys:Syslog to Send Messages to the syslog Daemon

You can easily send messages to syslog using the standard libraries that accompany Perl. This brief example shows how you can use a program to send messages to syslog using a subroutine call:

 #!/usr/bin/perl ### ### An example that sends a message to syslog using Perl ### # a required library use Sys::Syslog qw (:DEFAULT setlogsock); # a default message and priority $user = $ENV{'USER'}; $msg='w grep $user wc -l'; $msg="$user is logged in $msg times"; $priority="info"; # call the subroutine &log_message($priority,$msg); sub log_message{ my ($priority, $msg) = @_; setlogsock('unix'); openlog( 
 #!/usr/bin/perl ### ### An example that sends a message to syslog using Perl ### # a required library use Sys::Syslog qw (:DEFAULT setlogsock); # a default message and priority $user = $ENV{'USER'}; $msg='w grep $user wc -l'; $msg="$user is logged in $msg times"; $priority="info"; # call the subroutine &log_message($priority,$msg); sub log_message{ my ($priority, $msg) = @_; setlogsock('unix'); openlog($0,'pid,cons','user'); syslog($priority, $msg); closelog(); return 1; } 
 ,'pid,cons','user'); syslog($priority, $msg); closelog(); return 1; } 

Save this example in a file, and make it executable by typing chmod +x filename . When you execute it, you will get a message sent to syslog that looks like:

 May  1 00:16:29 chim ./syslogmessage.pl[20659]: root is logged in  7  times 

Manage Logfiles

Logfiles can get large fast, depending on the amount of activity you have on your network and the verbosity of the applications being logged. Eventually, it will be necessary to rotate your logfiles. There is a very useful and highly configurable tool for rotating logs called logrotate . logrotate has been around for many years and is scheduled in root s crontab to run on a daily basis by default on most current versions of Linux. logrotate has too many options to cover them all here, but we will cover the basics. The first thing to know is that logrotate supports automatic rotation, compression, removal, and mailing of system or application logfiles. The second thing to know is logrotate has a default configuration file located in /etc/logrotate.conf. logrotate supports multiple configuration files and it s important to note that local definitions override global ones and later definitions override earlier ones. Any number of configuration files can be given to logrotate on the command line. Alternatively, the include directive allows one configuration file to point at many others. Since order can influence configuration precedence, this should be taken into account when planning things out. Application-specific configuration files are in the /etc/logrotate.d/ directory.

Tip  

If you are centralizing your logs, you can tell logrotate to delete logs locally instead of rotating them by using a maxage setting equal to the rotate time. Make sure you confirm that the logs are going to the centralized log server before enabling this option.

Since logrotate usually runs as a daily cron job it will not modify a log multiple times in one day unless the log s size meets the size criteria and logrotate is invoked more than once a day. To force logrotate to perform an activity like when you are testing use logrotate -f or logrotate --force .

Finding Logfiles

In general, most of the system logfiles live in /var/ as specified by the syslog.conf. If there are other applications running on the server, they may have other logfiles in other places. To make all the applications put their logs under /var/ you will need to edit each application s configuration file(s). Having logs under /var/ is a good idea because it means that all logs are in a single place when you need to check them, restrict access, or back them up. Second, /var/ is often in a file system independent from slash (/), which is important in case your logs get out of hand and fill up all the available space. If logfiles are written anywhere below the slash (/) file system, a hacker can crash your machine by causing enough logged activity that it fills the entire hard drive. You can find logfiles you didn t know about using the find command. Change to the slash directory and run the following command as root to locate files that were recently modified.

 find . -type f -mtime -5 print  grep -v proc  grep -v lock