Hack87.Centralize System Logs Securely

Hack 87. Centralize System Logs Securely

Protect your valuable logfiles from prying eyes

In "Fine-Tune the syslog Daemon" [Hack #86], we discussed configuration of the syslog daemon. As useful and even necessary as this logging service is, though, it's beginning to show its age. In response to that, a company name BalaBit has devoted both time and resources to bringing us the next generation of syslog, syslog-ng, which addresses many of the problems that plague the original. Improvements include using TCP instead of UDP to communicate with remote log hosts and a much more configurable interface to your system's logging capabilities. From a security standpoint, the implementation of TCP is a great advancementthat allows us to use additional applications such as stunnel to create encrypted tunnels to protect the contents of logfiles as they are sent to the central log host. In this hack, we examine such a deployment.

9.11.1. Getting Started

To implement encrypted remote logging, you'll need to download and compile three programs. Let's start with stunnel. Grab the latest instance of the source code from http://www.stunnel.org/download/source.html. Once you've got the tarball, unpack it and navigate to your newly created directory. You can now follow the typical installation procedure:

 $ ./configure $ make # make install 

You'll now need to grab the source for syslog-ng and libol, a library required by syslog-ng. You can download each of these from http://www.balabit.com/downloads/syslog-ng/. Untar and install libol first, then syslog-ng. Installation of these two applications uses the previous typical source install three-step.

Once you've successfully installed stunnel, syslog-ng, and libol, you'll need to create encryption certificates for all the machines between which you want to transfer secure log information.

9.11.2. Creating Your Encryption Certificates

To transfer log data securely between a remote host and a central log host, communication between the two must be encrypted. In order to successfully use encryption, both hosts must be able to verify their identities and share the encryption keys used for reading and writing the encrypted data. This information is provided by SSL certificates, which can either be granted by a third party or created yourself for use within your organization. (For more than you probably want to know about SSL and certificates, see the SSL HOWTO at http://www.tldp.org/HOWTO/SSL-Certificates-HOWTO/.)

At this point, you must create multiple certificates: one for use by the central log server, and one for each client that sends log information to the server. Later in this section, you'll install the server certificate on your server and distribute the client certificates to the hosts for which they were created.

The process for creating certificates varies slightly based on the Linux distribution you're using. For a Red Hat system, it is as follows:

 # cd /usr/share/ssl/certs # make syslog-ng-server.pem # make syslog-ng-client.pem 

As each certificate is generated, the script will ask you several questions regarding your location, hostname, organization, and email address. Once all the questions have been answered, your certificates are generated. Your next step is to verify that only root has access to them:

 [root@aardvark certs]# ls -l *.pem -rw-------1 root root 2149 Aug 14 12:12 syslog-ng-client.pem -rw-------1 root root 2165 Aug 14 12:12 syslog-ng-server.pem [root@aardvark certs]# 

There is one last thing you'll need to do before you start distributing your certificates: extract the CERTIFICATE section from each certificate that is going to a client machine and concatenate the extracted sections into a single file named syslog-ng-client.pem, which you will put on your server along with the server key. The CERTIFICATE key data in a certificate file is the information between the following two lines:

 -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- 

Copy the syslog-ng-client.pem file over to the /etc/stunnel directory on the server and place a copy of each client's own certificate in that client's /etc/stunnel directory. This may sound somewhat complicated, so let's summarize: all you're doing here is extracting the CERTIFICATE from each client's certificate file, concatenating that information into one large client certificate that will reside on your server (along with the server's certificate), and then copying the individual client certificates to the hosts for which they were intended.

9.11.3. Configuring stunnel

Now, on the server side, edit your stunnel.conf file to read as follows:

 cert = /etc/stunnel/syslog-ng-server.pem CAfile = /etc/stunnel/syslog-ng-client.pem verify = 3 [5140] accept = your.server.ip:5140 connect = 127.0.0.1:514 

Then make similar changes to stunnel.conf on the client side:

 client = yes cert = /etc/stunnel/syslog-ng-client.pem CAfile = /etc/stunnel/syslog-ng-server.pem verify = 3 [5140] accept = 127.0.0.1:514 connect = your.server.ip:5140 

9.11.4. Configuring syslog-ng

Once those changes have been made, it's time to start working on creating your syslog-ng.conf file. The syntax of this file has a steep learning curve and is well beyond the scope of this hack, so use what I'm about to show you as a starting point, and work from there. Far more detail can be found online and in the manpages. On your central log server, add the following to /etc/syslog-ng/syslog-ng.conf:

 options { long_hostnames(off); sync(0); keep_hostname(yes); chain_hostnames(no); }; source src {unix-stream("/dev/log"); pipe("/proc/kmsg"); internal();}; source stunnel {tcp(ip("127.0.0.1") port(514) max-connections(1));}; destination remoteclient {file("/var/log/remoteclient");}; destination dest {file("/var/log/messages");}; log {source(src); destination(dest);}; log {source(stunnel); destination(remoteclient);}; 

Then, add the following to your syslog-ng.conf file on each client:

 options {long_hostnames(off); sync(0);}; source src {unix-stream("/dev/log"); pipe("/proc/kmsg"); internal();}; destination dest {file("/var/log/messages");}; destination stunnel {tcp("127.0.0.1" port(514));}; log {source(src);destination(dest);}; log {source(src);destination(stunnel);}; 

9.11.5. Testing

Once you've done all this, you can start stunnel and syslog-ng to see if everything is working. Before you do so, though, make sure you stop the syslogd service. You don't want the two of them stepping on each other. To test whether your remote logging is working, use the logger command:

 # logger This is a Test 

Then, on your log server, search (or grep) /var/log/messages (or wherever you have remote logs) for "This is a Test". If you get a response, congratulationseverything is working fine, and you now have encrypted remote logging!

9.11.6. Where Next?

While remote logging has always been a useful and even necessary process, sending valuable system information unencrypted across the void has long been a security risk. Thanks to syslog-ng and stunnel, we no longer have to worry about that. In addition, the flexibility of syslog-ng has moved leaps and bounds beyond what syslogd was ever capable of. It truly is the Next Generation of system logging daemons.

That flexibility comes with a price, thoughthe syslog-ng configuration file is a complex beast. If you spend a little time getting to know it, however, you'll find that it's not quite as hard as it looks. I can assure you that the complexity of the syntax is proportional to its adaptability once you understand it. Listed below are some resources you can consult online for help in configuring your syslog-ng instance to meet your needs.

9.11.7. See Also

• http://www.balabit.hu/static/syslog-ng/reference/book1.html

• http://www.stunnel.org/examples/syslog-ng.html

• "Fine-Tune the syslog Daemon" [Hack #86]

Brian Warshawsky