Section 11.7. The Logs Are Overloading My Hard Drive

11.7. The Logs Are Overloading My Hard Drive

Logs can grow quickly, especially for services with a lot of activity. Logs for commercial web sites can easily add several hundred megabytes of files every day.

Unmanaged, this kind of growth can overwhelm your system, taking space needed by your users, occupying the empty space required to run a GUI, and making it impossible to boot your system.

If your logs grow quickly, you should consider creating dedicated partitions. Even with dedicated partitions and search scripts of dazzling sophistication, it can take quite a while to search through large logfiles for the data you may need. Therefore, you may consider configuring your system to start new logfiles more often, perhaps daily or even hourly.

Even large, dedicated partitions may not be good enough. The demands of logfiles can grow so large that you may need to move logfiles to different systems.

The associated cron jobs are run in alphabetical order; files starting with numbers come first. For example, the 00logwatch script in /etc/cron.daily is run before others.

If you have more than one log-management service installed, such as logrotate or logwatch, the associated jobs may not be fully compatible.

11.7.1. Logfile Partitions

Logs can become quite large, and can easily grow by hundreds of megabytes of space (or more) every day. There are two basic options in this regard:

A dedicated log partition

With a dedicated log partition, the space taken by a service or kernel log doesn't overwhelm the space required to run a Linux system. If you use a standard Linux distribution, the way to set this up is to mount the /var/log directory on a dedicated partition.

A dedicated log server

Even if you configure a dedicated server to collect logs from other Linux systems, I still recommend a dedicated log partition.

For most organizations, the data associated with logs isn't nearly as critical as, say, that associated with user home directories. Because logs grow quickly, one method to manage this growth is a RAID 0 volume with daily backups.

RAID 0 is the fastest possible media for large files and may be suitable for a log server. With appropriate controllers, it allows you to add more disks as logs grow.

Your management may have different feelings about the importance of logfiles. Perhaps you'll want to protect them as evidence, to help you track the activity of certain users, to establish patterns of visits to your web sites, or possibly even as evidence usable in a court of law. If logfiles are that important, you may want to use a more robust data storage system, such as RAID 5, or even back them up to stable archives such as DVDs.

11.7.2. Log Rotation Frequency

Log rotation means starting new files to contain incoming log messages, so that old logs can easily be backed up. Rotation also involves removing old messages to return needed disk space to the system. In Linux, log rotation is configured through the logrotate configuration files. In our preferred distributions, these files are stored in /etc/logrotate.conf. To understand how this process works, it's useful to analyze it in detail.

Every day, on a schedule defined by your /etc/crontab configuration file, Linux runs the /etc/cron.daily/logrotate script. It includes the following command, which runs the logrotate service, based on the settings in /etc/logrotate.conf:

 /usr/sbin/logrotate /etc/logrotate.conf 

To see how rotation works, we can analyze /etc/logrotate.conf. The first four default commands in this file are identical in our preferred distributions:

weekly

Logfiles are rotated by default on a weekly basis. You can set this to daily or monthly, or specify a maximum log size after which rotation occurs with a command such as:

 size=100k 

rotate 4

Linux distributions normally store four weeks of backlogs for each service.

create

A new, empty file is created in place of the logfile that is now rotated.

include /etc/logrotate.d

Configuration files in the /etc/logrotate.d directory are included in the rotation process.

In some cases, the distribution developers have configured rotation of the wtmp and btmp access logs in /var/log, as they are not associated with any specific package, nor are they maintained by any of the /etc/logrotate.d configuration files.

If you add the following directive, you can enable compression of your logfiles, saving even more room:

 compress 

Compression still allows access by some dedicated logfile viewers and editors, including vi. There are a substantial number of options available; Sourceforge.net includes several hundred log-management suites, many of which can even search through directories of compressed logfiles.

11.7.3. Deleting Old Logs

Logs should normally be deleted automatically. However, if you see logs more than five weeks old, that suggests a problem with your logrotate script, or perhaps that your cron jobs aren't being run as scheduled.

For example, on my newer laptop computer, I haven't configured my winmodem to allow external logins by modem. I have no modem getty (mgetty) logs in my /var/log directory. When I run the daily cron logrotate script, I get a related error:

 # /etc/cron.daily/logrotate error: error accessing /var/log/mgetty: No such file or directory error: mgetty:1 glob failed for /var/log/mgetty/*.log 

There are several ways to address this issue. I could configure mgetty, but that would be a waste of time. I could delete the mgetty configuration file in /etc/logrotate.d, but that would cause more problems if I choose to configure it in the future. The option I chose was to create the /var/log/mgetty directory, as the root user. After creating that directory, I ran the logrotate script again, without errors.

I also ran touch .placeholder in that directory, to make sure the directory wouldn't get deleted at the next update.

11.7.4. Create Jobs to Move Logs

If you've made some of the changes suggested in "So Many Server Logs," earlier in this chapter, you may have already sent your logs to remote systems. In short, you'll need to configure the System Log daemon on the log server to receive remote logs, and configure the other computers to send their logs to that log server. See the previous annoyance for details.