11.7. The Logs Are Overloading My Hard Drive
Logs can grow quickly, especially for services with a lot of activity. Logs for commercial web sites can easily add several hundred megabytes of files every day.
Unmanaged, this kind of growth can overwhelm your system, taking space needed by your users, occupying the empty space required to run a GUI, and making it impossible to boot your system.
If your logs grow quickly, you should consider creating dedicated partitions. Even with dedicated partitions and search scripts of dazzling sophistication, it can take quite a while to search through large logfiles for the data you may need. Therefore, you may consider configuring your system to start new logfiles more often, perhaps daily or even hourly.
Even large, dedicated partitions may not be good enough. The demands of logfiles can grow so large that you may need to move logfiles to different systems.
The associated cron jobs are run in alphabetical order; files starting with numbers come first. For example, the 00logwatch script in /etc/cron.daily is run before others.
11.7.1. Logfile Partitions
Logs can become quite large, and can easily grow by hundreds of megabytes of space (or more) every day. There are two basic options in this regard:
For most organizations, the data associated with logs isn't nearly as critical as, say, that associated with user home directories. Because logs grow quickly, one method to manage this growth is a RAID 0 volume with daily backups.
RAID 0 is the fastest possible media for large files and may be suitable for a log server. With appropriate controllers, it allows you to add more disks as logs grow.
Your management may have different feelings about the importance of logfiles. Perhaps you'll want to protect them as evidence, to help you track the activity of certain users, to establish patterns of visits to your web sites, or possibly even as evidence usable in a court of law. If logfiles are that important, you may want to use a more robust data storage system, such as RAID 5, or even back them up to stable archives such as DVDs.
11.7.2. Log Rotation Frequency
Log rotation means starting new files to contain incoming log messages, so that old logs can easily be backed up. Rotation also involves removing old messages to return needed disk space to the system. In Linux, log rotation is configured through the logrotate configuration files. In our preferred distributions, these files are stored in /etc/logrotate.conf. To understand how this process works, it's useful to analyze it in detail.
Every day, on a schedule defined by your /etc/crontab configuration file, Linux runs the /etc/cron.daily/logrotate script. It includes the following command, which runs the logrotate service, based on the settings in /etc/logrotate.conf:
To see how rotation works, we can analyze /etc/logrotate.conf. The first four default commands in this file are identical in our preferred distributions:
In some cases, the distribution developers have configured rotation of the wtmp and btmp access logs in /var/log, as they are not associated with any specific package, nor are they maintained by any of the /etc/logrotate.d configuration files.
If you add the following directive, you can enable compression of your logfiles, saving even more room:
Compression still allows access by some dedicated logfile viewers and editors, including vi. There are a substantial number of options available; Sourceforge.net includes several hundred log-management suites, many of which can even search through directories of compressed logfiles.
11.7.3. Deleting Old Logs
Logs should normally be deleted automatically. However, if you see logs more than five weeks old, that suggests a problem with your logrotate script, or perhaps that your cron jobs aren't being run as scheduled.
For example, on my newer laptop computer, I haven't configured my winmodem to allow external logins by modem. I have no modem getty (mgetty) logs in my /var/log directory. When I run the daily cron logrotate script, I get a related error:
# /etc/cron.daily/logrotate error: error accessing /var/log/mgetty: No such file or directory error: mgetty:1 glob failed for /var/log/mgetty/*.log
There are several ways to address this issue. I could configure mgetty, but that would be a waste of time. I could delete the mgetty configuration file in /etc/logrotate.d, but that would cause more problems if I choose to configure it in the future. The option I chose was to create the /var/log/mgetty directory, as the root user. After creating that directory, I ran the logrotate script again, without errors.
I also ran touch .placeholder in that directory, to make sure the directory wouldn't get deleted at the next update.
11.7.4. Create Jobs to Move Logs
If you've made some of the changes suggested in "So Many Server Logs," earlier in this chapter, you may have already sent your logs to remote systems. In short, you'll need to configure the System Log daemon on the log server to receive remote logs, and configure the other computers to send their logs to that log server. See the previous annoyance for details.