55 Rotating Log Files


#55 Rotating Log Files

Users who don't have much experience with Unix can be quite surprised by how many commands, utilities, and daemons log events to system log files. Even on a computer with lots of disk space, it's important to keep an eye on the size of these files and, of course, on their contents too.

As a result, most sysadmins have a set of instructions that they place at the top of their log file analysis utilities, similar to the following:

 mv $log.2 $log.3 mv $log.1 $log.2 mv $log $log.1 touch $log 

If run weekly, this would produce a rolling one-month archive of log file information divided into week-size portions of data. However, it's just as easy to create a script that accomplishes this for all log files in the /var/log directory at once, thereby relieving any log file analysis scripts of the burden .

The script steps through each file in the /var/log directory that matches a particular set of criteria, checking each matching file's rotation schedule and last-modified date to see if it's time for it to be rotated .

The Code

 #!/bin/sh # rotatelogs - Rolls logfiles in /var/log for archival purposes. #    Uses a config file to allow customization of how frequently #    each log should be rolled. The config file is in #       logfilename=duration #    format, where duration is in days. If, in the config #    file, an entry is missing for a particular logfilename, #    rotatelogs won't rotate the file more frequently than every seven days. logdir="/var/log" config="/var/log/rotatelogs.conf" mv="/bin/mv" default_duration=7     count=0 duration=$default_duration if [ ! -f $config ] ; then   echo " 
 #!/bin/sh # rotatelogs - Rolls logfiles in /var/log for archival purposes. # Uses a config file to allow customization of how frequently # each log should be rolled. The config file is in # logfilename=duration # format, where duration is in days. If, in the config # file, an entry is missing for a particular logfilename, # rotatelogs won't rotate the file more frequently than every seven days. logdir="/var/log" config="/var/log/rotatelogs.conf" mv="/bin/mv" default_duration=7 count=0 duration=$default_duration if [ ! -f $config ] ; then echo "$0: no config file found. Can't proceed." >&2; exit 1 fi if [ ! -w $logdir -o ! -x $logdir ] ; then echo "$0: you don't have the appropriate permissions in $logdir" >&2 exit 1 fi cd $logdir # While we'd like to use ':digit:' with the find, many versions of # find don't support POSIX character class identifiers, hence [0-9] for name in $(find . -type f -size +0c ! -name '*[0-9]*' \ ! -name '\.*' ! -name '*conf' -maxdepth 1 -print  sed 's/^\.\///') do count=$(( $count + 1 )) # Grab this entry from the config file duration="$(grep "^${name}=" $configcut -d= -f2)" if [ -z $duration ] ; then duration=$default_duration elif [ "$duration" = "0" ] ; then echo "Duration set to zero: skipping $name" continue fi back1="${name}.1"; back2="${name}.2"; back3="${name}.3"; back4="${name}.4"; # If the most recently rolled log file (back1) has been modified within # the specific quantum, then it's not time to rotate it. if [ -f "$back1" ] ; then if [ -z $(find \"$back1\" -mtime +$duration -print 2>/dev/null) ] then echo -n "$name's most recent backup is more recent than $duration " echo "days: skipping" ; continue fi fi echo "Rotating log $name (using a $duration day schedule)" # Rotate, starting with the oldest log if [ -f "$back3" ] ; then echo "... $back3 -> $back4" ; $mv -f "$back3" "$back4" fi if [ -f "$back2" ] ; then echo "... $back2 -> $back3" ; $mv -f "$back2" "$back3" fi if [ -f "$back1" ] ; then echo "... $back1 -> $back2" ; $mv -f "$back1" "$back2" fi if [ -f "$name" ] ; then echo "... $name -> $back1" ; $mv -f "$name" "$back1" fi touch "$name" chmod 0600 "$ name " done if [ $count -eq 0 ] ; then echo "Nothing to do: no log files big enough or old enough to rotate" fi exit 0 
: no config file found. Can't proceed." >&2; exit 1 fi if [ ! -w $logdir -o ! -x $logdir ] ; then echo "
 #!/bin/sh # rotatelogs - Rolls logfiles in /var/log for archival purposes. # Uses a config file to allow customization of how frequently # each log should be rolled. The config file is in # logfilename=duration # format, where duration is in days. If, in the config # file, an entry is missing for a particular logfilename, # rotatelogs won't rotate the file more frequently than every seven days. logdir="/var/log" config="/var/log/rotatelogs.conf" mv="/bin/mv" default_duration=7 count=0 duration=$default_duration if [ ! -f $config ] ; then echo "$0: no config file found. Can't proceed." >&2; exit 1 fi if [ ! -w $logdir -o ! -x $logdir ] ; then echo "$0: you don't have the appropriate permissions in $logdir" >&2 exit 1 fi cd $logdir # While we'd like to use ':digit:' with the find, many versions of # find don't support POSIX character class identifiers, hence [0-9] for name in $(find . -type f -size +0c ! -name '*[0-9]*' \ ! -name '\.*' ! -name '*conf' -maxdepth 1 -print  sed 's/^\.\///') do count=$(( $count + 1 )) # Grab this entry from the config file duration="$(grep "^${name}=" $configcut -d= -f2)" if [ -z $duration ] ; then duration=$default_duration elif [ "$duration" = "0" ] ; then echo "Duration set to zero: skipping $name" continue fi back1="${name}.1"; back2="${name}.2"; back3="${name}.3"; back4="${name}.4"; # If the most recently rolled log file (back1) has been modified within # the specific quantum, then it's not time to rotate it. if [ -f "$back1" ] ; then if [ -z $(find \"$back1\" -mtime +$duration -print 2>/dev/null) ] then echo -n "$name's most recent backup is more recent than $duration " echo "days: skipping" ; continue fi fi echo "Rotating log $name (using a $duration day schedule)" # Rotate, starting with the oldest log if [ -f "$back3" ] ; then echo "... $back3 -> $back4" ; $mv -f "$back3" "$back4" fi if [ -f "$back2" ] ; then echo "... $back2 -> $back3" ; $mv -f "$back2" "$back3" fi if [ -f "$back1" ] ; then echo "... $back1 -> $back2" ; $mv -f "$back1" "$back2" fi if [ -f "$name" ] ; then echo "... $name -> $back1" ; $mv -f "$name" "$back1" fi touch "$name" chmod 0600 "$ name " done if [ $count -eq 0 ] ; then echo "Nothing to do: no log files big enough or old enough to rotate" fi exit 0 
: you don't have the appropriate permissions in $logdir" >&2 exit 1 fi cd $logdir # While we'd like to use ':digit:' with the find, many versions of # find don't support POSIX character class identifiers, hence [0-9] for name in $(find . -type f -size +0c ! -name '*[0-9]*' \ ! -name '\.*' ! -name '*conf' -maxdepth 1 -print sed 's/^\.\///') do count=$(( $count + 1 )) # Grab this entry from the config file duration="$(grep "^${name}=" $configcut -d= -f2)" if [ -z $duration ] ; then duration=$default_duration elif [ "$duration" = "0" ] ; then echo "Duration set to zero: skipping $name" continue fi back1="${name}.1"; back2="${name}.2"; back3="${name}.3"; back4="${name}.4"; # If the most recently rolled log file (back1) has been modified within # the specific quantum, then it's not time to rotate it. if [ -f "$back1" ] ; then if [ -z $(find \"$back1\" -mtime +$duration -print 2>/dev/null) ] then echo -n "$name's most recent backup is more recent than $duration " echo "days: skipping" ; continue fi fi echo "Rotating log $name (using a $duration day schedule)" # Rotate, starting with the oldest log if [ -f "$back3" ] ; then echo "... $back3 -> $back4" ; $mv -f "$back3" "$back4" fi if [ -f "$back2" ] ; then echo "... $back2 -> $back3" ; $mv -f "$back2" "$back3" fi if [ -f "$back1" ] ; then echo "... $back1 -> $back2" ; $mv -f "$back1" "$back2" fi if [ -f "$name" ] ; then echo "... $name -> $back1" ; $mv -f "$name" "$back1" fi touch "$name" chmod 0600 "$name" done if [ $count -eq 0 ] ; then echo "Nothing to do: no log files big enough or old enough to rotate" fi exit 0

To truly be useful, the script needs to work with a configuration file that lives in /var/log , which allows different log files to be set to different rotation schedules. The contents of a typical configuration file are as follows :

 # Configuration file for the log rotation script. # Format is    name=duration    where 'name' can be any # filename that appears in the /var/log directory. Duration # is measured in days. ftp.log=30 lastlog=14 lookupd.log=7 lpr.log=30 mail.log=7 netinfo.log=7 secure.log=7 statistics=7 system.log=14 # Anything with a duration of zero is not rotated wtmp=0 

How It Works

The heart of this script is the find statement:

 for name in $(find . -type f -size +0c ! -name '*[0-9]*' \      ! -name '\.*' ! -name '*conf' -maxdepth 1 -print  sed 's/^\.\///') 

This creates a loop, returning all files in the /var/log directory that are greater than 0 characters in size, don't contain a number in their name, don't start with a period (Mac OS X in particular dumps a lot of oddly named log files in this directory; they all need to be skipped ), and don't end with the word "conf" (we don't want to rotate out the rotatelogs.conf file, for obvious reasons!). The maxdepth 1 ensures that find doesn't step into subdirectories. Finally, the sed invocation removes any leading ./ sequences.

Lazy is good!  

The rotatelogs script demonstrates a fundamental concept in shell script programming: the value of avoiding duplicate work. Rather than have each log analysis script rotate logs, a single log rotation script centralizes the task and makes modifications easy.

The Results

 $  sudo rotatelogs  ftp.log's most recent backup is more recent than 30 days: skipping Rotating log lastlog (using a 14 day schedule) ... lastlog -> lastlog.1 lpr.log's most recent backup is more recent than 30 days: skipping 

Notice that of all the log files in /var/log , only three matched the specified find criteria, and of those only one, lastlog , hadn't been backed up sufficiently recently, according to the duration values in the configuration file shown earlier.

Hacking the Script

One example of how this script could be even more useful is to have the oldest archive file, the old $back4 file, emailed to a central storage site before it's over-written by the mv command in the following statement:

 echo "... $back3 -> $back4" ; $mv -f "$back3" "$back4" 

Another useful enhancement to rotatelogs would be to compress all rotated logs to further save on disk space, which would also require that the script recognize and work properly with compressed files as it proceeded.




Wicked Cool Shell Scripts. 101 Scripts for Linux, Mac OS X, and Unix Systems
Wicked Cool Shell Scripts
ISBN: 1593270127
EAN: 2147483647
Year: 2004
Pages: 150
Authors: Dave Taylor

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net