The Implementation

Team-Fly    

Solaris™ Operating Environment Boot Camp
By David Rhodes, Dominic Butler
Table of Contents
Chapter 6.  The Filesystem and Its Contents


For the example servers within this book, we will be sticking to a standard design for filesystems on all our servers. This greatly simplifies the management of the servers and also simplifies the process of documenting them. In a large company with many Solaris servers, it is of great benefit to design a standard set of filesystems and use this design every time a new server is built. It means the build process is simpler and backups from one system can more easily be used to help recover from an emergency situation on another server. It also means that the support staff know what to expect and where to look when they get problems on any of their servers. A good idea is to define and document naming standards and stick to these on all servers.

For our servers we will use the information in Table 6.11 as our standard.

Table 6.11. Standard Disk Layout

Filesystem Name

Mountpoint

Device

root

/

c0t2d0s0

usr

/usr

c0t2d0s6

var

/var

c0t2d0s1

opt

/opt

c0t2d0s5

home

/export/home

c0t2d0s7

tmp

/tmp

swap (c0t2d0s3)

system_tools

/usr/local/utils

c0t2d0s4

The filesystem mounted on /usr/local/utils is used to hold our own scripts that help us manage and support the servers. Among the scripts included are one that will housekeep log files to prevent them from filling up filesystems and another that can check how full each filesystem is and send a mail message to a specified user if any are getting too full. As long as the mail gets checked regularly, we should have time to do something before the filesystem actually fills up.

Housekeeping the Log Files

We could have written a script to clear out log files with the names of the logs hard coded in the script. But this is not a good practice, since it means that if one system has different log files on it you need to edit the script itself. This can lead to problems in the future since when you roll out a new version of the script to all your systems you will lose the changes you made and you then have to spend time managing the scripts rather than the systems. To get around this problem, we use a separate file to hold the names of each log file to be housekept along with the rule for housekeeping it. The script reads the information from this file so we can have the same version of the script on all servers; if we improve it later, we can deliver it to all the servers knowing we will not overwrite any local change.

From examining the log files in the previous section, we can see that there are at least three different ways we might want to housekeep a log file. We might want to empty it, remove some old lines in it, or actually delete it. We have designed the file that holds the log file names so that we can specify what we want to do to each file to housekeep it. Instead of creating an action that simply deletes a log file, we have decided to call the action "move." It will cause the specified log file to be backed up before emptying it, so that we can still access the information in the backup copy. Next time the script runs, the backup copy will be overwritten with the log file's current contents. Here is an example of the housekeeping configuration file:

 # This file is used by the script tidyLogs. # It holds the name of each log to tidy and rules # on how to tidy it. # The format is: # name_of_file:action(zero,move,reduce):lines_to_keep # (only applies to reduce) /var/adm/wtmpx:zero: /var/adm/wtmp:move: /var/adm/sulog:reduce:100 

The file contains one log file per line and the fields are colon-separated, as many other Solaris system and configuration files are (e.g., /etc/passwd).

This example contains one line for each type of housekeeping we perform. The first entry is for the wtmpx file. We aren't interested in using the information stored in here so we have chosen to empty the file each time the tidyLogs housekeeping script runs. For the wtmp file, we will opt to back up the file before emptying it, and for the sulog file we will keep the 100 most recent entries.

The script that reads this file and actually keeps the log files in check is shown here in its entirety:

 #!/bin/ksh # shell script to be called from the root crontab that will tidy # log files using the file /usr/local/utils/logsToTidy # to provide the log filename and the action to take. # logCfg=/usr/local/utils/logsToTidy tidyLog=/usr/local/utils/logs/tidyLog.log prog=$(basename $0) if [[ ! -f "$logCfg" ]] then   echo "$0: cannot find $logCfg"   exit 1 else   for file in $(grep -v '^#' $logCfg | awk -F: '{print $1}')   do     action=$(grep "^$file:" $logCfg | awk -F: '{print $2}')     case $action in       zero) cp /dev/null $file             echo "$prog:$(date):$file emptied" >>$tidyLog             ;;       move) cp $file ${file}.old             cp /dev/null $file             echo "$prog:$(date):$file backed up and emptied" >>$tidyLog             ;;       reduce) keep=$(grep "^$file:" $logCfg | awk -F: '{print $3}')               lines=$(cat $file | wc -l)               if [[ "$lines" -gt "$keep" ]]               then                 cp $file /tmp/tidy.$$                 tail -$keep /tmp/tidy.$$ >$file                 rm -f /tmp/tidy.$$                 echo "$prog:$(date):$file reduced to $keep lines"                      >>$tidyLog               fi               ;;     esac   done fi 

This script is fairly basic, but you could use it as a starting point to build a script capable of more complex housekeeping tasks. For example, you could amend it to cycle a log file, so the current log gets moved to logname.1, logname.1 gets moved to logname.2, and so forth. Of course, you would need to move the current one last to avoid overwriting the older logs before you have moved them. You could then define an extra action for this in the logsToTidy configuration file.

Another improvement to the script would be to make it handle wildcards in the logsToTidy file. With a bit more judicial coding, you could end up with a script that can perform actions as complicated as deleting all log files in a directory that are older than 30 days, for example.

We have chosen to run this script once a month. If we ran it more often, the log files might not always have enough information in them to help us do our job; if we ran it less frequently, there is a chance that they might fill up a filesystem in between runs. The following cron entry will run the script on the first day of each month at 07:15 in the morning:

 15 7 1 * * /usr/local/utils/bin/tidyLogs >/dev/null 2>&1 

The part of the entry from the "greater than" symbol to the end of the line ensures that any output from the script is redirected to a special file called /dev/null. This is usually added to all cron entries because cron always mails the output of any jobs it runs to the owner of the cron file, and this tends to just fill the user's mail file with unwanted mail. The process of redirection is covered in Chapter 5, "Shells."

Checking Filesystem Usage

The script shown below is intended to be executed from the root crontab file at a regular interval. It checks each filesystem's percentage of fullness and alerts if one is greater than 90 percent full.

 #!/bin/ksh # simple shell script to be called from the root crontab that will # check the usage of each filesystem and mail an alert if any are # more than 90% full # mailWarn=/usr/local/utils/mailWarnings fswarnLog=/usr/local/utils/logs/fswarn.log mailProg=mailx alarm=90 prog=$(basename $0) # alerts will be mailed to the mail list/user specified in $mailWarn if # it exists, otherwise mail to root if [[ -f "$mailWarn" ]] then   mailTo=$(cat ${mailWarn} | grep -v "^#") else   mailTo=root fi df -k | while read line do   if echo $line | grep '^Filesystem' >/dev/null   then     continue   fi   set $line   pcntFull=$(echo $5 | sed 's/%//')   fs=$6   if [[ "$pcntFull" -gt "$alarm" ]]   then     # send warning to logfile and mail to the administrator     echo "WARNING: $fs is $pcntfull percent full at $(date)"          >> $fswarnLog     echo "WARNING: $fs is $pcntfull percent full at $(date)"          | $mailProg $mailTo   fi done 

There are two ways you could improve upon this script. Some filesystems (such as /usr) are fairly static, so you may not care if they are more than 90 percent full since there is little chance of them filling up quickly. Also, if it is a very large filesystem, even if it is 95 percent full there could still be over 100 MB free. So one improvement you could make would be to create a configuration file that contained the name of the filesystem and the percentage it needs to be full before it alerts. The entry for /usr might read:

 /usr:98 

The other improvement you could make would be to check how many inodes are free. Remember, if you run out of inodes you cannot create any more files in that filesystem no matter how much free space there is. Running the df command with the "-e" option will display the number of free inodes for each filesystem.

The script is run by the following cron entry in the root crontab file:

 0,15,30,45 * * * * /usr/local/utils/bin/fswarn >/dev/null 2>&1 

This will cause the script to run every 15 minutes to ensure that when a filesystem reaches the 90 percent threshold we have a chance of doing something before it actually reaches 100 percent. If your system is heavily loaded and you do not wish this script to interfere as much, it could be run every half-hour or even every hour.


    Team-Fly    
    Top
     



    Solaris Operating Environment Boot Camp
    Solaris Operating Environment Boot Camp
    ISBN: 0130342874
    EAN: 2147483647
    Year: 2002
    Pages: 301

    flylib.com © 2008-2017.
    If you may any questions please contact us: flylib@qtcs.net