89 Tracking FTP Usage


#89 Tracking FTP Usage

If you're running an anonymous FTP server, you should already be constantly monitoring what happens in the ~ftp/pub directory (which is usually where uploads are allowed), but any FTP server requires you to keep an eye on things.

The ftp daemon's transfer log ( xferlog ) file format is definitely one of the most cryptic in Unix, which makes analyzing it in a script rather tricky. Worse, there's a standard, common xferlog file format that just about everyone uses (and which this script expects), and there's an abbreviated ftpd .log format that some BSD versions of ftpd use that's just about impossible to analyze in a script.

So we'll focus on the xferlog format. The columns in an xferlog are as shown in Table 10-2.

Table 10-2: Field values in the xferlog file

Column

Value

1 “5

Current time

6

Transfer time (secs)

7

Remote host

8

File size

9

Filename

10

Transfer type

11

Special action flag

12

Direction

13

Access mode

14

Username

15

Service name

16

Authentication method

17

Authenticated user ID

18-?

Additional codes as added by the specific fptd program (usually omitted)

A sample line from an xferlog is as cryptic as you might expect:

 Mon Nov  4 12:22:46 2002 2 192.168.124.152 2170570 \ /home/ftp/pub/openssl-0.9.5r.tar.gz b _ i r leoftp 0 * c 

This script quickly scans through xferlog , highlighting connections and files uploaded and downloaded, and producing other useful statistics.

The Code

 #!/bin/sh # xferlog - Analyzes and summarizes the FTP transfer log. A good doc # detailing the log format is http://aolserver.am.net/docs/2.3/ftp-ch4.htm. stdxferlog="/var/log/xferlog" temp="/tmp/$(basename 
 #!/bin/sh # xferlog - Analyzes and summarizes the FTP transfer log. A good doc # detailing the log format is http://aolserver.am.net/docs/2.3/ftp-ch4.htm. stdxferlog="/var/log/xferlog" temp="/tmp/$(basename $0).$$" nicenum="$HOME/bin/nicenumber" # Script #4 trap "/bin/rm -f $temp" 0 extract() { # Called with $1 = desired accessmode, $2 = section name for output if [ ! -z "$(echo $accessmode  grep $1)" ] ; then echo "" ; echo "$2" if [ "$1" = "a" -o "$1" = "g" ] ; then echo " common account (entered password) values:" else echo " user accounts accessing server: " fi awk "\$13 == \"$1\" { print \$14 }" $log  sort  \ uniq -c  sort -rn  head -10  sed 's/^/ /' awk "\$13 == \"$1\" && \$12 == \"o\" { print \$9 }" $log  sort  \ uniq -c  sort -rn  head -10  sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files downloaded from server:" ; cat $temp fi awk "\$13 == \"$1\" && \$12 == \"i\" { print \$9 }" $log  sort  \ uniq -c  sort -rn  head -10  sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files uploaded to server:" ; cat $temp fi fi } ###### The main script block case $# in 0 ) log=$stdxferlog ;; 1 ) log="$1" ;; * ) echo "Usage: $(basename $0) {xferlog name}" >&2 exit 1 esac if [ ! -r $log ] ; then echo "$(basename $0): can't read $log." >&2 exit 1 fi # Ascertain whether it's an abbreviated or standard ftp log file format. If # it's the abbreviated format, output some minimal statistical data and quit: # The abbreviated format is too difficult to analyze in a short script, # unfortunately . if [ ! -z $(awk '$6 == "get" { short=1 } END{ print short }' $log) ] ; then bytesin="$(awk 'BEGIN{sum=0} $6=="get" {sum+=$9} END{print sum}' $log)" bytesout="$(awk 'BEGIN{sum=0} $6=="put" {sum+=$9} END{print sum}' $log)" echo -n "Abbreviated ftpd xferlog from " echo -n $(head -1 $log  awk '{print $1, $2, $3 }') echo " to $(tail -1 $log  awk '{print $1, $2, $3}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" exit 0 fi bytesin="$(awk 'BEGIN{sum=0} $12=="i" {sum += $8} END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} $12=="o" {sum += $8} END{ print sum }' $log )" time="$(awk 'BEGIN{sum=0} {sum += $6} END{ print sum }' $log)" echo -n "Summary of xferlog from " echo -n $(head -1 $log  awk '{print $1, $2, $3, $4, $5 }') echo " to $(tail -1 $log  awk '{print $1, $2, $3, $4, $5}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" echo " transfer time: $time seconds" accessmode="$(awk '{print $13}' $log sort -u)" extract "a" "Anonymous Access" extract "g" "Guest Account Access" extract "r" "Real User Account Access" exit 0 
).$$" nicenum="$HOME/bin/nicenumber" # Script #4 trap "/bin/rm -f $temp" 0 extract() { # Called with = desired accessmode, = section name for output if [ ! -z "$(echo $accessmode grep )" ] ; then echo "" ; echo "" if [ "" = "a" -o "" = "g" ] ; then echo " common account (entered password) values:" else echo " user accounts accessing server: " fi awk "$13 == \"\" { print $14 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' awk "$13 == \"\" && $12 == \"o\" { print $9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files downloaded from server:" ; cat $temp fi awk "$13 == \"\" && $12 == \"i\" { print $9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files uploaded to server:" ; cat $temp fi fi } ###### The main script block case $# in 0 ) log=$stdxferlog ;; 1 ) log="" ;; * ) echo "Usage: $(basename
 #!/bin/sh # xferlog - Analyzes and summarizes the FTP transfer log. A good doc # detailing the log format is http://aolserver.am.net/docs/2.3/ftp-ch4.htm. stdxferlog="/var/log/xferlog" temp="/tmp/$(basename $0).$$" nicenum="$HOME/bin/nicenumber" # Script #4 trap "/bin/rm -f $temp" 0 extract() { # Called with $1 = desired accessmode, $2 = section name for output if [ ! -z "$(echo $accessmode  grep $1)" ] ; then echo "" ; echo "$2" if [ "$1" = "a" -o "$1" = "g" ] ; then echo " common account (entered password) values:" else echo " user accounts accessing server: " fi awk "\$13 == \"$1\" { print \$14 }" $log  sort  \ uniq -c  sort -rn  head -10  sed 's/^/ /' awk "\$13 == \"$1\" && \$12 == \"o\" { print \$9 }" $log  sort  \ uniq -c  sort -rn  head -10  sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files downloaded from server:" ; cat $temp fi awk "\$13 == \"$1\" && \$12 == \"i\" { print \$9 }" $log  sort  \ uniq -c  sort -rn  head -10  sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files uploaded to server:" ; cat $temp fi fi } ###### The main script block case $# in 0 ) log=$stdxferlog ;; 1 ) log="$1" ;; * ) echo "Usage: $(basename $0) {xferlog name}" >&2 exit 1 esac if [ ! -r $log ] ; then echo "$(basename $0): can't read $log." >&2 exit 1 fi # Ascertain whether it's an abbreviated or standard ftp log file format. If # it's the abbreviated format, output some minimal statistical data and quit: # The abbreviated format is too difficult to analyze in a short script, # unfortunately . if [ ! -z $(awk '$6 == "get" { short=1 } END{ print short }' $log) ] ; then bytesin="$(awk 'BEGIN{sum=0} $6=="get" {sum+=$9} END{print sum}' $log)" bytesout="$(awk 'BEGIN{sum=0} $6=="put" {sum+=$9} END{print sum}' $log)" echo -n "Abbreviated ftpd xferlog from " echo -n $(head -1 $log  awk '{print $1, $2, $3 }') echo " to $(tail -1 $log  awk '{print $1, $2, $3}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" exit 0 fi bytesin="$(awk 'BEGIN{sum=0} $12=="i" {sum += $8} END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} $12=="o" {sum += $8} END{ print sum }' $log )" time="$(awk 'BEGIN{sum=0} {sum += $6} END{ print sum }' $log)" echo -n "Summary of xferlog from " echo -n $(head -1 $log  awk '{print $1, $2, $3, $4, $5 }') echo " to $(tail -1 $log  awk '{print $1, $2, $3, $4, $5}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" echo " transfer time: $time seconds" accessmode="$(awk '{print $13}' $log sort -u)" extract "a" "Anonymous Access" extract "g" "Guest Account Access" extract "r" "Real User Account Access" exit 0 
) {xferlog name}" >&2 exit 1 esac if [ ! -r $log ] ; then echo "$(basename
 #!/bin/sh # xferlog - Analyzes and summarizes the FTP transfer log. A good doc # detailing the log format is http://aolserver.am.net/docs/2.3/ftp-ch4.htm. stdxferlog="/var/log/xferlog" temp="/tmp/$(basename $0).$$" nicenum="$HOME/bin/nicenumber" # Script #4 trap "/bin/rm -f $temp" 0 extract() { # Called with $1 = desired accessmode, $2 = section name for output if [ ! -z "$(echo $accessmode  grep $1)" ] ; then echo "" ; echo "$2" if [ "$1" = "a" -o "$1" = "g" ] ; then echo " common account (entered password) values:" else echo " user accounts accessing server: " fi awk "\$13 == \"$1\" { print \$14 }" $log  sort  \ uniq -c  sort -rn  head -10  sed 's/^/ /' awk "\$13 == \"$1\" && \$12 == \"o\" { print \$9 }" $log  sort  \ uniq -c  sort -rn  head -10  sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files downloaded from server:" ; cat $temp fi awk "\$13 == \"$1\" && \$12 == \"i\" { print \$9 }" $log  sort  \ uniq -c  sort -rn  head -10  sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files uploaded to server:" ; cat $temp fi fi } ###### The main script block case $# in 0 ) log=$stdxferlog ;; 1 ) log="$1" ;; * ) echo "Usage: $(basename $0) {xferlog name}" >&2 exit 1 esac if [ ! -r $log ] ; then echo "$(basename $0): can't read $log." >&2 exit 1 fi # Ascertain whether it's an abbreviated or standard ftp log file format. If # it's the abbreviated format, output some minimal statistical data and quit: # The abbreviated format is too difficult to analyze in a short script, # unfortunately . if [ ! -z $(awk '$6 == "get" { short=1 } END{ print short }' $log) ] ; then bytesin="$(awk 'BEGIN{sum=0} $6=="get" {sum+=$9} END{print sum}' $log)" bytesout="$(awk 'BEGIN{sum=0} $6=="put" {sum+=$9} END{print sum}' $log)" echo -n "Abbreviated ftpd xferlog from " echo -n $(head -1 $log  awk '{print $1, $2, $3 }') echo " to $(tail -1 $log  awk '{print $1, $2, $3}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" exit 0 fi bytesin="$(awk 'BEGIN{sum=0} $12=="i" {sum += $8} END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} $12=="o" {sum += $8} END{ print sum }' $log )" time="$(awk 'BEGIN{sum=0} {sum += $6} END{ print sum }' $log)" echo -n "Summary of xferlog from " echo -n $(head -1 $log  awk '{print $1, $2, $3, $4, $5 }') echo " to $(tail -1 $log  awk '{print $1, $2, $3, $4, $5}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" echo " transfer time: $time seconds" accessmode="$(awk '{print $13}' $log sort -u)" extract "a" "Anonymous Access" extract "g" "Guest Account Access" extract "r" "Real User Account Access" exit 0 
): can't read $log." >&2 exit 1 fi # Ascertain whether it's an abbreviated or standard ftp log file format. If # it's the abbreviated format, output some minimal statistical data and quit: # The abbreviated format is too difficult to analyze in a short script, # unfortunately. if [ ! -z $(awk ' == "get" { short=1 } END{ print short }' $log) ] ; then bytesin="$(awk 'BEGIN{sum=0} =="get" {sum+=} END{print sum}' $log)" bytesout="$(awk 'BEGIN{sum=0} =="put" {sum+=} END{print sum}' $log)" echo -n "Abbreviated ftpd xferlog from " echo -n $(head -1 $log awk '{print , , }') echo " to $(tail -1 $log awk '{print , , }')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" exit 0 fi bytesin="$(awk 'BEGIN{sum=0} =="i" {sum += } END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} =="o" {sum += } END{ print sum }' $log )" time="$(awk 'BEGIN{sum=0} {sum += } END{ print sum }' $log)" echo -n "Summary of xferlog from " echo -n $(head -1 $log awk '{print , , , , }') echo " to $(tail -1 $log awk '{print , , , , }')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" echo " transfer time: $time seconds" accessmode="$(awk '{print }' $log sort -u)" extract "a" "Anonymous Access" extract "g" "Guest Account Access" extract "r" "Real User Account Access" exit 0

How It Works

In an xferlog , the total number of incoming bytes can be calculated by extracting just those lines that have direction="i" and then summing up the eighth column of data. Outgoing bytes are in the same column, but for direction="o" .

 bytesin="$(awk 'BEGIN{sum=0} =="i" {sum += } END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} =="o" {sum += } END{ print sum }' $log )" 

Ironically, the slower the network connection, the more accurate the total connection time is. On a fast network, smaller transfers are logged as taking zero seconds, though clearly every transfer that succeeds must be longer than that.

Three types of access mode are possible: a is anonymous, g is for users who utilize the guest account (usually password protected), and r is for real or regular users. In the case of anonymous and guest users, the account value (field 14) is the user's password. People connecting anonymously are requested by their FTP program to specify their email address as their password, which is then logged and can be analyzed .

Of this entire xferlog output stream, the most important entries are those with an anonymous access mode and a direction of i , indicating that the entry is an upload listing. If you have allowed anonymous connections and have either deliberately or accidentally left a directory writable, these anonymous upload entries are where you'll be able to see if skript kiddies, warez hackers, and other characters of ill repute are exploiting your system. If such an entry lists a file uploaded to your server, it needs to be checked out immediately, even if the file-name seems quite innocuous .

This test occurs in the following statement in the extract function:

 awk "$13 == \"\" && $12 == \"i\" { print $9 }" $log  sort  \   uniq -c  sort -rn  head -10  sed 's/^/    /' > $temp 

In this rather complex awk invocation, we're checking to see whether field 13 matches the anonymous account code (because extract is called as extract "a" "Anonymous Access" ) and whether field 12 indicates that it's an upload with the code i . If both of these conditions are true, we process the value of field 9, which is the name of the file uploaded.

If you're running an FTP server, this is definitely a script for a weekly (or even daily) cron job.

Running the Script

If invoked without any arguments, this script tries to read and analyze the standard ftpd transfer log /var/log/xferlog . If that's not the correct log file, a different filename can be specified on the command line.

The Results

The results depend on the format of the transfer log the script is given. If it's an abbreviated form, some minimal statistics are generated and the script quits:

 $  xferlog  succinct.xferlog  Abbreviated ftpd xferlog from Aug 1 04:20:11 to Sep 1 04:07:41        bytes in: 215,300,253        bytes out: 30,305,090 

When a full xferlog in standard format is encountered , considerably more information can be obtained and displayed by the script:

 $  xferlog  Summary of xferlog from Mon Sep 1 5:03:11 2003 to Tue Sep 30 17:38:50 2003        bytes in: 675,840       bytes out: 3,989,488   transfer time: 11 seconds Anonymous Access   common account (entered password) values:        1 taylor@intuitive.com        1 john@doe   files downloaded from server:        1 /MySubscriptions.opml   files uploaded to server:        1 /tmp/Find.Warez.txt Real User Account Access   user accounts accessing server:        7 rufus        2 taylor   files downloaded from server:        7 /pub/AllFiles.tgz        2 /pub/AllFiles.tar 

Security Alert! Did you notice that someone using anonymous FTP has uploaded a file called /tmp/Find.Warez.txt ? "Warez" are illegal copies of licensed software ” not something you want on your server. Upon seeing this, I immediately went into my FTP archive and deleted the file.




Wicked Cool Shell Scripts. 101 Scripts for Linux, Mac OS X, and Unix Systems
Wicked Cool Shell Scripts
ISBN: 1593270127
EAN: 2147483647
Year: 2004
Pages: 150
Authors: Dave Taylor

Similar book on Amazon

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net