89 Tracking FTP Usage
#89 Tracking FTP Usage
If you're running an anonymous FTP server, you should already be constantly monitoring what happens in the ~ftp/pub directory (which is usually where uploads are allowed), but any FTP server requires you to keep an eye on things.
The ftp daemon's transfer log ( xferlog ) file format is definitely one of the most cryptic in Unix, which makes analyzing it in a script rather tricky. Worse, there's a standard, common xferlog file format that just about everyone uses (and which this script expects), and there's an abbreviated ftpd .log format that some BSD versions of ftpd use that's just about impossible to analyze in a script.
So we'll focus on the xferlog format. The columns in an xferlog are as shown in Table 10-2.
| Column | Value |
|---|---|
| 1 “5 | Current time |
| 6 | Transfer time (secs) |
| 7 | Remote host |
| 8 | File size |
| 9 | Filename |
| 10 | Transfer type |
| 11 | Special action flag |
| 12 | Direction |
| 13 | Access mode |
| 14 | Username |
| 15 | Service name |
| 16 | Authentication method |
| 17 | Authenticated user ID |
| 18-? | Additional codes as added by the specific fptd program (usually omitted) |
A sample line from an xferlog is as cryptic as you might expect:
Mon Nov 4 12:22:46 2002 2 192.168.124.152 2170570 \ /home/ftp/pub/openssl-0.9.5r.tar.gz b _ i r leoftp 0 * c
This script quickly scans through xferlog , highlighting connections and files uploaded and downloaded, and producing other useful statistics.
The Code
#!/bin/sh # xferlog - Analyzes and summarizes the FTP transfer log. A good doc # detailing the log format is http://aolserver.am.net/docs/2.3/ftp-ch4.htm. stdxferlog="/var/log/xferlog" temp="/tmp/$(basename#!/bin/sh # xferlog - Analyzes and summarizes the FTP transfer log. A good doc # detailing the log format is http://aolserver.am.net/docs/2.3/ftp-ch4.htm. stdxferlog="/var/log/xferlog" temp="/tmp/$(basename $0).$$" nicenum="$HOME/bin/nicenumber" # Script #4 trap "/bin/rm -f $temp" 0 extract() { # Called with $1 = desired accessmode, $2 = section name for output if [ ! -z "$(echo $accessmode grep $1)" ] ; then echo "" ; echo "$2" if [ "$1" = "a" -o "$1" = "g" ] ; then echo " common account (entered password) values:" else echo " user accounts accessing server: " fi awk "\$13 == \"$1\" { print \$14 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' awk "\$13 == \"$1\" && \$12 == \"o\" { print \$9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files downloaded from server:" ; cat $temp fi awk "\$13 == \"$1\" && \$12 == \"i\" { print \$9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files uploaded to server:" ; cat $temp fi fi } ###### The main script block case $# in 0 ) log=$stdxferlog ;; 1 ) log="$1" ;; * ) echo "Usage: $(basename $0) {xferlog name}" >&2 exit 1 esac if [ ! -r $log ] ; then echo "$(basename $0): can't read $log." >&2 exit 1 fi # Ascertain whether it's an abbreviated or standard ftp log file format. If # it's the abbreviated format, output some minimal statistical data and quit: # The abbreviated format is too difficult to analyze in a short script, # unfortunately . if [ ! -z $(awk '$6 == "get" { short=1 } END{ print short }' $log) ] ; then bytesin="$(awk 'BEGIN{sum=0} $6=="get" {sum+=$9} END{print sum}' $log)" bytesout="$(awk 'BEGIN{sum=0} $6=="put" {sum+=$9} END{print sum}' $log)" echo -n "Abbreviated ftpd xferlog from " echo -n $(head -1 $log awk '{print $1, $2, $3 }') echo " to $(tail -1 $log awk '{print $1, $2, $3}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" exit 0 fi bytesin="$(awk 'BEGIN{sum=0} $12=="i" {sum += $8} END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} $12=="o" {sum += $8} END{ print sum }' $log )" time="$(awk 'BEGIN{sum=0} {sum += $6} END{ print sum }' $log)" echo -n "Summary of xferlog from " echo -n $(head -1 $log awk '{print $1, $2, $3, $4, $5 }') echo " to $(tail -1 $log awk '{print $1, $2, $3, $4, $5}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" echo " transfer time: $time seconds" accessmode="$(awk '{print $13}' $log sort -u)" extract "a" "Anonymous Access" extract "g" "Guest Account Access" extract "r" "Real User Account Access" exit 0).$$" nicenum="$HOME/bin/nicenumber" # Script #4 trap "/bin/rm -f $temp" 0 extract() { # Called with = desired accessmode, = section name for output if [ ! -z "$(echo $accessmode grep )" ] ; then echo "" ; echo "" if [ "" = "a" -o "" = "g" ] ; then echo " common account (entered password) values:" else echo " user accounts accessing server: " fi awk "$13 == \"\" { print $14 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' awk "$13 == \"\" && $12 == \"o\" { print $9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files downloaded from server:" ; cat $temp fi awk "$13 == \"\" && $12 == \"i\" { print $9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files uploaded to server:" ; cat $temp fi fi } ###### The main script block case $# in 0 ) log=$stdxferlog ;; 1 ) log="" ;; * ) echo "Usage: $(basename#!/bin/sh # xferlog - Analyzes and summarizes the FTP transfer log. A good doc # detailing the log format is http://aolserver.am.net/docs/2.3/ftp-ch4.htm. stdxferlog="/var/log/xferlog" temp="/tmp/$(basename $0).$$" nicenum="$HOME/bin/nicenumber" # Script #4 trap "/bin/rm -f $temp" 0 extract() { # Called with $1 = desired accessmode, $2 = section name for output if [ ! -z "$(echo $accessmode grep $1)" ] ; then echo "" ; echo "$2" if [ "$1" = "a" -o "$1" = "g" ] ; then echo " common account (entered password) values:" else echo " user accounts accessing server: " fi awk "\$13 == \"$1\" { print \$14 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' awk "\$13 == \"$1\" && \$12 == \"o\" { print \$9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files downloaded from server:" ; cat $temp fi awk "\$13 == \"$1\" && \$12 == \"i\" { print \$9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files uploaded to server:" ; cat $temp fi fi } ###### The main script block case $# in 0 ) log=$stdxferlog ;; 1 ) log="$1" ;; * ) echo "Usage: $(basename $0) {xferlog name}" >&2 exit 1 esac if [ ! -r $log ] ; then echo "$(basename $0): can't read $log." >&2 exit 1 fi # Ascertain whether it's an abbreviated or standard ftp log file format. If # it's the abbreviated format, output some minimal statistical data and quit: # The abbreviated format is too difficult to analyze in a short script, # unfortunately . if [ ! -z $(awk '$6 == "get" { short=1 } END{ print short }' $log) ] ; then bytesin="$(awk 'BEGIN{sum=0} $6=="get" {sum+=$9} END{print sum}' $log)" bytesout="$(awk 'BEGIN{sum=0} $6=="put" {sum+=$9} END{print sum}' $log)" echo -n "Abbreviated ftpd xferlog from " echo -n $(head -1 $log awk '{print $1, $2, $3 }') echo " to $(tail -1 $log awk '{print $1, $2, $3}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" exit 0 fi bytesin="$(awk 'BEGIN{sum=0} $12=="i" {sum += $8} END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} $12=="o" {sum += $8} END{ print sum }' $log )" time="$(awk 'BEGIN{sum=0} {sum += $6} END{ print sum }' $log)" echo -n "Summary of xferlog from " echo -n $(head -1 $log awk '{print $1, $2, $3, $4, $5 }') echo " to $(tail -1 $log awk '{print $1, $2, $3, $4, $5}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" echo " transfer time: $time seconds" accessmode="$(awk '{print $13}' $log sort -u)" extract "a" "Anonymous Access" extract "g" "Guest Account Access" extract "r" "Real User Account Access" exit 0) {xferlog name}" >&2 exit 1 esac if [ ! -r $log ] ; then echo "$(basename#!/bin/sh # xferlog - Analyzes and summarizes the FTP transfer log. A good doc # detailing the log format is http://aolserver.am.net/docs/2.3/ftp-ch4.htm. stdxferlog="/var/log/xferlog" temp="/tmp/$(basename $0).$$" nicenum="$HOME/bin/nicenumber" # Script #4 trap "/bin/rm -f $temp" 0 extract() { # Called with $1 = desired accessmode, $2 = section name for output if [ ! -z "$(echo $accessmode grep $1)" ] ; then echo "" ; echo "$2" if [ "$1" = "a" -o "$1" = "g" ] ; then echo " common account (entered password) values:" else echo " user accounts accessing server: " fi awk "\$13 == \"$1\" { print \$14 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' awk "\$13 == \"$1\" && \$12 == \"o\" { print \$9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files downloaded from server:" ; cat $temp fi awk "\$13 == \"$1\" && \$12 == \"i\" { print \$9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp if [ -s $temp ] ; then echo " files uploaded to server:" ; cat $temp fi fi } ###### The main script block case $# in 0 ) log=$stdxferlog ;; 1 ) log="$1" ;; * ) echo "Usage: $(basename $0) {xferlog name}" >&2 exit 1 esac if [ ! -r $log ] ; then echo "$(basename $0): can't read $log." >&2 exit 1 fi # Ascertain whether it's an abbreviated or standard ftp log file format. If # it's the abbreviated format, output some minimal statistical data and quit: # The abbreviated format is too difficult to analyze in a short script, # unfortunately . if [ ! -z $(awk '$6 == "get" { short=1 } END{ print short }' $log) ] ; then bytesin="$(awk 'BEGIN{sum=0} $6=="get" {sum+=$9} END{print sum}' $log)" bytesout="$(awk 'BEGIN{sum=0} $6=="put" {sum+=$9} END{print sum}' $log)" echo -n "Abbreviated ftpd xferlog from " echo -n $(head -1 $log awk '{print $1, $2, $3 }') echo " to $(tail -1 $log awk '{print $1, $2, $3}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" exit 0 fi bytesin="$(awk 'BEGIN{sum=0} $12=="i" {sum += $8} END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} $12=="o" {sum += $8} END{ print sum }' $log )" time="$(awk 'BEGIN{sum=0} {sum += $6} END{ print sum }' $log)" echo -n "Summary of xferlog from " echo -n $(head -1 $log awk '{print $1, $2, $3, $4, $5 }') echo " to $(tail -1 $log awk '{print $1, $2, $3, $4, $5}')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" echo " transfer time: $time seconds" accessmode="$(awk '{print $13}' $log sort -u)" extract "a" "Anonymous Access" extract "g" "Guest Account Access" extract "r" "Real User Account Access" exit 0): can't read $log." >&2 exit 1 fi # Ascertain whether it's an abbreviated or standard ftp log file format. If # it's the abbreviated format, output some minimal statistical data and quit: # The abbreviated format is too difficult to analyze in a short script, # unfortunately. if [ ! -z $(awk ' == "get" { short=1 } END{ print short }' $log) ] ; then bytesin="$(awk 'BEGIN{sum=0} =="get" {sum+=} END{print sum}' $log)" bytesout="$(awk 'BEGIN{sum=0} =="put" {sum+=} END{print sum}' $log)" echo -n "Abbreviated ftpd xferlog from " echo -n $(head -1 $log awk '{print , , }') echo " to $(tail -1 $log awk '{print , , }')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" exit 0 fi bytesin="$(awk 'BEGIN{sum=0} =="i" {sum += } END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} =="o" {sum += } END{ print sum }' $log )" time="$(awk 'BEGIN{sum=0} {sum += } END{ print sum }' $log)" echo -n "Summary of xferlog from " echo -n $(head -1 $log awk '{print , , , , }') echo " to $(tail -1 $log awk '{print , , , , }')" echo " bytes in: $($nicenum $bytesin)" echo " bytes out: $($nicenum $bytesout)" echo " transfer time: $time seconds" accessmode="$(awk '{print }' $log sort -u)" extract "a" "Anonymous Access" extract "g" "Guest Account Access" extract "r" "Real User Account Access" exit 0
How It Works
In an xferlog , the total number of incoming bytes can be calculated by extracting just those lines that have direction="i" and then summing up the eighth column of data. Outgoing bytes are in the same column, but for direction="o" .
bytesin="$(awk 'BEGIN{sum=0} =="i" {sum += } END{ print sum }' $log )" bytesout="$(awk 'BEGIN{sum=0} =="o" {sum += } END{ print sum }' $log )" Ironically, the slower the network connection, the more accurate the total connection time is. On a fast network, smaller transfers are logged as taking zero seconds, though clearly every transfer that succeeds must be longer than that.
Three types of access mode are possible: a is anonymous, g is for users who utilize the guest account (usually password protected), and r is for real or regular users. In the case of anonymous and guest users, the account value (field 14) is the user's password. People connecting anonymously are requested by their FTP program to specify their email address as their password, which is then logged and can be analyzed .
Of this entire xferlog output stream, the most important entries are those with an anonymous access mode and a direction of i , indicating that the entry is an upload listing. If you have allowed anonymous connections and have either deliberately or accidentally left a directory writable, these anonymous upload entries are where you'll be able to see if skript kiddies, warez hackers, and other characters of ill repute are exploiting your system. If such an entry lists a file uploaded to your server, it needs to be checked out immediately, even if the file-name seems quite innocuous .
This test occurs in the following statement in the extract function:
awk "$13 == \"\" && $12 == \"i\" { print $9 }" $log sort \ uniq -c sort -rn head -10 sed 's/^/ /' > $temp In this rather complex awk invocation, we're checking to see whether field 13 matches the anonymous account code (because extract is called as extract "a" "Anonymous Access" ) and whether field 12 indicates that it's an upload with the code i . If both of these conditions are true, we process the value of field 9, which is the name of the file uploaded.
If you're running an FTP server, this is definitely a script for a weekly (or even daily) cron job.
Running the Script
If invoked without any arguments, this script tries to read and analyze the standard ftpd transfer log /var/log/xferlog . If that's not the correct log file, a different filename can be specified on the command line.
The Results
The results depend on the format of the transfer log the script is given. If it's an abbreviated form, some minimal statistics are generated and the script quits:
$ xferlog succinct.xferlog Abbreviated ftpd xferlog from Aug 1 04:20:11 to Sep 1 04:07:41 bytes in: 215,300,253 bytes out: 30,305,090
When a full xferlog in standard format is encountered , considerably more information can be obtained and displayed by the script:
$ xferlog Summary of xferlog from Mon Sep 1 5:03:11 2003 to Tue Sep 30 17:38:50 2003 bytes in: 675,840 bytes out: 3,989,488 transfer time: 11 seconds Anonymous Access common account (entered password) values: 1 taylor@intuitive.com 1 john@doe files downloaded from server: 1 /MySubscriptions.opml files uploaded to server: 1 /tmp/Find.Warez.txt Real User Account Access user accounts accessing server: 7 rufus 2 taylor files downloaded from server: 7 /pub/AllFiles.tgz 2 /pub/AllFiles.tar
Security Alert! Did you notice that someone using anonymous FTP has uploaded a file called /tmp/Find.Warez.txt ? "Warez" are illegal copies of licensed software ” not something you want on your server. Upon seeing this, I immediately went into my FTP archive and deleted the file.
