Getting Performance Metrics

Problem

I want to use the new perfmonitor preprocessor to help gather some passive statistics on my network load and sensor loads. How can I do that?

Solution

The perfmonitor preprocessor has to be enabled at Snort compile time to enable all the performance counters.

./configure --enable-perfmonitor -my -other -options

You can have more than one perfmonitor operational in your snort.conf file, but be aware that this causes a noticeable load increase on Snort and possibly on your sensor platform for disk reads/writes as well. Depending on how you invoke Snort, you may find the following scenario useful.

This example will output to the screen every five minutes:

preprocessor perfmonitor: flow console time 300

This will log the raw data to a logfile every 10 minutes instead of outputting to the screen:

preprocessor perfmonitor: flow snortfile time 600

Then run Snort like this example under /bin/sh.

This will send all STDOUT (standard output, the screen) and STDERR (standard error, also usually the screen) to the file console.log. However, remember that this works only under the shell interpreter sh.

sh 

/path/to/snort -c /path/to/snort.conf -i SNIFF_INT -l /path/to/log -z > 

snort_console.log 2>&1

This will create the comma-separated values (CSV) file for you to parse through later with snort-rrd.pl, snortgraph, or other tools to graph the data, while at the same time displaying to a file some nicely formatted statistics every five minutes (time 300 value is seconds) much like this.

# displayed by running snort as in example above 

# with a perfmonitor setting of 

# preprocessor perfmonitor: snortfile perfstats.log console flow 

# time 60 events pktcnt 100

# 

Snort Realtime Performance : Sun Aug XX 00:42:12 2004

--------------------------

Pkts Recv: 1602

Pkts Drop: 0

% Dropped: 0.00%

 

KPkts/Sec: 0.03

Bytes/Pkt: 1052

 

Mbits/Sec: 0.12 (wire)

Mbits/Sec: 0.10 (rebuilt)

Mbits/Sec: 0.22 (total)

 

PatMatch: 171.00%

 

CPU Usage: 0.11% (user) 0.03% (sys) 99.87% (idle)

 

Alerts/Sec : 0.0

Syns/Sec : 0.0

Syn-Acks/Sec : 0.0

New Sessions/Sec: 0.0

Del Sessions/Sec: 0.0

Total Sessions : 3

Max Sessions : 3

Stream Flushes/Sec : 10.2

Stream Faults/Sec : 0

Stream Timeouts : 1

Frag Completes( )s/Sec: 0.0

Frag Inserts( )s/Sec : 0.0

Frag Deletes/Sec : 0.0

Frag Flushes/Sec : 0.0

Frag Timeouts : 0

Frag Faults : 0

 

....more available to the user

There are a several options to the type of data the perfmonitor preprocessor logs. These options will have to be outputted as console or screen data, as seen earlier. Otherwise, the data logged with the file/snortfile keyword is output as CSV-formatted logs written to a logfile for later analysis, as in this example:

1093754532,0.000,0.1,0.0,0.0,1052,171.00,0.0,0.0,0.0,0.0,3,3,10.2,0,1,

0.0,0.0,0.0,0.0,0,0,0.1,0.0,99.9

1093755035,0.000,0.3,0.0,0.1,919,164.21,2.7,2.7,2.8,2.7,6,12,41.2,0,1,

0.0,0.0,0.0,0.0,0,0,0.2,0.1,99.6

1093755096,0.000,0.1,0.0,0.0,1048,171.03,0.0,0.0,0.0,0.1,3,12,14.0,0,2,

0.0,0.0,0.0,0.0,0,0,0.1,0.0,99.8

1093755180,0.000,0.1,0.0,0.0,749,156.76,1.5,1.5,1.5,1.5,3,7,21.8,0,1,

0.0,0.0,0.0,0.0,0,0,0.2,0.0,99.8

For a quick meaning of this data, you can compile perfstats.c, which comes with the Snort source code, in the contrib directory.

 "gcc -o perfstats.exec perfstats.c"

Then using your newly compiled program, you can get a quick idea of what kind of data is being logged with the preprocessor, as you can see in the following code. The following output is available through the perfstats program.

# Run the program like this to get a quick statistics page 

# The "-q" flag just tells the perfstats program to only display a 

# summary of the information. With no options the perfstats program

# displays the summary information for each line its analyzing then

# the full summary like with "-q" at the bottom. 

 

cat .log | ./perfstats.exec -q 

4 statistics lines read

 

 Mbits/Sec: 0.1 0.0 0.3

 Drop Rate: 0.0000% 0.0000% 0.0000%

 Alerts/Sec: 0.0 0.0 0.0

 K-Pkts/Sec: 0.0 0.0 0.1

 Avg Bytes/Pkt: 942.0 749.0 1052.0

 Pat-Matched: 165.8 156.8 171.0

 Syns/Sec: 1.1 0.0 2.7

 SynAcks/Sec: 1.1 0.0 2.7

 New/Sec: 1.1 0.0 2.8

 Del/Sec: 1.1 0.0 2.7

 Active: 3.8

 Max Active: 15.0

 Flushes/Sec: 21.8 10.2 41.2

 Faults: 0.0

 Timeouts: 1.2

Frag-Completes/Sec: 0.0 0.0 0.0

 Frag-Inserts/Sec: 0.0 0.0 0.0

 Frag-Deletes/Sec: 0.0 0.0 0.0

 Frag-Flushes/Sec: 0.0 0.0 0.0

 Frag-Timeouts: 0.0

 Frag-Faults: 0.0

 Usr: 0.2 0.1 0.2

 Sys: 0.0 0.0 0.1

 Idle: 99.8 99.6 99.9

If you find this type of data useful, the program supports specifying the number of lines to read back out of the logfile. For example, if you are writing to this file every five minutes, run perfstats every hour to get the statistics for the past hour by passing it the -c flag. You could call the following shell script out of a cron job on the sensor to write a status update of the load on the sensor and that portion of the network.

#!/bin/sh 

# 

# Create an hourly report from the sensor

#

# Variables 

# Time or date stamp on the file anyone? This can be useful to 

# determine if you have failure as to the last run time. 

mydate='date "+DATE: %Y-%m-%d%nTIME: %H:%M:%S"'

 

# Clean the old file and create a blank new one 

rm /path/to/status.txt 

touch /path/to/status.txt 

 

# ROUGH formatting of the new file 

echo " " > /path/to/status.txt 

echo " " >> /path/to/status.txt 

echo "THIS THE LAST HOURS performance data for: " >> /path/to/status.txt

echo "$mydate" >> /path/to/status.txt 

echo " " >> /path/to/status.txt

echo " " >> /path/to/status.txt 

 

# Execute the perfstats program only showing the data for the past 

# hour by using the -c to only analysis the last 12 lines in the 

# log file 

 

cat /path/to/perfstats.log | ./path/to/perfstats.exec -q -c 12 >> 

/path/to/status.txt

 

Discussion

There are some other options that you can enable in the perfmonitor preprocessor to help you gather better data for your organization. The previous examples can be tweaked with some options to the perfmonitor preprocessor with some keywords to the preprocessor.

One example is the flow option; this can be replaced with the events option or combined. The flow option on the Snort perfmonitor preprocessor calculates the protocol and traffic distribution going past the sensor. This was shown earlier in the example of real-time performance data. However, the events keyword turns on Snort ruleset measurements. Following is a partial example of the events keyword's data.

Snort Setwise Event Stats

-------------------------

Total Events: 2444

Qualified Events: 466

Non-Qualified Events: 1978

%Qualified Events: 19.0671%

%Non-Qualified Events: 80.9329%

When the events keyword is enabled, it tracks qualified events. Qualified events are packets that have triggered a rule, while nonqualified events are packets that either didn't match up to a rule or were found to be non-hostile. The Snort documentation from the author explains it as:

This prints out statistics as to the number of signatures that were matched by the set wise pattern matcher and the number of those matches that were verified with the signature flags. We call these non-qualified and qualified events. It shows the user if there is a problem with the rule set that they are running."

In other words, this means that for about 80 percent of the traffic, signatures are not matching on this sensor! This could point out a problem with your signatures or that you might want to consider looking at the flow data to determine if this is correct for the network segment this sensor is monitoring.

Another tweak of the perfmonitor preprocessor is to improve performance. By now, you can see that when not used properly, this preprocessor might cause some severe load on Snort and the sensor. Two more keywords can help with this task: time and pktcnt.

The time keyword tells the perfmonitor when intervals in the counts should take place, while the pktcnt (packet count) keyword counts the number of packets that have passed the sensor since the last time the file or console was written to. The default for this is 10,000 packets, which might either be too many or too few for some of the higher-speed networks, even within a 5 to 10 minute time frame. Feel free to adjust as needed as in our previous example.

preprocessor perfmonitor: snortfile perfstats.log console flow time 60

events pktcnt 100

This example tells Snort to log to a CSV-formatted file perfstats.log, while at the same time writing nicely formatted output to the screen. It tracks traffic distribution and signature distribution information on a one-minute window of time after passing only 100 packets. This configuration works for a cable modem or DSL line, so adjust it to better suit your sensor's network segment.

One last keyword helps calculate the theoretical limits of your network segment based on the performance data within each data set or write. However, since many multiprocessor machines don't keep accurate kernel statisticsor in some cases, any kernel statistics at allthis helps only on single processor sensors. The following is an example of the max keyword turned on in the previous perfmonitor configuration:

#preprocessor perfmonitor: snortfile perfstats.log console flow 

# time 60 events pktcnt 100 max

 

Snort Maximum Performance

-------------------------

 

Mbits/Second

----------------

Snort: 109.16

Sniffing: 477.56

Combined: 88.85

 

uSeconds/Pkt

----------------

Snort: 41.06

Sniffing: 9.39

Combined: 50.45

 

KPkts/Second

------------------

Snort: 24.35

Sniffing: 106.55

Combined: 19.82

 

See Also

Beale, Jay. Snort 2.1 Intrusion Detection. Rockland, MA: Syngress, 2004.

Snort source code documentation

Snort-devel mailing list

Experimental Preprocessors