Detecting and Normalizing HTTP Traffic

Problem

With all the available HTTP evasion attacks, how can Snort help detect these types of attacks? Can I create unique profiles for each of my web servers? Can I detect HTTP proxy usage?

Solution

As of Snort 2.0, there is a preprocessor to handle all the HTTP traffic coming through the Snort engine. This preprocessor has grown in flexibility and features and now has two parts: a global and a server section. As there are four variables for the global section and three server default server profiles, you have 24 specific alarms and variations from which to choose.

Global examples

These options are set for all the server(s) you create.

Snort has a default language interpreter of English. This means Snort can translate Unicode characters to English for ASCII comparison in rules. However, for organizations that need to have other language support, there is a tool in the Snort source code distribution, called ms-unicode-generator.c, that needs to be compiled to run. Once compiled, it will build a new Unicode map for Snort to use from a new language file.

preprocessor http_inspect: global iis_unicode_map unicode.map 1252

The following example turns on the proxy detection for our server to use. If this is not enabled here, proxy through your web server will not be detected.

Preprocessor http_inspect: global iis_unicode_map unicode.map  

1252 proxy_alert

The following example detects new web servers coming online. Be warned that this works through stateless detection, which makes it highly unusable on your core network. The reason for the flood of alarms is that the detection method at this point in Snort is stateless. This means that every packet that has your HTTP_PORTS variable port is going to be checked as a new web server! On even a medium-sized network, that means that the moment you turn on this detection, every workstation shows up as a new web server! The reason for this "flaw" is that the current implementation of the web server detection code is stateless. This means that the preprocessor doesn't differentiate between who started the HTTP connection; it will determine that whoever gets port 80/tcp packets must be a web server!

Preprocessor http_inspect: global iis_unicode_map unicode.map 1252 

detect_anomalous_servers

 

Server examples

The server portion of the http_inspect engine gets very granular and specific for web servers you want to monitor.

The first option is the default server config. This will apply to all HTTP servers not specifically named in a server configuration. The following example is the default build from the snort.conf file. It uses the server profile all and listens for HTTP on ports 80, 8080, and 8180 TCP. It alerts on all the events turned on by the profile all option. It also detects when URL directories are larger than 500 characters long.

preprocessor http_inspect_server: server default 

 profile all ports { 80 8080 8180 } oversize_dir_length 500

The following solution might be good for your IIS web servers. Please note that all uncommented options for a profiled server are actively used.

Preprocessor http_inspect_server: server  profile iis 

ports {80 8080 }  # The ports to filter for HTTP traffic to/from this server

flow_depth 200  # How many bytes to down down into the server response

inspect_uri_only # Performance improvement to only look at the url field 

#

# oversize_dir_length  

## number of characters outside of the web root that this will trigger an alarm on. 

# iis_unicode_map   

## can be specified for other language servers within your organization 

# allow_proxy_use 

## turn off alarms for HTTP proxying through the server 

# no_alerts 

## disables all http_inspect alarms for this server

As you might have noticed, when you use a server profile, you lose most of the flexibility to enable and disable http_inspect alarms. Mimicking all the options in the profile iis while allowing the flexibility to change would look like this:

Preprocessor http_inspect_server: server  

ports {80 8080} 

flow_depth 300 

ascii no 

multi_slash no

directory no 

webroot

double_decode yes  

u_encode yes  

bare_byte yes  

iis_unicode yes 

iis_backslash no 

iis_delimiter no 

apache_whitespace no  

non_strict

Creating the same server configuration for the Apache profile is smaller, as it has less application data to decode and normalize under normal conditions:

# Remember the yes/no option only turns on or off alerting

Preprocessor http_inspect_server: server  

ports {80 8008} 

flow_depth 300 

non_strict 

chunk_length 500000 

acsii no 

multi_slash no  

directory no  

apache_whitespace yes 

webroot 

utf_8 no

However, if you want to just place your server's IPs and ports in use with no other options, this is what you will get:

Preprocessor http_inspect_server: server  

ports {80 8080 }  # If not specified defaults to only port 80 

flow_depth 300 

chunk_length 500000 

ascii no 

utf_8 no 

multi_slash no 

directory no 

webroot 

apache_whitespace no 

iis_delimiter no 

non_strict

 

Discussion

The http_inspect preprocessor breaks down into two parts: global and server. The global portion enables some of the server options, such as proxy detection for each server and Unicode-to-English mappings. The Unicode mapping just requires a Unicode map file, and then a number to the proper mapping for your language. For most situations, the unicode.map file found in the etc directory of the Snort installation will serve most users. However, should you need to create your own Unicode mapping, there is a file called ms_unicode_generator.c that is found with the Snort source code in the contrib directory. If you are on a non-English version of Windows, you can compile this tool that will create your map file for that specific language.

The proxy_alert keyword allows defined web servers to alarm when they're being used in an HTTP proxy. This can be helpful in determining when users are bypassing a defined proxy server. For example, turn on the global variable proxy_alert, and then enable by proxy server a defined host with proxy traffic allowed through it. If you leave the default server to alert on proxy HTTP traffic, this tells you when users are using an unauthorized proxy server.

# Global 

Preprocessor http_inspect: global iis_unicode_map unicode.map 1252 proxy_alert 

# Proxy Server

preprocessor http_inspect_server: server  

ports {80 3128 } 

allow_proxy_use 

# Everything else 

preprocessor http_inspect_server: server default 

ports { 80 } 

profile all

The server portion of the http_inspect preprocessor can be tailored to most of the common web server configurations. As shown in the previous code, this preprocessor can handle out of the box some of the idiosyncrasies of the Microsoft IIS web server. For example, it handles the successful translation of the Unicode characters into normalized data. Table 4-2 should help show all the Unicode characteristics the preprocessor can handle.

Table 4-2. Http_inspect Unicode keywords and their meanings

Keyword	Detection
ascii	This keyword decodes ASCII characters, such as "ping." It can help in detecting unencoded attacks such as a default directory traversal attack.
iis_unicode	This keyword decodes Microsoft IIS Unicoded characters such as "%2fscripts." It uses theunicode.map file for proper unicode to ascii translations..
double_decode	This handles the fact that IIS runs two passes through each URI request: the first for encoding (Unicode, UTF), and the second for plain text.
bare_byte	This handles the use of non-ASCII characters to decode UTF-8 characters. There are no known legitimate uses of this encoding. However, it does seem to be popular with the way certain IIS served web sites communicate with IE browsers, so be aware of a high false-positive hit count.

Finally, along with the server-side features is the normalization of any HTTP traffic that passes the preprocessor. For example, with the multi_slash keyword, any HTTP traffic that comes in looks flaky, such as this URL:

GET /etc///////////passwd HTTP/1.0

This would get normalized back to the rules engine as:

GET /etc/passwd HTTP/1.0

Then correctly trigger the following rule:

alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"WEB-MISC 

/etc/passwd"; flow:to_server,established; content:"/etc/passwd"; nocase; 

classtype:attempted-recon; sid:1122; rev:5;)

Without the help of the http_inspect preprocessor, that attack would have probably gone right past Snort without triggering an alarm.

These attacks aren't limited to IIS servers. Apache servers benefit from normalization as well. For example, a while ago there was a "chunked-encoding" exploit in Apache that could be detected by the HTTP preprocessor. It didn't detect the exploit based on the content of the packetas this string was used by valid applicationsbut rather by the size of the request. In the previous solutions, we see our use of the chunk_length keyword. In the previous example, we filtered out normal-sized chunks of data. Chunk is a base size of an HTTP session payload that the preprocessor will examine for any given HTTP session. When we set it to 500,000 bytes long, this will successfully detect the buffer overflow portion of the chunked encoding exploit. Another example of Apache normalization would be the use of the apache_whitespace keyword to help normalize the use of a tab versus a space keyboard key to be handled as a space within a URL string.

Finally, if you have http_inspect normalization enabled, certain signature rules are never going to alert. This is because in the Snort rules language, there are two keywords that deal with payload data: content and uricontent. The content keyword looks through the raw data being handed back to the Snort rules engine. The uricontent keyword handles only the normalized data headed back after the http_inspect preprocessor handles it. From the Snort documentation, see this example:

For example, the URI:

/scripts/..%c0%af../winnt/system32/cmd.exe?/c+ver

will get normalized into:

/winnt/system32/cmd.exe?/c+ver

This example illustrates what the http_inspect preprocessor normalizes in an HTTP session. For example, the attacker is going to use the actual directory traversal attack, so you write a rule to detect that part of the attack /..%. However, when you wrote your rule, you used the uricontent keyword to look only within the HTTP Get statements, not realizing that uricontent strips off the Unicode characters from an HTTP session. This act makes your rule useless, as it will never trigger. When creating your HTTP rules, be aware of when you might be stepping on the functions of one of the preprocessors.

See Also

Snort-devel mailing list

Beale, Jay. Snort 2.1 Intrusion Detection. Rockland, MA: Syngress, 2004.

Snort documentation (http://www.snort.org)

Decoding Application Traffic