Before we can get into the details of what a network sniffer and an IDS can and cannot do, we need to cover some network protocol basics. Since most network traffic is TCP/IP, I’ll use that network protocol suite as it applies to Windows for this discussion. All network protocol discussions start with the OSI model.
The Open Systems Interconnection (OSI) model is something you either love or hate. If you plan to do network analysis for a living, you should probably learn to love it. The truth is that explaining network and computer problems using it makes life easier.
I liken most new administrator’s use of the OSI model to new programmers using flowcharts when learning how to program. Every beginning programming course always preaches the value of flowcharting the program and defining customer requirements before the developer ever writes the first line of program code. This is to prevent rework and mistakes because of logical gaps in the first program attempt. Programmers learn this early on, but then skip that step because it slows them down and takes too long. Then about five to ten years into their programming career, they start yelling because the client isn’t defining their requirements well enough, and they are constantly rewriting their code just to get the first version out. The now experienced programmers start demanding that flowcharts be used, and customer expectations signed off and documented before coding begins, because they’ve realized it will save them time. Experienced network administrators learn the same thing about the OSI model. Once you embrace it and use it to troubleshoot your network problems, life becomes easier.
The OSI model (http://www.geocities.com/SiliconValley/Monitor/3131/ne/osimodel.html), shown in Figure 9-1, is a theoretical abstraction of how most networks and their protocols interact. The idea is that every network has all of these layers, whether it knows it or not. Now, few real networks neatly fit the model, but most can be retrofitted to fit it. For example, some models of the TCP/IP protocol suite have only four or five layers, not seven layers, as the OSI model does. But usually the other model just mixes the layers a bit differently, and the OSI model is a bit more precise, even if a particular protocol doesn’t neatly fit it.
Figure 9-1: The OSI model
Microsoft uses a four-layer network model to describe the Windows TCP/IP stack. The upper three layers of the OSI model are combined into a single application layer in Microsoft’s model, and the lowest two OSI layers are combined into a one-layer network interface layer. See http://www.microsoft.com/resources/documentation/Windows/2000/server/reskit/en-us/Default.asp?url=/resources/documentation/windows/2000/server/reskit/en-us/cnet/cnbb_tcp_kscb.asp for more details. Despite the different layer representations, both models describe the same types of network traffic.
The model assumes that all networks have these layers, and each upper-layer protocol depends on the lower-layer protocol to do its job. The upper-layer protocol doesn’t need to worry about any of the characteristics or mechanics of the lower layer. The layers can be, and often are, completely independent of each other. For example, an IP packet doesn’t care about MAC addresses anymore than a TCP packet cares about destination IP address.
The OSI model begins with the bottom layer, the physical layer. This is the wired or optical connections that move the electricity or light that makes up your network bits along from the source to the destination. It’s also where you should start when troubleshooting any network problem. We all have spent hours trying to fix some problem, only to find out that we didn’t have a link light on our network device, or that our network interface card was bad, or just that the network cable wasn’t plugged in. Come on, admit it!
The physical layer includes all physical components on the network that get data from the source to the destination. This includes networking hardware—hubs, switches, routers, and anything else that moves data along.
The next layer is the data-link layer. This layer is where the software interacts with the hardware. Networks are often described by their data-link characteristics—as Ethernet, Token-Ring, FDDI, ARCnet, or some other type of network defined by physical and logical characteristics.
At this level, we refer to packets as frames, and source and destination addresses are described by their network interface card’s MAC address. All network packets eventually need to be delivered using the destination computer’s MAC address. Switches and other bridge-like devices work on this layer, and they are mostly concerned with frame types and MAC addresses. They don’t care about upper-layer protocols like IPX versus IP. They just move frames.
The third layer of the OSI model is the network layer. This layer is mostly concerned with getting packets from the source to the destination hosts. It doesn’t care about port numbers and frame types; it cares only about source and destination computer addresses. If the network protocol is IP or ICMP, it cares about host IP addresses. If it is IPX, it cares about logical computer numbers. If it is NetBEUI, it cares about NetBIOS names. What host addresses the network protocol uses depend on the protocol.
Routers function at this level and accept or deny packets based purely on the network layer. Once the network-layer protocol has delivered a packet to its final destination, the layer is stripped off to reveal the upper-layer transport protocol.
The transport layer describes network protocols that deal primarily with moving data from higher-level protocols and applications to the lower layers, and vice versa. In TCP/IP, applications are assigned transport port numbers that they use to send and receive information.
The transport layer in the TCP/IP protocol suite is usually TCP or UDP. Applications can use one or both transport protocols to deliver their contents depending on their needs. The transport layer usually holds the application’s data in its payload. Each transport protocol type has its benefits and disadvantages, as discussed in the “TCP/IP Suite Basics” section of this chapter.
The session layer assists in setting up communications between applications and the transport protocols. It gets involved in establishing communication sessions, breaking them down, and controlling the flow of information between two hosts. In the Windows world, the NetBIOS protocols, NetBIOS Name Service, NetBIOS Datagram Service, and NetBIOS Session service are part of this layer.
The presentation layer is involved in formatting the data in the payload correctly so that the source and destination applications can understand it. This layer is typically handled by the OS or even the application itself. This layer is involved in data conversion when different hosts use different character sets (such as ASCII versus EBCDIC). In the Windows world, the Server Message Block (SMB) protocol is defined here.
The last layer, the application layer, describes the actual applications running on each host machine. This is where SMTP, FTP, telnet, RDP, RealPlayer, and HTTP come into play.
Most network analysis systems display network data along the lines of the OSI model. For example, any decent network protocol analyzer will display packet data, layer by layer. Each available layer is shown as a distinct subsection of that captured packet.
Hackers may attack along any layer of the OSI model. In my experience, hackers develop interests in certain layers of the OSI model and rarely move outside that layer. For example, some hackers spend their entire life attacking the data-link layer (ARP spoofing), network layer (man-in-the-middle and SYN floods), transport layer (UDP floods), or the application layer (buffer overflows). They never seem to jump from one layer to another. Unfortunately, you never know who your opponent will be, so you’ll need to be familiar with all the layers and their protocols. Fortunately, most networks run TCP/IP, so you’ll just need to master one protocol suite’s behaviors and quirks.
Of course, if the hacker attacks the physical layer (for example, inserts an Ethernet tap to capture packets), you’ll be hard-pressed to notice it using a software tool. That’s why physical protection is the first tenet of computer security.
The TCP/IP protocol suite is made up dozens of protocols spread across the seven OSI layers. The ones you’ll need to be the most familiar with are IP, TCP, UDP, and ICMP.
When one host computer sends data to another host, it usually sends the data to the computer’s network name. The name can be a DNS name, as is common in today’s Internet-connected world, or some other sort of name, as defined by the computer’s network. For instance, slightly older Windows networks use NetBIOS instead of DNS naming and name resolution.
Let’s go through a theoretical TCP/IP communication session between Workstation1 and Server1 to demonstrate how the different protocol levels interact. Workstation1 needs to transmit data to Server1. Workstation1’s application sends the data to the Server1 computer. The application has been predefined to use a particular transport protocol, usually TCP or UDP, and a particular source and destination port number. This is called the transport layer. Transport-layer packets contain (among other things) source and destination port numbers, and they also identify the protocol type. The transport packet is placed into an IP packet. The IP packet’s job is to get the upper-layer transport packets from the source computer to the destination computer using IP addresses. For the purposes of our example, let’s assume Workstation1 does not know Server1’s IP address.
Workstation1’s DNS client program (called a DNS resolver) immediately begins to try to convert Server1’s name into an IP address. The DNS client looks to see if Server1’s IP address is in Workstation1’s DNS cache. This is an area in memory where previously resolved DNS names are stored for a temporary period of time (usually two hours). If it doesn’t find the IP address there, it looks at the client’s HOST file. This file is usually located in the %windir%\ system32\driver\etc folder (on Windows NT and later systems) and contains manually entered computer names and their IP addresses. If Server1’s name and IP address are not found in the HOST file, then Workstation1’s defined primary DNS server is queried for the answer. The DNS server goes through a process similar to the client, using its cache and HOST file. The DNS server will then look to see if it contains the answer in its DNS zone files. If not, it will either resolve the request by querying other DNS servers or tell the client it cannot resolve the name.
Let’s assume Server1’s IP address is resolved and returned to Workstation1. If on the same physical network, Workstation1's IP stack sends the packet to the IP address of Server1. If Server1 is not on the same network as Workstation1, the IP packet is sent to Workstation1’s defined gateway (router) address. If other computers are able to see the IP packet (for instance, attached to a hub), they will ignore or drop the packet, by default, unless it is destined for them. However, network sniffers, in promiscuous mode, will capture any packet that their IP stack can see. If Server1 is not located on the same local network, one or more routers, send the IP packet on to its eventual destination network. Once on the destination network, the IP packet’s destination IP address must be converted to its destination MAC address before it can be delivered to Server1.
If Workstation1 and Server1 are on the same network, Workstation1 issues an Address Resolution Protocol (ARP) request query to convert the destination IP address into a MAC address. An ARP request essentially asks, “Who has such-and-such IP address?” The answer allows the network’s layer 2 (data-link layer) to transfer the traffic from the source to destination host. If the destination host is on the same network segment, it will respond with its ARP reply and identify itself. If the destination host is not on the same network segment, the packet is sent to the local gateway router. When the packet reaches the destination network, the closest gateway interface of the last router involved (the one closest to Server1 and on its local segment) will send the ARP request. If the source machine needing the destination ARP address has it already resolved in its ARP cache, no ARP request is sent. If an ARP request is sent, it is broadcast to all computers on the same network segment. The entire TCP/IP OSI process is summarized by Figure 9-2.
Figure 9-2: TCP/IP protocol flow example
Once the packet arrives at its destination, the layer 2 frame is removed, then the IP network frame is removed, then the transport layer packet is stripped, and finally, the encapsulated payload is delivered to the application listening on the correct destination port. Once Server1 has received the packet, it will wait for more packets, send an acknowledgment, or send its own packet to Workstation1.
Workstation1 and Server1 may be handling hundreds of separate TCP/IP sessions, with various hosts, all at once. There are dozens of calculations and checks made in each packet transmission, and the typical network is handling hundreds to hundreds of thousands of packets per second. Of course, each packet transmission, from source to destination, usually happens in under 100 milliseconds. Even when a packet is sent around the world, with an average of more than 20 Internet routers involved (each handling hundreds to hundreds of thousands of packets per second), it usually takes only one or two seconds in the worst-case scenario.
A good network sniffer will break down those layers into separate packets or frames, where you can begin to make logical order of it all. In order to use a sniffer or IDS, you must understand the basics of an IP, TCP, and UDP packet.
IP is the workhorse protocol behind the scenes ferrying almost all other TCP/IP traffic between hosts. Figure 9-3 illustrates the IP packet structure.
Figure 9-3: IP packet structure
Although all the fields of an IP packet need to be inspected during a forensics investigation, the following are the most important fields besides the source and destination IP addresses:
Version: The version number will indicate if the packet is an IPv4 packet or IPv6. Although IPv6 is rare these days, it is quickly gaining use throughout the Internet and on private LANs. Expect it to become the dominating protocol in the next three to five years.
Identification: The Identification field is unique per IP packet. If an IP packet must be fragmented into smaller units for transmission, each fragment will share the same identification number.
IP Flags: The IP Flags field indicates whether the packet is part of a fragmented set, and whether it is the last in the set of fragments.
Fragment Offset: The Fragment Offset field indicates at what byte position a fragmented packet should begin when it is reassembled with other fragmented packets into the larger packet.
Protocol Type: The Protocol Type field indicates what upper-layer protocol the IP packet is transporting in its payload data field. It should be 6 for TCP or 17 for UDP, but it can contain many other values assigned to the various upper-layer protocols.
Hackers routinely manipulate the fragmentation offset so that when the fragments are reassembled, they form unauthorized, malicious traffic. Fragmentation attacks are done in the hopes of defeating computer security defenses, some of which look at only each individual fragment, not the entire packet as a whole. For instance, if I were a hacker, I might be able to slip a buffer overflow past an IDS by splitting the malicious signature of the attack between two packets. If the IDS inspects each fragment separately, it might not find the necessary evidence of a buffer overflow and let both fragments pass, where they get reassembled on the destination host and exploit it. Fortunately, Snort and other IDSs have antifragmentation attack settings available.
TCP is the dependable and flexible protocol of the TCP/IP suite. It is used for its reliable, connection-oriented, and stateful features. With TCP, dropped packets can be retransmitted, large messages can be split over multiple packets (fragmentation), packets can arrive out of sequence and be placed back in order (sequencing), and network throughput speed can be changed on the fly according to network conditions between the sender and the receiver (flow control and window size negotiation). Figure 9-4 illustrates the TCP packet structure.
Figure 9-4: TCP packet structure
Along with the source and destination ports, the following are the most important fields for analysis:
Sequence Number: The sequence number identifies the first byte of data in a packet.
Acknowledgment Number: The acknowledgment number is a number sent by the sender to the receiver telling it what data to send next in the current session. This value tells the originating host which data has been correctly received, so that it can drop the already sent information in its cache buffer (where data is held in case it needs to be retransmitted).
TCP Flags: The TCP flags are one-bit settings, either on (1) or off (0). Not only do these flags play a large part in the reliability and stateful connectedness of TCP, but manipulating TCP flags is a favorite hacker trick. The six flags are as follows:
URG—The Urgent flag indicates the data is urgent. It is usually set to 0. If the Urgent flag is set, the Urgent pointer field tells an IP stack where the urgent data stops and where to begin regular communications again.
ACK—The Acknowledgment flag is set as an affirmation to a valid connection request and continues to be set during normal communications. When data sent by the sender is received by the receiver, the Acknowledgment bit will be set to indicate successful receipt. If the Acknowledgment bit isn’t set when the sender expects it to be (known as a NAK or negative acknowledgment), or the sender doesn’t receive an acknowledgment packet at all, the sender retransmits the TCP packet.
PSH—The Push flag is another bit indicating that a packet should have a high priority. The Push flag is frequently set on packets after a session has been negotiated.
RST—The Reset flag is used to immediately end a communication’s session. This flag is set when a TCP port is closed and a remote probe tries to contact it, or when a port receives an unexpected or malformed packet.
SYN—The Synchronization flag is used to begin a new TCP communication’s session, and it is usually set to 0 after the initial session setup.
FIN—The Finish flag is set to end a communication session normally.
Window Size: The Window Size field tells the sending computer how many bytes of data the receiver can hold before the data must be transferred to the related application. Theoretically, the larger the window size, the faster the communications.
Some of the notable components of TCP (e.g. window size, flags, handshake, etc.) are repeated in this chapter for completeness even though they were covered earlier in Chapter 5.
TCP is known as the reliable protocol because, unlike UDP, it has built-in mechanisms to retransmit unacknowledged data. The receiver verifies the data and acknowledges it with a corresponding acknowledgment packet, as well as an acknowledgment number telling the sender what data it is expecting to receive next. For this reason, most applications use TCP rather than UDP.
Hackers and malicious programs sometimes find it to their benefit to use UDP, which doesn’t have all the acknowledgment overhead traffic. For example, the SQL Slammer worm sent a buffer overflow to Microsoft SQL Server programs on UDP port 1434. The buffer overflow worked in 376 bytes and compromised most infectable machines (those running Microsoft SQL Server and not protected by a firewall) on the Internet in the first ten minutes of its execution. If it had used TCP, it would have been slowed down significantly. It doesn’t always take a guaranteed, reliable connection to do the job.
Another one of TCP’s unique features is its three-way handshake. When one computer needs to start a TCP communication’s session with another, it must first complete the TCP three-way handshake process. The first step involves the originating computer sending the receiver a SYN packet (a TCP packet with just the SYN flag set). If the receiver agrees to start a new session, it sends back an ACK packet with the SYN flag also set. The receiver is saying, “Yes, I’ll communicate. Will you communicate with me?” The third handshake step is the original computer sending back an ACK packet to acknowledge the receiver’s SYN request. Thus, the three-way TCP handshake sequence is represented as SYN, ACK-SYN, ACK. When viewing TCP traffic, you’ll need to get used to all the SYN and ACK packets you’ll see, and notice when all three parts of the handshake aren’t there.
Hackers often manipulate the TCP/IP flags in order to produce illogical (unexpected and undefined) sequences that confuse the target computer, and maybe get around its security defenses. One example of this technique is the SYN flood attack. Hackers can also use flags when conducting port scans.
In a SYN flood DoS attack, an attacker may send the SYN packet to pretend to begin communications with the targeted host. The unsuspecting host replies back with the normal ACK-SYN packet and awaits the third step of the handshake, which never comes. The receiver computer allocates resources and memory to the initiated connection session and begins to wait. When it doesn’t get a reply, it waits more and retransmits its ACK-SYN acknowledgment. Usually, the receiver will respond three times, after waiting for a reply every few seconds between each retransmission.
If the hacker sends enough of these packets (SYN flood attacks are often in the range of hundreds of thousands of malformed packets per second), they use up the receiver’s system resources and can cause a DoS condition. The hacker often randomly spoofs the source IP address during these attacks (and others) for three reasons:
So it cannot be traced as easily to catch the hacker.
Falsifying multiple, random source addresses makes it hard to use routers to block the malicious traffic.
When the targeted host starts retransmitting the ACK-SYN sequence to a faked source IP address, if there is an active host at that forged IP address, it will respond with a NACK packet because it wasn’t expecting the second part of a three-way handshake from the target. This just adds to the victim’s congestion.
DoS floods can be difficult to impossible to stop with the current version of IP. In many cases, the affected sites have no defense, and can only wait for the hackers to stop the flood. IPv6 has built-in authentication mechanisms that should decrease DoS attacks.
During a port scan, hackers will often send the SYN packet to probe the target’s ports. If the ports are closed, the target computer will respond with an RST packet. Often, system logs (and many security logs, including the one in Windows) would not even note the connection attempt. Using SYN probes, a hacker could map all the open and closed ports located on a host (called a port scan) and begin probing exposed services.
Firewalls and IDSs can now capture such events, and if a port scan is noticed, the security device will deny all traffic from the originating host. So, hackers came up with other interesting combinations to send in a port probe that might not set off the security devices. One is called a Null scan and involves sending an initiating packet with no flags set. Another probe type involves setting three or more flags at the same time during the probe, called a Christmas Day probe. Another example is when a hacker sends a FIN packet to a target computer’s different ports, and is known as a FIN scan.
In most cases, if TCP is used to send the malformed probe, most host computers will respond with an RST packet. All of these tricks are efforts to find out which ports a PC has open without setting off security logs and firewalls. Luckily, most modern computer security defenses will note the port scan, no matter how it comes.
If you’re going to be a honeypot administrator, you, too, will need to recognize which flags sequences are normal and when they should occur. Any protocol analyzer you use should readily show which flags are and are not set. Any IDS you use should examine the flags set in any TCP packet, track what is expected, and note any abnormal conditions.
UDP is known as the quick but unreliable cousin of TCP. UDP is used in NetBIOS traffic, DNS queries, DHCP traffic, Windows Cluster Services, Microsoft Real Time Communications Server, and all sorts of other protocol traffic that we rely on every day. Figure 9-5 illustrates the UDP packet structure.
Figure 9-5: UDP packet structure
UDP is used by applications that don’t need to always have the data reliable and in the right order. Of course, any application needs the data it requested. But, in certain cases, not getting the data reliably isn’t going to shut down the entire application. The application can simply ignore the dropped data (such as is often the case in audio applications) or ask for the data to be retransmitted.
Another reason why an application might use UDP instead of TCP is that all of the data it is requesting fits in a 512-byte packet, the UDP maximum size by default. For example, if a DNS client requests resolution for a particular domain name, if it doesn’t get the answer the first time, it will resubmit the request or use another DNS server source. The answer it receives will usually fit in 512 bytes. Because of this, a DNS client can use UDP for its request and get a reliable, quick answer most of the time.
Although UDP packets can be large (upwards of 16,000 bytes) most UDP-using applications limit it to 512 bytes in length. UDP requires that all data in a single transaction fit in a single packet. You can usually modify your IP stack to send larger UDP packet sizes, but most routers won’t pass them, so you end up with dropped packets. Of course, UDP’s small packet size also makes it very fast. There is no overhead involved with setting up and breaking down communications; UDP just sends packets along.
ICMP was invented as a way for IP stacks, applications, and users to quantify and troubleshoot network connections. ICMP packets work at the network layer, and as such, don’t use IP packets for their transportation.
ICMP is known as the protocol for sending and receiving pings (ICMP Echo Requests and ICMP Echo Replies), but it is used for much more. Windows Active Directory uses it to determine slow link speeds. Network devices use it to determine if routers are down or if a network is suffering under a heavy workload. Hackers use it to fingerprint host OSs, in Ping of Death attacks, and to send malicious attacks such as Smurf amplification.
Ping of Death attacks (http://www.clavister.ru/support/kb/10067) happen when an attacker is able to send overly large ping requests. Ping requests are 64 bytes in length or smaller. Most ping utilities allow you to change the size to aid in troubleshooting connections. A Ping of Death attack sends a ping packet larger than the destination host can answer. Years ago, many IP stacks and routers were susceptible to this type of attack and locked up, resulting in a DoS condition.
In a Smurf attack (http://www.cert.org/advisories/CA-1998-01.html), the hacker sends a ping to a subnet’s broadcast address. On many networks, this will result in all hosts on the subnet responding to the ping. The attacker forges the source IP address, so that all responses are sent back to an intended victim, which isn’t expecting a ping reply. The hacker sends hundreds of thousands of these packets, overwhelming the target computer and possibly the entire network.
The Windows OS was never susceptible to Smurf attacks.
As a Windows honeypot administrator, you should become familiar with the internal workings of IP, ICMP, TCP, and UDP traffic, especially as they apply in Windows network communications.
Although TCP/IP is the most ubiquitous network protocol in the world today, it wasn’t always this way. Just a decade ago, NetWare networks ruled the world with Novell’s IPX protocol, Macintosh had AppleTalk, Windows used NetBIOS, and IBM “big iron” used SNA. Only an elite group of scientific and educational computers used TCP/IP as their primary network protocol. Then the Internet became everything computing had to offer, and it started to seem silly to run IPX or NetBIOS on the LAN when everyone needed TCP/IP to connect to the Internet. In short order, TCP/IP became the rule of the land.
In Windows NT, TCP/IP was installed as an optional protocol. By the time Windows 2000 was released, TCP/IP was the default protocol. Now, Windows Server 2003 would break without TCP/IP. Active Directory, DNS, and DHCP all depend on TCP/IP. Still, unless all your PCs and software programs are brand new, you probably have some NetBIOS traffic flying around your network. So, TCP/IP is the default protocol suite in Windows OSs, but you still need to learn all the session and application protocols that it uses.
As covered in Chapter 3, the NetBIOS ports 137 (UDP), 138 (UDP), 139 (TCP), and 445 (UDP and TCP) are the most common ports a Windows hacker expects to see. This is, of course, on top of the normal service traffic like RPC, HTTP, FTP, LDAP, telnet, SMTP, and so on. Today, most of these network protocols, including NetBIOS, run over TCP/IP. There are literally hundreds of other, not always as well-known, upper-layer protocols used by Microsoft Windows that run over lower-layer protocols.
In order to make your job easier, your network analysis tool should have protocol decoders (also called parsers or filters) to interpret the traffic for you. The protocol analysis tool should be able to recognize the traffic by type, name it (for example, Browser Announcement), and automate some of the hard work. Ethereal lists the following Microsoft-specific display filters in its latest version:
MS Proxy Protocol
Microsoft Distributed File System (DFS)
Microsoft Exchange MAPI (MAPI)
Microsoft Local Security Architecture (LSA)
Microsoft Network Logon (RPC_Netlogon)
Microsoft Registry (Winreg)
Microsoft Security Account Manager (SAMr)
Microsoft Server Service (SrvSvc)
Microsoft Spool Subsystem (Spoolss)
Microsoft Telephony API Service (TAPI)
Microsoft Windows Browser Protocol (Browser)
Microsoft Windows Lanman Remote API Protocol (LanMan)
Microsoft Windows Logon Protocol (NetLogon)
Microsoft Workstation Service (WksSvc)
Of course, Microsoft Windows has hundreds of application-layer protocols, such as Windows Media streams, Remote Desktop Protocol (RDP), ActiveX, and OLE. Again, you’ll need to be familiar with those that are known to be vulnerable and/or let the IDS do most of the hard work for you.
Now that we’ve talked about the OSI model and protocols, it is time to discuss how you can capture and analyze those protocol packets.