Transport Protocols (Layer 4)

When programmers create applications that need to communicate across a network, they typically choose the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), or both protocols. These protocols help applications deliver data in either a reliable, connection-oriented (TCP) fashion or an unreliable, fast, connectionless-oriented (UDP) fashion. The PIX firewall tracks the TCP or UDP traffic traversing the firewall in slightly different ways, but both are monitored to provide users with stateful sessions across the firewall.

TCP Overview

TCP is a connection-oriented protocol. Connection-oriented protocols provide reliability that guarantees the delivery of data to its destination and that has enough know-how to retransmit missing data. TCP contains a defined state machine mechanism similar to a modem calling another computer. The following are the three states of TCP:

1. Set up the connection.

2. Send the data.

3. Disconnect the connection.

During the connection setup state, a process known as the three-way handshake takes place. This handshake helps the two communicating computers establish connection parameters that will be used during the transmission of data. The first step of the handshake opens a connection by sending a synchronization (SYN) request. This is sent to the destination computer, which in step two sends its own SYN request back with an acknowledgement (ACK) for the SYN it has received. In step three, the original computer responds to the SYN and ACK received with the last and final ACK. At this point the two computers are ready to send data. After the data transfer is finished, a FIN message is sent to disconnect the session.

For example, in Figure 5.1 Jack wants to communicate with Peter's computer. The following steps take place:

1. Jack sends a synchronization message to open a connection.

2. Peter's computer responds with its own SYN to Jack and an acknowledgement to Jack's SYN.

3. Jack receives Peter's SYN and ACK and responds with a final ACK back to Peter.

4. Data transmission begins.

5. The computers are disconnected by the FIN message being sent.

Figure 5.1. A three-way handshake.

How the PIX Works with TCP

The PIX is a stateful firewall that monitors the traffic flowing through it. The TCP protocol is very predictable, and the PIX system uses this to its advantage. Using the previous example, when Jack first communicates with Peter, the PIX records the connection information into its connection table. It then allows the SYN and ACK messages from Peter to pass back through the firewall to Jackand only Jack. After Jack sends the final ACK, the TCP session is established and a connection slot is established in the PIX that allows Jack and Peter to send data back and forth. If you were to view the connection table, you would see that a connection is being made between Jack and Peter. As they disconnect from each other, a FIN message is sent that closes each TCP session. The PIX monitors the traffic for this FIN message and uses it as a flag to remove the connection table entry created for the two. If Peter were to send more information to Jack after the FIN message has been sent, no entry would be found in the table and Peter's data would be discarded, thus protecting Jack from uninitiated traffic.

By default, the PIX firewall drops all initiated traffic from lower-level interfaces (Peter) to higher-level interfaces (Jack). A manual static entry can be added to override this, though.

Embryonic Connections

The process TCP uses is very predictable; hackers can use this to their advantage when trying to hack a system. For example a SYN attack occurs when a hacker sends hundreds or even thousands of SYN open connection requests to a computer, with the intention of never acknowledging those requests with an ACK. This leaves the victim computer wasting resources for each of those half-open connections. If too much of the victim's resources are wasted , the computer can fail, causing a denial of service. These half-open connections can be limited by using embryonic connection values on several of the PIX commands. Limiting the number of half-open connections helps guarantee that the computer won't be overloaded with false open requests.

UDP Overview

The User Datagram Protocol is a connectionless protocol. Connectionless protocols do not guarantee that data will be delivered to the destination through the use of SYN, ACK, or FIN messages, thus they're faster than TCP. If reliability is necessary, however, the application layer needs to monitor for missing data. UDP causes the PIX several problems. Because UDP does not use a three-way handshake or FIN messages, the PIX never really knows when to remove the connection slot entry between two computers. As a result, the PIX uses an idle timer to monitor whether traffic is passing between the two computers. If the timer expires before any traffic has passed, the connection is assumed ended and the connection slot entry is removed.