Using IPSec Security | Professional Windows Desktop and Server Hardening (Programmer to Programmer)

IPSec has incredible versatility. With appropriate setup planning, Windows IPSec can be used to protect any Windows 2000 and later computer, including workstations, domain controllers, web servers, and file servers. Successful IPSec takes a lot of setup planning. I'll use another example to illustrate the usefulness of IPSec.

Setup Planning

Here are some of the questions to consider:

What role will the two IPSec endpoints assume in the network and with each other?
Do you want to use IPSec to secure traffic between two endpoints or prevent all unauthorized traffic?
Figure out what services are involved and what ports need to be secured.
Using the IP rule and filter characteristics noted above, think about how your IPSec policy needs to be created.
Will IPSec be used to authenticate, for encryption, or both?
Do both endpoints recognize the same hashes and cipher algorithms?
What authentication methods are possible?

When to Use IPSec

IPSec is an excellent solution for the following scenarios:

Between two PC endpoints needing guaranteed traffic confidentiality and authentication
Between two local area networks needing access to each other over the Internet
As a requirement between critical computers and domain clients, ensuring that only managed domain computers and members can connect
Along with Group Policy to allow only certain groups to access sensitive computers
To allow only administrators to access management ports and consoles on remote computers
Between laptop user at home and the corporate firewall (VPNs of this nature should terminate at a point that allows the perimeter firewall and other computer defenses to inspect and scan the traffic — to prevent the introduction of malware into the corporate network)
To protect other insecure remote protocols — for example, to protect RDP traffic originating from insecure networks (currently, RDP traffic is vulnerable to man-in-the-middle attacks)
To isolate a secure network from an insecure network (i.e., network or security domain isolation) when sharing common physical network media
As a firewall to prevent unauthorized port connections

Example Scenario

Recently the author had an opportunity to manage a "Hack IIS 6" contest (located at www.hackiis6.com while the site was live) and security team. It involved a hardened IIS 6 honeypot server that the world was invited to attempt to hack. It was connected to a back-end SQL database and a side channel intrusion detection computer (see Figure 8-20).

image from book
Figure 8-20

It was remotely administrated using Remote Desktop Protocol using TCP port 48064. The following IPSec rules were created:

The web server was only supposed to allow untrusted attacks access to port 80. An IPSec rule was created that said to Permit traffic from any source IP to the web server's TCP port 80.
All other ports to the web server Required Security using certificates. This effectively removed over 130,000 different UDP and TCP ports from attacker scrutiny.
An IPSec rule was created to allow IPSec secured traffic (AH, and ESP integrity and authentication) from the remote management workstation's IP address from any port to the web server's RDP port, TCP 48064. It required a certificate to connect. Another similar IPSec policy was created to allow RDP over a non-default port to the SQL server to manage it.
An IPSec rule was created to Require Security using certificates between the web server (any port) to the SQL server's TCP port 14034 (i.e., a non-default SQL port).
Using IPSec on the SQL server, all other connections, other than from the web server to the SQL's server's non-default SQL server, were blocked (again using Required Security and certificates for all other ports). This allowed the SQL server to serve up just the one port that was needed between it and the web server. Another IPSec rule was created to allow the RDP remote administrator to connect to the SQL box, again using Required Security and certificates. All other combinations were blocked.
An IPSec rule was created to Require Security between the web server and the IDS server to allow log collection from the web server using the FTP service. Initially, File and Printer sharing were enabled to allow the IDS server to pick up the IIS server's web server logs, but this was felt to be too much risk for the contest. Instead, FTP services were installed on the IIS server and secured with IPSec.
The IDS server was locked down using IPSec. In order for the IDS server to capture all attack traffic headed to the web server, we could not block any ports completely. If we did, it would prevent the IDS server from sniffing network traffic. Instead we allowed all incoming traffic (so the IDS could sniff the traffic), but blocked all outgoing traffic (except to the one FTP port on the web server and one port on the RDP remote management machine). While not perfect security, it worked out pretty well.
The RDP remote management machine could only connect to the hacking contest web server location using port 80, and the non-default RDP ports pointed to each participating server.

Using IPSec, we were able to block most of the 130,000 plus UDP and TCP ports each involved computer had and significantly tighten all the remaining ports (except for the web server's port 80). We had hackers attempting to hack night and day. For over three weeks the web site sustained over 150,000 attack packets per second. In the end, the web site wasn't hacked.

The contest didn't prove that IIS 6 wasn't hackable. Anything is hackable, and IIS 6 is bound to have holes. What it did prove was that by following simple hardening steps and implementing the correct tools, IIS 6 can be made fairly secure against most attacks. The hardened IIS 6 server and its well-researched applications probably had more to do with the web server going unhacked, but I'm certain the stringent IPSec policies contributed significantly to the project's success.