DOS COUNTERMEASURES

Because of its intractable nature, DoS more than anything should be confronted with multipronged defenses involving resistance, detection, and response. None of the approaches will ever be 100-percent effective, but by using them in combination, you can achieve proper risk mitigation for your online presence.

A Quick Note on Practical Goals

At the risk of sounding like defeatists right out of the gate, we want to make it clear that at least theoretically, DoS can never be fully prevented, and you shouldn't delude yourself (or your organization) that you will ever achieve 100-percent DoS immunity. Rather, we believe the goal of DoS mitigation is to maintain the best level of service for the largest number of customers at any given time. By setting this as a design goal, putting some hard metrics around "level of service" and "largest number of customers," you will achieve resiliency for your online applications and earn your customers' respect, even in times of crisis.

We also want to touch briefly on the organizational politics of DoS. Although most authorities define security as protecting the confidentiality, integrity, and availability (C, I, A) of assets, in our experience availability runs a distant second to the priorities of most IT security groups (who in many instances would be content if sensitive corporate assets were in fact unavailable as long as they remained confidential and retained their integrity). Combine this propensity with the fact noted earlier in this chapter that most modern DoS techniques are focused on capacity depletion, and you have a recipe for failure if DoS remains the sole responsibility of the organizational IT security group. Our advice: share accountability for DoS to the IT infrastructure/operations group that actually has their hands on the levers (network, systems) and should rightly be held accountable for managing capacity, whether impacted by DoS or just normal usage variations that result in availability issues. Another option in this regard is to place this accountability with the business continuity planning (BCP) team, who should be well versed in modern DoS tactics if they're worth their salt. Although it pains us to depart from the purist approach in which accountability for C, I, and A are logically grouped into one organization, very rarely have we encountered organizations whose approach to risk was mature enough to properly staff an availability function within security for an extended period of time.

Another area where organizational tensions can brew is accountability for application and infrastructure DoS. As is fairly obvious from the discussion in this chapter, infrastructure-layer DoS is the most common attack paradigm today, and most mitigation technologies and techniques are focused on those types of attacks. However, as we've also noted, application-layer DoS is coming into vogue , and this is an area that tends to remain outside the domain of typical IT infrastructure groups. Unfortunately, most software development teams consider DoS to be an infrastructure-layer problem that they don't have to engineer around (because the infrastructure team is covering it!), resulting in an accountability gap. Here's where the security team can provide some unifying glue by coordinating a cross-team effort to confront DoS holistically, across the infrastructure and application layers .

So much for our sojourn into organizational politics. Let's move on to discussing more tangible deflection , detection, and response measures you can implement to mitigate DoS attacks.

Resisting DoS

Here are our recommendations for deploying a DoS-resistant technical infrastructure.

Anti-DoS Products

First of all, if you have a mid- sized or larger online presence that is at least moderately important to the success of your business, we strongly recommend assessing off-theshelf technology for blocking DoS attacks. One such product that we've seen used in real-world environments with tremendous success is the Cisco Guard (formerly produced by a company called Riverhead Networks, which was acquired by Cisco in March 2004; see http://www.cisco.com/en/US/products/ps5888). Top Layer has also surfaced in anti-DDoS conversations we've had of late (http://www.toplayer.com). We've also heard good things about Juniper's line-rate-filtering capabilities on their routers, which might allow you to kill two birds with one stone. Both of these products employ techniques for blocking or severely restricting common DoS attack techniques such as SYN floods (through the use of SYN cookies, for example; see http://cr.yp.to/syncookies.html) and valid HTTP connection floods.

Tip 

Be careful when employing aggressive IP source blocking mechanismsif you block Internet superproxies such as AOL or major corporate networks, you may find yourself appearing just as dark to your customers as if the actual attack had been successful!

Capacity Planning

We also strongly recommend that you plan your online capacity appropriately to endure as much DoS as your business is willing to pay for beforehand. We've witnessed packet rates in the multiple millions of packets per second (pps) range at some top- tier online service providers, to give you some idea of the volume of DDoS that your company could face at some point. Of course, spending money wildly on capacity without rationally assessing the unique risks faced by your organization is not recommended. Work with your ISP and internal risk management functions to come up with a reasoned plan from both the server and network perspective.

Work with Your Internet Service Provider (ISP)

You can have the best DoS defenses in the world, but if your ISP sags under attack, it just won't matter to your online customers. Meet with your ISP technical staff, if possible, and understand what DoS countermeasures they employ, the nature of their own DoS capacity planning, and whether they've had any independent audits or testing done to determine their actual level of DoS resilience. The savvier ISPs will have thought about keeping malicious traffic off of their networks to begin with using inbound filters at their network edges; quiz them about such measures (and see our discussion of egress filtering in the next section). Also get chummy with your ISP's network operations center staff and their standard procedures for escalation of DoS attacks.

Hardening the Network Edge

Without going into excruciating detail on network equipment configurations that can severely curtail DoS attacks, we highly recommend reading "Cisco Strategies to Protect Against Distributed Denial of Service (DDoS) Attacks" at http://www.cisco.com/warp/public/707/newsflash.html. It discusses Cisco IOS configurations such as verify unicast reverse- path , filtering of RFC 1918 private addresses, applying ingress and egress filtering, rate-limit , and ICMP and UDP filtering strategies that should be basic common sense in the current Internet environment.

If you're not using Cisco equipment, be sure to ask your vendor what features they can implement to protect you from DoS. (For example, SYN Defender is a feature commonly used by customers of the popular firewall vendor Check Point.)

The remainder of this section will provide some clarifying detail around basic DoS mitigation recommendations:

• Block ICMP and UDP As we've noted, DoS attacks have traditionally attempted to leverage these protocols to achieve maximum abuse. Because neither is commonly used much anymore (at least for broad public access), we recommend heavily restricting these at the network edge (disable them outright if possible).

• Ingress filtering This means blocking obviously invalid inbound traffic, such as private and reserved address ranges that should normally never be honored as valid source addresses. For a good list of such addresses, see http://www.cymru.com/Bogons.

• Egress filtering If more ISPs would simply implement egress filtering, DoS would probably be a much less significant threat. Egress filtering essentially stops spoofed IP packets from leaving your network. The best way to do this is to permit your sites' valid source addresses to the Internet and then deny all other source addresses.

• Disable directed IP broadcast To prevent your site being used as an amplifying site (say, from a Smurf or Fraggle attack), you should disable directed broadcast functionality at your border router. For Cisco routers, you would use the following command:

   no ip directed-broadcast 

  This will disable directed broadcasts. As of Cisco IOS version 12, this functionality is enabled by default. For other devices, consult the user documentation to disable directed broadcasts. We also recommend reading "Stop Your Network from Being Used as a Broadcast Amplification Site," RFC 2644, a Best Current Practice RFC by Daniel Senie, which updates RFC 1812 to state that router software must default to denying the forwarding and receipt of directed broadcasts.

• Implement Unicast Reverse Path Forwarding (RPF) When Unicast RPF is enabled on an interface, the router examines all packets received as input on that interface to make sure that the source address and source interface appear in the routing table and match the interface on which the packet was received. This helps to cleanse traffic of packets with potentially modified or forged source addresses. See http://www.cisco.com/univercd/cc/td/doc/product/software/ios111/cc111/uni_rpf.htm.

• Rate limit Rate filtering at your border routers can be used to blunt the effects of DoS, although ultimately some customers will lose out if you pick the interfaces to rate limit injudiciously. Cisco routers provide the rate limit command to configure Committed Access Rate (CAR) and Distributed CAR (DCAR) policies to control the amount of traffic you are willing to accept on an interface. You can also use Context Based Access Control (CBAC) in Cisco IOS 12.0 and later to limit the risk of SYN attacks. Search http://www.cisco.com for more information on CAR and CBAC.

• Authenticate routing updates If you allow unauthenticated access to your routing infrastructure, routing table entries can be abused to deny service to legitimate systems or networks. Most routing protocols, such as Routing Information Protocol (RIP) v1 and Border Gateway Protocol (BGP) v4, have no or very weak authentication. What little authentication they do provide seldom gets used when implemented. This presents a perfect scenario for attackers to alter legitimate routes, often by spoofing their source IP address, to create a DoS condition. Victims of such attacks will either have their traffic routed through the attackers ' network or into a black hole, a network that does not exist.

• Implement sink holes An interesting mechanism for filtering invalid addresses such as bogons while simultaneously tracking from which segments they originate is the notion of sink holes . By configuring a sacrificial router to advertise routes with bogon destination addresses, you can set up a central "trap" for malicious traffic of all types. For example, we know one large network provider that implemented a sink hole in January of 2003 to identify systems on the network that were infected with the Slammer worm, which causes machines to spew packets to randomly selected destination addresses. For greater detail, we recommend reading the excellent presentation by Cisco and Arbor Networks on the topic (see http://research. arbor .net/downloads/Sinkhole_Tutorial_June03.pdf).

Hardening Servers

Although the network is usually the first casualty in a DoS attack, the ultimate target of DoS is the information housed on servers, so they, too, must be considered when hardening against DoS. For a good overall discussion of DoS-prevention configuration recommendations for Windows, see Hacking Exposed: Windows Server 2003 (McGrawHill/Osborne, 2003).

Keep Up with Patches This is simply common sense wisdom that we've dispensed time and again in this book, and as you might've guessed, it applies to DoS as well. Many of the recommendations we make next rely on running the latest version of a given vendor's IP stack.

System-Level SYN Protection As of this writing, most modern operating systems have enabled SYN flood detection and prevention mechanisms. See CERT Advisory CA-96:21, "TCP SYN Flooding and IP Spoofing Attacks," for a list of operating system workarounds and patches. The standard approach is to dial up or down the queue of half- open connections they're willing to accept, in addition to reducing the connection-establishment timeout period.

Due to the ongoing prevalence of SYN attacks, other solutions have been developed to deal with this DoS condition. For example, modern Linux kernels (2.0.30 and later) employ an option called SYN cookie . If this option is enabled, the kernel will detect and log possible SYN attacks. It will then use this cryptographic challenge (SYN cookie) to enable legitimate users to continue to connect even under heavy attacks. For more information on SYN cookies, see http://cr.yp.to/syncookies.html.

Microsoft Windows NT 4.0 SP2 and later employs a dynamic backlog mechanism (see Microsoft Knowledge Base Article Q142641). When the connection queue drops below a preconfigured threshold, the system will automatically allocate additional resources. Thus, the connection queue is never exhausted. For more information on configuring Windows servers to resist DoS, see Hacking Exposed: Windows Server 2003 and/or http://support.microsoft.com/default.aspx?kbid=120642.

Miscellaneous System-Level Configurations Some operating systems can be configured to discard broadcast ICMP ECHO packets that would create a Smurf or Fraggle-type attack.

To prevent Solaris systems from responding to broadcast ECHO requests , add the following line to /etc/rc2.d/S69inet:

 ndd -set /dev/ip ip_respond_to_echo_broadcast 0 

To prevent Linux systems from responding to broadcast ECHO requests, you can use kernel-level firewalling. Firewall packages vary by kernel revision, with iptables and ipchains being the most common. For more information on setting up these packages, visit http://www.redhat.com/support/resources/networking/firewall.html. The following commands can be used with iptables:

 iptables -A INPUT -p icmp -d 192.168.1.1/32 -j DROP iptables -A FORWARD -p icmp -d 192.168.100.255/24 -j DROP 

The first command will drop any ICMP messages addressed to the host itselfin this case, 192.168.1.1. The second command will prevent the system from relaying broadcast ECHO requests to the internal network if the system is being used as a firewall or router.

FreeBSD versions 2.2.5 and later disable directed broadcasts by default. This functionality can be turned on or off by modifying the sysctl parameter net.inet.icmp.bmcastecho.

AIX 4. x disables responses to broadcast addresses by default. The no command can be used to turn this functionality on or off by setting the bcastping attribute. The no command is used to configure network attributes in a running kernel. These attributes must be set each time the system has been restarted.

All Linux/UNIX variants should disable the echo and chargen services to prevent simple DoS attacks that point these services at each other. To do this, disable echo and chargen in /etc/inetd/conf by putting "#" in front of the service.

DoS Testing

It should come as no shock that the authors of this book recommend attacking your Internet- facing infrastructure just like real malicious hackers, and that includes DoS. Of course, few organizations have the ability (or lack of ethics) to leverage the distributed power of zombie networks, so they have to find test harnesses that can emulate a similar load. Some of the tools available in this area include RadView's WebLOAD (see http://www.radview.com). A good updated list of current software load and performance testing tools can be found at http://www.softwareqatest.com/qatweb1.html.

Detecting DoS

You really don't want your customers to find out about a DoS condition before you do. Therefore, gaining as much early warning as possible is critical to maintaining at least the appearance of resiliency in the face of a real DoS attack.

Keep Tabs on Malware

The earliest warning you're likely to get probably won't come from targeted DoS detection activities, but rather by keeping tabs on the activities of the malware community. All of the recent significant DoS attacks were driven by virus/worm infections whose payloads were disassembled well in advance of the coordinated DoS event. For example, the Blaster worm and MyDoom.B were revealed to be DoS bots with payloads set to fire off on a certain date. This allowed the targeted companies (Microsoft and SCO, in these cases) to analyze the nature of the impending DoS logic and take steps to mitigate it (some of which we'll outline in our section on responding to DoS, later in this chapter). One of the best ways to keep up to date on malware-driven DoS is to subscribe to a good security newsletter, such as Bugtraq, or antivirus vendor information services.

DoS Detection Technology and Techniques

Of course, we also believe in deploying DoS detection technology. Although a lot of intrusion detection system (IDS) vendors claim to detect DoS attacks, in our experience most IDS vendors ' signature-based approach just isn't up to the task. Anomaly detection is what you want for DoS, and the product we recommend is Peakflow from Arbor Networks (http://www.arbornetworks.com). Deployed at the network perimeter, Peakflow can detect variations in traffic at thresholds configured uniquely for each site, giving an instantaneous warning if traffic anomalies such as DoS appear.

On individual hosts , you can issue the netstat-na command if it is supported by your operating system. If you see many connections in a SYN_RECV state, this may indicate that a SYN attack is in progress. Although if the system is under genuine DoS, it may not respond in a timely manner to console input.

Scan for DoS Bots on Your Networks

Especially if you are a service provider with fat Internet pipes that could be directed against others, you really don't want to be a source of DoS. It is therefore important to find and eliminate DoS clients and bots on your networks.

The best way to identify DoS client/bot infections is to deploy antivirus technology across your entire infrastructure (and, yes, that means datacenter servers as well, even if they aren't supposed to be high-risk for infection because they aren't used for browsing the web or reading e-mail). We recommend reading our discussion of malware prevention and detection in Chapter 13 for more information, as well as checking out the Windows and Linux/UNIX chapters to understand tools and techniques for preventing your systems from getting hacked in the first place.

Some freeware programs for detecting individual DDoS clients that are worth checking out include Foundstone's DDOSPing (http://www.foundstone.com), Zombie Zapper by Bindview's Razor team (http:// razor .bindview.com), and find_ddos (http://www.nipc.gov) by the National Infrastructure Protection Center (NIPC).

Responding to DoS

Although it is important to understand how to resist and detect DoS, it is perhaps most important to understand what to do should your site come under attack. Consider DoS to be inevitable for a successful online business, lay out plans in advance, and conduct fire drills to ensure you've got it right. We lay out some best practices for consideration in this section.

Plan and Practice Your Response Process

In the heat of an actual attack (which usually comes in the wee hours of a weekend ), the natural tendency is for chaos to reign. You'll have no shortage of would-be experts shouting advice on how to proceed, some of your best engineers may be working on zero sleep and coffee fumes after a multiday attack, and management will be sounding a consistent drumbeat along the lines of how soon you can get their business back online. Trust us, you do not want to face this sort of situation unprepared. Formulate an incident response plan in advance, and conduct fire drills at least once per year if you don't actually get attacked , because no DoS response plan is worth the paper it's written on unless it has been through trial by fire. Some additional advice in this regard follows .

If they exist, leverage documented incident escalation procedures in your organization. If these procedures don't exist, you should drive management to create them. We'd caution against trying to build a generic incident response function within the security orgyou will be distracted from your core mission of protecting the confidentiality, integrity, and availability of valuable organizational assets. Security should certainly be a key adviser and stakeholder in the design of the process, but in our experience the vast majority of online commerce incidents don't involve security, so security shouldn't be the primary driver.

As for the components of a good plan, we'd recommend mimicking the several available security incident response reference plans available on the Internet. The Computer Emergency Response Team (CERT) has some good advice at http://www.cert.org/csirts/Creating-A-CSIRT.html, and Microsoft has published some of their internal response processes at http://www.microsoft.com/technet/itsolutions/msit/security/msirsec.mspx. From our perspective, key points to consider are rapid escalation, aggressive triage, thorough investigation, carefully orchestrated communication through resolution, and collaborative post-mortem.

When an actual attack does arise, execute on the plan and don't deviate. Again, the point of planning and practicing in advance is to let clearer heads prevail. There will sometimes be great pressure to take radical steps during an attack that will appear to be quite unwise in hindsight. A solid response plan can mitigate against such knee-jerk reactions .

Finally, hold a post-mortem after the DoS attack to determine what worked and didn't work with your plan. By continuously improving your response process, you can relegate DoS attacks to minor annoyances rather than major catastrophes.

Filter or Redirect Offending Traffic

Although spoofing DDoS attacks have made this approach less effective, many of the organizations we have consulted for still rely heavily on blocking traffic from offending IP addresses. This is for good reason: Blocking is 100-percent effective in mitigating DoS from that source address. Plus, current DoS trends are moving toward unspoofed HTTP connections from distributed zombies , so there is greater potential to identify real attackers versus spoofed SYN flooding.

Tip 

Implement blocking as far upstream as possible, because blocking downstream may still leave your infrastructure drowning in packets. This may require coordination with your ISP. See the upcoming section titled "Call Your ISP and Initiate traceback" for more suggestions.

As we noted in our previous discussion of DoS resistance in this chapter, anti-DoS technologies can greatly assist here. For example, the Cisco Guard product we recommended implements multiple layers of filtering to weed out bad traffic from good. Manual process also works well here, but it can take more time and is usually less granular. For example, simply deploying a router ACL that blocks an offending address or network can quickly alleviate load from DoS and free up access for legitimate customers.

Caution 

Avoid blocking Internet super-proxies such as AOL; otherwise , you may find yourself creating your own DoS condition for many customers.

Besides filtering, you should also consider deploying sink holes on your network to redirect malicious DoS traffic. This concept is sort of like a DoS honeypot that shunts malicious traffic away from production infrastructure and permits more time for analysis, traceback, and so on. For more information about using sink holes, see http://www.arbornetworks.com/downloads/research36/Sinkhole_Tutorial_June03.pdf.

Call Your ISP and Initiate traceback

If your site comes under attack, you should quickly contact the network operations center (NOC) of your ISP(s) and attempt to identify the source. Keep in mind that it may be very difficult to trace the attack to the perpetrator, but it is possible. You or your ISP will have to work closely with the ISPs who are the source of the attack. Remember, it is frequently unwitting customers of these other ISPs who are the culprits in DDoS. Nevertheless, these ISPs are usually best positioned to filter the traffic coming out of their network than you are to blindly filter inbound. If you are a service provider yourself, you should have handy a list of NOC contacts at all major ISPs (especially non-U.S. providers where most DoS attacks originate) so that you can perform this service for your customers who are being DoS-ed. Make sure these contacts are capable of responding and taking authoritative action, even outside of business hours (we once wound up futilely talking to the U.S-based sales department of a Korean ISP when attempting to chase down a DoS attack originating from that region).

The tools and techniques of traceback are somewhat manual and specific to each service provider. By systematically reviewing each router, determining the interface on which spoofed packets are being received, and then tracing backwards , the source of an attack can be determined in many instances. For example, to help automate this process, the security team at MCI developed a Perl script called dostracker that can log into a Cisco router and begin to trace a spoofed attack back to its source. Of course, this program may be of limited value if you don't own or have access to all the routers involved.

We also recommend reviewing RFC 2267, "Network Ingress Filtering: Defeating Denial of Service Attacks Which Employ IP Source Address Spoofing," by Paul Ferguson of Cisco Systems and Daniel Senie of Blazenet, Inc.

Move the Target

DoS zombies typically use either static IP addresses or a single canonical DNS name to target victims. In response to repeated DoS attacks focused on the http://Microsoft.com service, it was later publicly revealed that Microsoft took a couple of steps with their DNS to give it some breathing room against these attacks. First, to deflect static IP address-targeted attacks, Microsoft simply changed the IP address for http://Microsoft.com in its DNS and also shortened the time to live (TTL) on the targeted domain name in advance of scheduled DoS attacks so that DNS clients on the Internet would rapidly receive updates to the new service address (of course, setting an aggressive TTL produces more load on DNS servers, so don't go overboard here). For other attacks, Microsoft also set up a CNAME entry in their DNS to point DoS attacks toward another service provider's infrastructure (for example, Akamai's HTTP caching network, http://www.akamai.com). Microsoft also simply removed one targeted DNS name from the DNS entirely, which essentially caused zombies to die without uttering a single DoS packet when they failed to resolve the name, and also spared the rest of the Internet a huge wallop of traffic. Of course, in this instance, the domain name Microsoft removed was a legacy name that was no longer used and therefore expendable.

Cut Over to Alternate Infrastructure or Application Modes

Because modern DoS attacks are focused on capacity, and at least in theory you will never be able to buy more capacity than what can be directed at you over the Internet, you should have an insurance policy to fall back on if such a doomsday attack occurs (and we've seen some DDoS attacks that would qualify as closemultiple million packet-per-second bit rates!so don't think it isn't possible). For example, as we noted with Microsoft in the previous section, they have relied on the additional capacity of Akamai's HTTP caching network to help absorb DoS attacks in the past. Similar services are available from service providers like Savvis Communications (http://www.savvis.net).

At the application level, consider designing different application modes that can be switched to during an application-layer DoS attack. For example, an online application that normally doesn't authenticate users (say, a corporate information portal) could be switched to a cookies-only mode that would require some minimal interaction from legitimate customers, but would derail automated bots. This is a rather simplistic example, but if you keep in mind the goal of maintaining the best level of service for the largest number of customers at any given time, your applications will be more resistant to DoS by design.

Finally, for those systems on your own network that are found to be emitters of DoS traffic, quarantine them as rapidly as possible to a network that is specifically designed to support sick machines and get them healthy again. For example, you might consider enforcing a redirect across all protocols to a web server that describes information on quarantine processes and how to get back online again.