What a Penetration Test Will Not Tell You

Many of us have been involved with pen testing either as a tester or as a client, or at least have been contacted by consultants who offer to perform a pen test for us. Pen tests are just what they sound likea test to see whether you can penetrate the network. Done correctly, a pen test has huge value; and even done incorrectly, it can be quite exciting to take part in. However, most of us are not paid to break things, but to protect them and ensure that the services they provide indeed work. On that note, you need to know several things about penetration testing.

First, most system administrators do not usually make very good penetration testers. Part of the reason for this is that they have already closed all the holes they know about; otherwise , they would be lousy system administrators. In addition, being a good penetration tester requires the ability to think like a criminal. After all, the objective is to demonstrate what an attacker would do. Most of us have been taught from a very early age to be good law- abiding people and are simply not good at thinking up very plausible and innovative criminal schemes. There are some people who are very good at doing so, however. Some of them are criminals, and you should stay away from anyone who brags about criminal activity. However, there are a lot of people with these skills who are "white hat" (as opposed to "black hat," or criminal) hackers. The "white hat" hackers have a reputable firm behind them, and have usually been vetted, and they can be hired through a number of different consulting firms.

Second, a penetration test gone wrong can have dire consequences for the stability of your network. For example, some of the tools used by attackers (white hat and black hat) are designed to probe a network for vulnerabilities. With some vulnerabilities, particularly denial-of-service (DoS) vulnerabilities, it is very difficult to tell prior to testing whether a system is vulnerable. Consequently, some of the tools simply fire off the attack. If the system still responds afterward, it was not vulnerable! Using such a tool incorrectly against a production network could have the effect of putting the entire network out of production, very quickly. Even in the absence of such follies, using attack tools and exploits against a system could misfire, destabilize a system or the entire network, or have other unintended consequences. A professional knows where to draw the line and how far she can push the network without breaking it, whereas an amateur usually does not.

Third, pen testing requires specialized tools, tools that, in many cases, are not commonly available. Although some system administrators are perfectly capable of writing these tools, usually the time spent doing so is time better spent protecting the network.

Fourth, the most important part of penetration testingwriting up the conclusionsis actually rather tedious . In the following narrative, notice just how detailed the descriptions of the attacks are. Keep in mind too that we will deliberately simplify some things to avoid giving a complete recipe for how to attack a network. By contrast, to write a proper penetration testing report, the pen tester must keep extremely detailed notes and then spend hours putting them together into a usable document. Unless the report has enough detail to allow someone to follow the attack, it is of little value.

All this boils down to one thing: although it is very important to know how attackers operate , the types of operational practices they rely on, and the attacks they use, being able to actually perform these attacks is not required of most system and security administrators. It is one thing to appreciate art, it is an entirely different matter to be able to create it. Pen testing is an art. My advice is to leave it to people who have the skills, mindset, and time to learn to do it well. Usually, this task is best left to consultants because they have the skills, the tools, the mindset, and are not tainted by prior knowledge of your network. Talk to colleagues in other companies, on mailing lists, and in newsgroups and learn who the really good pen testing consultants are. Bear in mind also that application pen testing and network pen testing are two completely different tasks . Just because someone has been able to find vulnerabilities in some particular product does not mean they are competent to attack a network. Look for firms that specialize in network pen testing, if that is what you are interested in. Application assessment is also available, but is a different type of engagement, with different goals, and is primarily useful for organizations that develop software in-house or that use custom software.

If you are actually charged with contracting for a pen test, what are the caveats to look out for? First and foremost, and we cannot emphasize this enough: Ensure that whoever signs the contract for a penetration test has the authority to authorize someone to break into the network, to grant a "get out of jail free" card .

There are people currently serving prison time because they broke into systems they thought they were authorized to break into, only to find out later that they, in fact, did not have this authority. The same fate may await someone who contracts someone to break into a system without having the authority to do so.

Second, ensure that you have sufficient nondisclosure agreements in place, and that the consultants are not allowed to retain any company-sensitive information, except under extremely strict data-protection standards. Nothing could be worse than paying someone to break into your network, only to find out that the network where that person stored the blueprint for how to do so was subsequently compromised. Or, worse , that this person took that information and sold it to someone else.

Third, treat the report as a healthy infusion of paranoia . This is especially true with a report from a black-box test (one where the testers do not have access to inside information, such as source code and account lists), which shows what an attacker with no inside information could do One of the worst mistakes a security administrator can make is to assume everything is okay.

Finally, be aware of the mythical "your network is secure" statement. With alarming frequency, consultants present a report that claims that a network is secure, based on the fact that they were unable to get into anything. This does not mean your network is secure . If a pen tester tells you that your network is secure, all he is saying is that he is not competent enough to prove that it is not. Your network is not secure. It is simply protected against the vulnerabilities and problems the pen tester knows how to exploit. The contract should specifically spell out what constitutes a successful break-in as well as the reduction in compensation if the pen tester is not able to achieve one. Most importantly, the output from a vulnerability analysis tool does not constitute a penetration test.