Testing | Protect Your Information with Intrusion Detection (Power)

Each new release of security tools (including intrusion detection systems) is more sophisticated than the previous one, which increases the risk of failure due to more implementation bugs. However, clients do not normally test purchased software (or if they do, they do not do it thoroughly enough). Why does this happen? Quite often, the customers tend to rely on the manufacturer or vendor (especially if this is a well-known brand, such as Cisco or Sun). Some customers do not consider this task a very important one, while others simply are short on time and human resources. For example, I have many times encountered the situation in which the client demanded that a purchased system be brought into routine operation within a month. The quite a reasonable recommendation to test the system in order to analyze its behavior in the customer's network often caused indignation and misunderstanding: "What for? We paid good money; it should work fine." Furthermore, the lack of testing experience is also a rather important aspect.

At the time of writing this book, there were no standards for testing intrusion detection systems. However, certain steps in this direction have already been taken, and research in this area continues. This is due to the fact that the technology under consideration is relatively new, and as of yet, developers have not achieved a mutual understanding. The lack of a mathematical basis for intrusion detection technology also prevents researchers from complete and detailed testing of the tools. Ways of testing that utilize statistical methods are those that are being studied in the most detailed manner and that have been developed most extensively. Such systems include profile-based intrusion detection systems. It is important to note that most research work in the field of testing intrusion detection systems has been carried out in the last three to five years.

There are virtually no open sources that provide descriptions of testing methods, since each organization performing tests is using its own testing methods. These methods are the commercial, intellectual property of the individual company, and therefore are kept top secret. The only exception to this rule is the Lincoln Laboratory, which is developing a standard for testing network-level intrusion detection tools. However, access to these materials is limited, they are very hard to obtain, and quoting from them is prohibited. If you are interested in becoming acquainted with these tools, visit the following web server (with limited access): http://ideval.ll.mit.edu

Quite recently, a new evaluation standard appeared - the Open Security Evaluation Criteria (OSEC), which is intended for testing and evaluating information security tools. The evaluation criteria for network intrusion detection systems (http://osec.neohapsis.com/) were the first to be released.

The lack of a unified testing methodology results in a large variety of testing methods, since each organization develops its own. Because of this, some quite interestuing situations, in which the same system shows different results in different tests, might occur. This happens because each test lab differs not only in the tests performed, but also in the environment in which the testing takes place, in the qualification level of the testers, the priorities of the evaluation criteria, and so on.

Sometimes there are even more serious situations. It is no secret that each manufacturer wants its product to be praised as the best, and IDS manufacturers are no exception. Certainly, if you are rated as the market leader, it is much more probable that the customer will choose your products. Not all companies can afford full-feature testing of the security system they are going to purchase. This is why manufacturers themselves often become customers of so-called "independent" consulting companies, test labs, and so on (although they certainly try not to advertise this fact). As a result, the analysis can not be considered objective, since it is naive to think that the company that sponsored the testing will not be proclaimed the market leader. Without mentioning specific names, I would like to mention that, recently, I have encountered reports from "respectable" consulting firms, released at the same time, but that give different companies the title of market leader. Also note that it is practically impossible to prove that a specific consulting firm is not correct, since their testing methods are their own "know-how," which must not be revealed, and therefore can not be checked for correctness. This is why it is so important to develop a unified method of IDS testing and evaluation.

Of course, for the moment, there are many good publications dedicated to IDS testing, such as [Shieply1-01], [Yocom1-00], [Jackson1-99], [Denmac1-99], [ICSA1-00], [Newman1-98], [Hurwicz1-98], [SC1-00], and [NSS1-02]. However, all of these and similar tests must be interpreted critically, just because all of them are not conducted in your network and can not take into account all the specific features of your organization. Moreover, a system that was considered by the testers to be the best one might be unsuitable for your organization. As was shown earlier, each customer must have their own priorities - parameters that are vitally important for one customer might be of little or no importance to another. For example, at the beginning of the test described in [Shieply1-00], the NFR system lost 2 million frames during the first 48 hours of operation, while the eTrust system showed that it requires a processor 4 thousand times as powerful as your 700 MHz Pentium III. Because of this, you should not rely completely on the manufacturer's advertisements, but rather test all the facts and parameters yourself.

How should you perform the testing? This question is the one most frequently asked by security administrators who do not want to rely completely on the vendor's word. All tests can be classified according to the above-described groups of criteria, for example:

Intrusion detection. This class is one of the most important ones, and these tests are conducted by all test labs. As I have already mentioned, it is not the number of signatures that is of primary importance, but rather the efficiency of detecting attacks in normal traffic. The same category of tests includes the detection of intrusions typical for your network, the capability of customizing existing attack signatures and creating new ones, etc. If you can not afford to check all possible attacks, at least try to check the attacks from the SANS Top 20 list - The Twenty Most Critical Internet Security Vulnerabilities [SANS1-02]. One might suppose that manufacturers would pay the most attention to these attacks. However, as was shown in [NSS1-02], this is not always so, which is rather surprising.
Performance. The efficiency of the IDS is not limited to the ability to detect attacks in normal traffic. A well-designed IDS will show quite good results under conditions of stress tests, during which the traffic intensity is significantly higher than under normal conditions. For example, the NSS laboratory [NSS1-02] conducted its tests with workloads of 25, 50, 75, and 100 Mbit/sec, while the Miercom company tested systems with workloads of 40, 60, and 90 Mbit/sec [Yocom1-00]. Only those intrusion detection systems that are capable of detecting attacks in heavily loaded networks can be considered efficient. Packet length is yet another parameter that must be considered during tests. When testing network-level intrusion detection systems, the packet transmission mode with a minimum packet length for each protocol is the most complex test, allowing you to test IDS functionality under the toughest conditions. For example, if we return to the NSS test, there are 3 tests that are used to evaluate this criterion - ideal conditions (the length of all IP packets is the maximum, and is equal to 1514 bytes for the Ethernet), the worst conditions (the IP packet length is equal to 64 bytes) and normal conditions, under which the average packet length is about 300 bytes. By the way, all things being equal, IDS will work more efficiently in FDDI networks than in Ethernet networks. This relates to the fact that MTU value for FDDI is equal to 4,352, while for Ethernet it is 1,500 bytes (see Table 10.3).
Self-protection. Being a security tool, the intrusion detection system must not itself become a cause of lessened security of the corporate network that it is meant to protect. Therefore, the third class of tests should include the self-protection capabilities of the intrusion detection system - i.e., checking its operation in stealth mode, protection of the collected data and the data transmitted between the sensors and the management console, the capabilities of role-based access control, etc. There is also another intermediate class of tests intended for checking the system's ability to detect attacks, specially designed to bypass the IDS (Stick, Snot, ADMutate, etc.).
Manageability. This class of tests is rather important, especially in large, geographically distributed networks that contain several dozens of sensors.
Functionality. This class of tests includes all the other tests that were not included in the other categories, and this approach is justified. Manufacturers, in order to attract as many customers as possible, try to provide not only intrusion detection functions, but also a wide range of other functional capabilities intended to achieve customer satisfaction.

The simplest example of a test bench for evaluating network-level intrusion detection systems is shown in Fig. 9.20. Note that for the sake of simplicity, this variant does not provide testing for the IDS's integration with other security tools, various schemes of sensor management, and so on.

click to expand
Fig. 9.20. An example of a test bench for evaluating network intrusion detection systems

Various software tools can be used for testing intrusion detection systems. For example, quite an interesting solution is provided by the IDS Informer system from Blade Software. However, if you can not afford to purchase such specialized testing systems, you can use normal security scanners to imitate a hacker's activities. If this method is also not available, you can download exploits from any hacker site and use them to test your IDS. When using this approach, remember that if you undertake such an attack without proper care, you might cause your network some significant damage. Therefore, it is recommended that you run such tests in a specially prepared, isolated test environment.