Gigabit intrusion-detection systems

In our tests of five leading network intrusion-detection systems and the popular open source Snort, performance was spotty during baseline testing and degraded by as much as 50% on some products when we opened the throttle to gigabit speeds.

Our first step was to run 28 well-known attacks against each product in an untuned state on a wire that had no other traffic running on it. Most products detected only about half the attacks.

When the systems were tuned, most products caught an additional two or three attacks, but still missed a good number of them.

IntruVert's IntruShield 4000 was a bright spot. It detected the greatest number of attacks in every test (see the performance chart), and wins the Network World Blue Ribbon Award. A newcomer to this market, IntruShield is a well-designed, and feature-rich.

Internet Security Systems' RealSecure Gigabit Network Sensor Version 7.0 didn't detect as many attacks as IntruVert's product overall, (16 out of 28 at baseline with no tuning and 25 with tuning), but deserves the runner-up prize because its ability to detect attacks did not change at gigabit speeds. The other three commercial products tested at Miercom's lab facility in Princeton Junction, N.J., were Dragon IDS Server Appliance and Dragon IDS Sensor Appliance; Intrusion's Intrusion SecureNet; and, Symantec's (formerly Recourse) ManHunt Version 2.11. We also tested the open source package, Snort on Acid.

Performance tests

Our primary focus was to determine how well these products performed under a gigabit traffic load, which was 970M bit/sec in our tests. We ran the tests at slightly less than a full gigabit load to ensure that the link was not overutilized and all our attacks could get through (see How we did it).

In our baseline tests with no traffic, we did not tune the systems in any way, but we did turn on all signatures and protocol anomalies. We delivered 28 attacks to each system, including commonly known denial-of-service, surveillance and probe attacks, and attacks, such as Stick and Fragrouter, designed to evade an IDS system (see Attack List).

IntruShield 4000 detected the highest number of attacks - 24 out of 28. Dragon, RealSecure and Snort each caught 16 of the 28 attacks. ManHunt detected 14 attacks, and SecureNet caught 11.

A key factor in IntruVert's strong showing is a good implementation of signature-based attack detection (in which packets' contents are compared against a database of known attack patterns), and protocol anomaly detection (PAD) (in which the product verifies that a traffic flow is not violating its defined protocol - signaling suspicious activity.). Except for Snort, all the products supported both techniques, but IntruVert married the two technologies especially well.

Overall, the products caught about half the attacks. What accounts for this lackluster showing on a nontuned system is that some vendors turn off signatures to heighten performance. Vendors also make this trade-off so that administrators are not overwhelmed by the many false-positive alarms they receive before systems are tuned (see review).

The extent of an IDS's signature database also is a factor. The more attack signatures a product supports, the better its rate of detection without system tuning. For example, although ManHunt supports a signature database and PAD, it relies more on the latter. If an exploit or attack follows protocol then it's not detected unless the product has a signature to catch it. ManHunt supports a small signature database and doesn't do as well in this type of test.

We next ran the set of attacks that each product detected at baseline traffic levels against these still untuned products, but filled the pipe. RealSecure caught 16 out of the 16 attacks, ManHunt caught 13 out of 14 attacks, and IntruShield 4000 caught 21 out of 24 attacks. Dragon and Snort had the poorest overall showing, catching only 3 out of 16 attacks and 6 out of 16 attacks, respectively.

Tuning helps, but not much

This same set of tests was conducted again on tuned systems. Tuning meant that the vendor could tweak any signature code included in the product's database to let it catch or identify an attack correctly. The vendor could change User Datagram Protocol (UDP) to TCP, or vice versa to catch a Back Orifice attack. Or it might decrease thresholds for the number of TCP connects to catch an NMAP attack. It also might turn off processor-intensive engines, signatures and features to enhance performance.