A vulnerability rated as a low risk this morning could turn into your worst nightmare tonight. To meet the ever-increasing speed with which exploits are written and propagated, traditional network-based vulnerability scanners have morphed into more full-scale vulnerability management products.

In our latest Clear Choice Test of eight products - assessing their accuracy in pinpointing holes in the network and their usefulness in addressing those vulnerabilities - we found vulnerability identification success rates are still low across the board and the scans can wreak havoc on wireless access points. They also can do damage to some printers, and can suck up network bandwidth and CPU utilization on target machines (see How we did it).

---

Getting a second scanning opinion
Citadel focuses on vulnerability remediation
How we did it
Archive of Network World reviews
Subscribe to the Product Review newsletter

---

Vulnerability remediation and tracking are the major management features added to these products since our last test, providing mechanisms to assign and alert administrators to new vulnerabilities. These additions range from providing vulnerability remediation information to offering full-blown ticketing systems that automatically verify if an issue has been fixed.

Business analysis features have been included in many products. With this functionality, assets can be given values - in terms of cash or business-critical value. How vulnerabilities potentially could affect business and give management a more accurate picture of the company's overall security posture can be correlated. A critical vulnerability on the core, Internet-facing system that generates revenue should be treated differently than a critical vulnerability on a system inside a test network that's isolated from the rest of the company, for example.

The companies that provided products and/or services for this test are Lockdown Networks, nCircle Network Security, PredatorWatch, Qualys, StillSecure, Tenable Network Security, TraceSecurity and Visionael. EEye Digital Security, Internet Security Systems, Foundstone, NetIQ, Bindview and Harris declined. We also tested Citadel's Hercules (see story) and Sunbelt Software (see story), but because they offer no scanning module or management features, respectively, we could not directly compare them.

Qualys' QualysGuard is our Clear Choice winner based on its accuracy and strong management capabilities. NCircle's IP360 comes in second, only slightly trailing Qualys in vulnerability identification and general ease of use. Visionael Enterprise Security Protector and Lockdown's Auditor also rose to the top based on their developing management capabilities.

QualysGuard 3.3

Click to see: QualysGuard

QualysGuard - one of the two vulnerability assessment services we tested - has a 1U appliance that sits on your network and lets Qualys scan your internal subnets. Setup is easy, and the quick start guide will have you scanning in no time. Because it is provided as a service, the Qualys team seamlessly adds the vulnerability checks.

Our discovery assessment focuses on how well the products find and identify systems, system software and services running on the network. Our accuracy measurement takes into account how well the product identified vulnerabilities that existed on a sample of lab systems (see "How we did it" ).

Qualys scored highest in our operating system identification checks and was the only product to correctly identify the wireless access point. It performed as well as any of the other products in the vulnerability accuracy tests, but still reported some false positives and false negatives. It did perform strongest among the products in identifying Windows system vulnerabilities, though.

Scan impact was low from a network perspective, but we did need to restart a Red Hat Enterprise system that became completely unresponsive after the scan.

Overall, QualysGuard is very flexible and easy to use. IT staff and/or corporate executives can be given varying levels of access to system groups and reports. Scan and report templates provide flexibility in the types of checks that are performed and how the results are viewed.

Remediation policies can be configured to automatically assign tickets in the Qualys ticketing system to defined individuals based on scan results. Qualys could improve on remediation if it added some preemptive notification mechanism to tell IT folks they have been assigned a remediation task.

In terms of providing some business analysis capabilities, Qualys lets you rank assets in terms of how critical they are to your business. A score is then provided in the summary based on your overall exposure level that can be weighted based on how critical the vulnerable asset might be.

One of the best features of QualysGuard is its mapping functionality, which provides a graphical representation of all the devices it discovers on your network. You can drill down on the map to identify the operating systems and services running on these devices, but can't see information on identified vulnerabilities from this vantage point. In addition to the mapping, we'd also like to see some sort of overview console that provides high-level information on the state of vulnerabilities on the network.

NCircle IP360 6.2

NCircle provided a central reporting server, VnE Manager, and scanning point, Device Profiler. With this tiered approach, nCircle runs in a more distributed model than some of the other products tested.

The IP360 provides the best business impact and risk-rating features, offering unparalleled levels of detail. Users can provide asset values for each host and calculate risk scores for each system based on the asset value. This value is a quantitative number, generally dollars, of the value of the asset to the company. As a consequence of this increased functionality, it is not as easy to use as some of the other products tested.

For system discovery, nCircle uses dynamic host discovery, its technique for continuously evaluating environments for new systems on the network. After running on the network for a few minutes, the system had found all the devices in the lab.

For operating system identification, nCircle joined Qualys as the only products to correctly identify the Cisco VPN Concentrator. But it missed a few key systems that most other products identified, including the FreeBSD 5.2 server and the Quantum Snap Server.

For vulnerability identification, nCircle consistently reported the smallest number of vulnerabilities, minimizing false positives, but potentially introducing some false negatives as well.

While nCircle's scan results might appear to include false negatives, following the remediation guidelines for identified vulnerabilities will address the known vulnerabilities in the system.

NCircle accrued the lowest network and system impact, with no identified issues or spikes in network traffic or CPU utilization.

One unique feature of the IP360 is its continuous scanning mode, which provides non-intrusive, back-to-back scans of the whole network or of only select segments.

This is ideal for critical systems or networks that need to be monitored at all times. NCircle provides a classic scanning model of scheduling scans, grouping systems and providing detailed user access.

NCircle takes a different approach in providing vulnerability remediation information. For the sample of vulnerabilities we reviewed, nCircle provided links to patched versions or specific patches for a variety of operating systems. In a few instances, the vulnerability remediation information did not match the specific vulnerability identified, although following the recommended course of action would in most cases have fixed the vulnerability because one patch would fix several issues.

Visionael Enterprise Security Protector

Visionael uses Nessus as its underlying scanning engine and focuses on providing some of the best vulnerability management functionality, such as a customizable portal for viewing security trending information.

Installing Visionael on Red Hat Enterprise worked well, although we'd like to see Visionael better secure the assessment server by default rather than leaving that up to the systems administrator.

Upon initial logon, Visionael provides the best portal functionality, allowing customization for each user and quick views of identified vulnerabilities, current risk level, trending and trouble ticket status.

There were a few issues in terms of system identification for the hosts on the lab network, namely system identification was not happening as we configured it. Working with support, we enabled the detailed operating system checks and reduced the concurrent threads from 200 to 20. With these changes in place, we got operating systems identification results, but they were not as detailed as we would like to see. For example, all Windows systems, regardless of version, reported back as "Windows."

For network and system impact, Visionael is quite loud. The scan locked up the wireless access point, bluescreened a Windows XP system and consumed 30% of the CPU on the monitored target system.

Viewing individual scan results provides an overview of identified vulnerabilities, with a breakout summary of the SANS Top 20, which is unique to this product. We would like to be able to drill down into the report directly from the vulnerability numbers reported in this overview screen.

The reporting module provides a wizard to create custom reports. But the customization options are so abundant that they are almost overwhelming.

The ticketing system is very strong, although tickets only can be auto-assigned for SANS20 or high-level vulnerabilities, which is fine if you prefer to do more detailed analysis on the other levels of vulnerabilities before tasking them out.

Visionael can auto-remediate identified vulnerabilities, but this functionality was not enabled in the license we received for testing.

For business analysis, Visionael provides strong trending information, executive reports and business rank, based on assigning systems one of four levels depending on how critical it is to your business.