Internet Security Systems is warning its customers about a critical security hole in a commonly used technology from the Mozilla Foundation called the Netscape Network Security Services library that could make Web servers vulnerable to remote attack.

ISS Tuesday issued a security bulletin about a flaw in the NSS library's implementation of the Secure Sockets Layer Version 2 protocol that could allow remote attackers to use an SSLv2 connection to take control of Web servers using the NSS library. The flaw in the NSS library affects the Netscape Enterprise Server and Sun's Sun Java System Web Server, but may also affect countless other products that use the open source NSS library, ISS said.

The problem stems from a flaw in the way the NSS library handles requests for new SSLv2 sessions. Servers using the NSS library do not check the length of a record field in the first part of the negotiation between two systems attempting to establish an SSLv2 session. Malicious hackers could use the absence of that length check in the first record sent in the negotiation, known as the "hello message," to cause a heap overflow, allowing them to place and run malicious code on a vulnerable server, ISS said.

In heap overflows, an area of a vulnerable computer's memory that is allocated for use by a software program is exceeded by a piece of data that is larger than the allocated space, causing adjacent areas of memory on the system to be overwritten with arbitrary data or malicious code sent by the attacker.

If successfully exploited, the NSS library vulnerability gives a remote attacker access to the vulnerable system with the same level of privileges as those given to the Web server. On Microsoft Windows systems, Web servers typically have full system privileges, ISS said.

While SSLv2 protocol support is disabled on the Netscape Enterprise Server and Java System Web Server, SSLv2 is a commonly used protocol for sending sensitive information over the Internet, and many installations may have the support for SSLv2 enabled, ISS said.

In addition to the Sun Java System Web Server and Netscape Enterprise Server, the flaw affects the Netscape Personalization Engine, Netscape Director Server and Netscape Certificate Management Server, the company said.

The Mozilla Foundation issued a patch for the NSS library that fixes the SSLv2 hole. Alternatively, NetScape Enterprise users can disable the SSLv2 protocol, ISS said.

The IDG News Service is a Network World affiliate.