A security vulnerability in one of the most commonly used e-mail server software packages could have a wide ranging impact, akin to the Microsoft SQL Server vulnerability that spawned the recent Slammer worm, according to an advisory published Monday by Internet Security Systems (ISS).

The buffer overflow vulnerability was found in a number of versions of the open source Sendmail Mail Transfer Agent (MTA), ranging from the most recent release of that software to versions that first appeared in the late 1980s. The vulnerability could allow a remote attacker to gain "root" (superuser) access to a Sendmail server, according to ISS.

Sendmail is the most popular Unix-based implementation of the Simple Mail Transfer Protocol (SMTP), which is used to transmit e-mail messages. Predating the modern Internet itself, Sendmail is used to process incoming e-mail messages.

A vulnerability exists in the software code that is used to evaluate whether addresses in the header field of an e-mail message are valid.

Attackers who understand the vulnerability could compromise a server by sending an e-mail message with an improperly formatted message header, causing a buffer overflow that would enable the attacker to place and execute their own malicious code on the server.

What makes the new vulnerability particularly pernicious is that attackers would need to know little about the server they were attacking other than its Internet address, according to Dan Ingevaldson, team leader of X-Force research and development at ISS.

"It's quite a dangerous vulnerability because an exploit could be contained in the e-mail message itself. The attacker doesn't need to set up an elaborate system to launch the attack. They could just send an e-mail message to a server, and if the server is vulnerable the attack would be launched," Ingevaldson said.

While the vulnerability requires sophisticated knowledge of the Sendmail program to understand and exploit, it could still be quickly leveraged by hackers in the form of a Slammer-like worm, according to Ingevaldson.

Part of the reason for that is that, as an open source product, the Sendmail source code and the new patch code are visible to hackers as well as e-mail server administrators. The recently released patch will immediately flag vulnerable areas of the Sendmail code.

The IDG News Service is a Network World affiliate.