Top spam fighters offer feature diversity

For the top of the heap of spam products, it's not what is good or bad that sets them apart. It's more a matter of what's different.

For example, if an anti-spam product doesn't allow for SNMP-based monitoring, you will only care if you're already using SNMP.

Our short list, based on the spam catch tests, included three services (Postini, Advascan and Mycom), four appliances (BorderWare, CipherTrust, Barracuda and Messaging Architects), three software packages tested on Unix (from Sophos, Proofpoint and Cloudmark) and two tested on Windows (Symantec and MailFrontier). We let the vendors choose the platform where more than one was supported.

To distinguish between the products, we looked at four key areas: spam-oriented features, per-user features, anti-virus and policy-based filtering, and logging and management.

Let's start with spam

The most important feature in an anti-spam system is how well it catches spam. All of our finalists turned in outstanding false-positive and false-negative scores, but there is considerable variation in how each product lets IT control the spam catch process. Most products offer a cocktail of techniques to catch spam.

The term "cocktail" is used by anti-spam vendors to explain how they make the go/no-go decision on spam. Early spam products had only one technique, such as searching for words in headers of message bodies, or a set of techniques that each could torpedo a message as spam. Modern products mix the results from multiple tests and analyses, combining and weighting them to come up with a final answer for each message. As the SpamAssassin team puts it when describing their anti-spam cocktail, "While any of these tests might by themselves mis-identify a message, their combined score is terribly difficult to fool." To implement the cocktail, each message runs through multiple filters or tests, and receives a set of scores. When enough tests agree (or when a single test gets a high enough score), the message gets its verdict: spam or not spam.

Many vendors sent elaborate white papers explaining how their spam cocktail was mixed to be superior to the competition. In our evaluation, we decided to not go down the path of evaluating the components of the cocktail. The proof of what works well (and what doesn't) comes out of the statistics on false positives and false negatives. In this market, the strategies each vendor uses to classify spam are in rapid flux as they search for better ways to outfox the spammers.

In our tests, products that let one test dominate the score - have only one test - tend to have a high false-positive rate. For example, just having the word Viagra in the subject line of a message does not make it spam. But having Viagra in the subject, in the body two or three times, a Web site URL of an online pharmacy and having the message come from the IP address of a suspected spammer all add up to the message being spam.

If you want to see the rules used to match spam and edit them, then Sophos' PureMessage and Messaging Architects' GWGuardian are your best choices. Both let you dive in and touch every aspect of the spam matching. This is a mixed blessing.

Corporate managers are moving away from tuning systems at this level because it's really not important. If the spam engine is doing its job properly, you don't have to look deep into the innards. However, there will always be exceptions. Sometimes the mail flow at a company can confound the spam engine, and this level of detail will be required.

A more likely requirement will be for coarse control over the factors that go into the spam scoring. Products we looked at range from virtually untouchable (Advascan and Symantec's Brightmail) to the relative openness of CipherTrust's IronMail and Proofpoint's Protection Server.

One critical factor is the ability to balance how well DNS features are incorporated into the spam score. With a notoriously high rate of false positives, DNS blacklists and DNS reverse lookups are dangerous to use in a go/no-go system. However, using DNS features as a component of the larger picture is a great way to filter out spam before it hits the device. BorderWare, Sophos, Proofpoint, CipherTrust, and even service Mycom let you pick which lists to look at, and what weight to give them. Other vendors, such as Postini and Symantec, maintain their own weighted DNS blacklists and whitelists to eliminate false positives that looking at any one list will cause. The ability to adjust these features is critical. For example, service vendor Sublimemail could not turn off DNS features built into its service, which increased its false-positive rate by a factor of 20.

Power to the users

No anti-spam product will have zero false positives. As we discovered in our tests, the better you are at catching spam, the worse your false-positive rate (and vice versa). The problem becomes how to deal with false positives that inevitably happen.

Vendors have taken three approaches. A popular one is to assume that false positives don't exist and to make a few pieces of mail vanish every now and then. These vendors didn't make our final cut.

Another strategy is to tag-and-deliver mail rather than delete it. With tag-and-deliver, some or all of the spam is actually passed onto the corporate mail server, but tagged in such a way that users don't see it unless they specifically look for it.

Tag-and-deliver has a huge problem, though: the volume of spam is so high it dominates Internet message flow. In our test, about 75% of the mail we received was spam. With tag-and-deliver, you would be storing, backing up, indexing and archiving four times the number of messages you really want.

Most products can distinguish between certain spam and mail they think is probably spam. Certain spam can be discarded, or even rejected before it is accepted, while mail with a more uncertain score can be sent to the quarantined, or tagged and sent for a "just in case" review by the user. The only product that doesn't separate spam and maybe-spam is GWGuardian. All other vendors offer the opportunity to separate at least two levels of spam with different actions (Postini doesn't let you tune the thresholds, but every other company does).

The third alternative is per-user quarantines. When a message is identified as spam, it is quarantined instead of delivered. Unlike a normal mailbox, quarantines clean themselves out regularly, and usually don't have to be built on the same kind of highly reliable infrastructure and high-performance servers that corporate mail servers require.

All the products in the top 12 have a quarantine, although it's less common when you consider the entire anti-spam market. By giving each user power over his own questionable spam, and by giving network managers the option to delete the most egregious and obvious unwanted mail, anti-spam products strike a balance between performance, user frustration and wasted effort, and the inevitable false positives.

Not all quarantines are created equal. There are some dark corners, especially with authentication. For example, the Advascan and Mycom services can't use your corporate Lightweight Directory Access Protocol (LDAP) or RADIUS authentication database, which means every user will have to maintain a separate password for his spam quarantine. CipherTrust's quarantine doesn't have any authentication at all - a user clicks on a URL via e-mail, and this acts as his authentication. We also ran into severe design limitations with Barracuda's LDAP authentication and Messaging Architect's SMTP-based authentication. The lesson learned was to dive into the details if you want to use a quarantine, because there are many deal-breakers out there.

We also considered per-user and per-group settings and user control over these settings. While many network managers might not want to let end users play with their spam settings, the argument in favor of empowering them is strong. When users are in control, they are happier, and having some black box filter their e-mail without a way for them to control it doesn't go over well. Several products put an enormous amount of control (perhaps too much) in the hands of the users.

In the top 12 products, we found 12 different group, user and customization strategies. The most flexible were from MailFrontier, Messaging Architects, Mycom and Postini. Each of these has group-level and user-level settings, and gives the network manager the opportunity to expose those settings to users (if desired). If you want to give users control over their own settings, BorderWare, Sophos and Barracuda offer partial or full control. Symantec, CipherTrust and Cloudmark don't really believe in defining per-user settings, while Sophos, Proofpoint and Cloudmark don't believe in per-group or per-domain settings. Cloudmark doesn't believe in any distinction between users - the Zen-like simplicity of its interface allows for only one set of spam settings for the entire server.