Network World maintains an online Buyer's Guide, which allows any anti-spam vendor to submit its product information. Based on our award-winning test last year and the 12 months that the Buyer's Guide has been in operation, we decided that any vendor who wasn't in the Buyer's Guide wasn't very serious about participating in a product test, so we used that as the basis for our initial e-mail invitation list.

Next, we set criteria for the test to include products that meet the needs of companies looking for large-scale, high-performance anti-spam systems. Specifically, we were looking for products that would:

1: Accept mail using SMTP, either as an application running on an existing server, as a dedicated appliance server or as a service.

2: Filter and/or tag mail according to the likelihood that it is undesirable (aka spam)

3: Forward the mail onto the corporate mail system using SMTP.

Because we were testing the products in parallel, we had to specifically exclude anti-spam products that depend exclusively on alternative technologies, such as challenge/response, or graylisting. The challenge/response system that is 'famous' is TMDA (tagged message delivery agent) from TMDA.NET, and the graylisting that is most famous is M-Switch from Isode.

After reviewing the 43 vendors that submitted products for the test, we asked four to drop out. Ipswitch, Kerio, Stalker Softare and Astaro all submitted products that had some SMTP in/out capabilities, but were designed and sold as mail servers with anti-spam capabilities. We also dropped Lightspeed Systems, because its anti-spam product, a transparent Layer 2 firewall, was not compatible with our test bed.

IronPort Systems, a messaging appliance vendor, was asked not to participate in the test because Opus One has an existing consulting contract with this company - including them in the test would have created a conflict of interest. If you are interested in IronPort's spam catch rates, you can infer them from Borderware or Symantec's numbers because all three are based on the same anti-spam engine.

We also reached out to the SpamAssassin community (see "What about SpamAssassin?"), but couldn't find someone who could act as a representative for support and configuration assistance. However, two commercial vendors, Roaring Penguin (on Unix) and NoSpamToday! (on Windows) sent products that exposed their SpamAssassin cores. Although neither met our false-positive threshold for inclusion in the top 12 finalists (probably because of difficulty of tuning Bayesian engines and neural networks in a test lab setting), we were very pleased to have them participate in the project.