Dear (vendor),
Network World is teaming up with The Tolly Group to launch a new switch performance study that will serve as the definitive industry benchmark for equipment buyers.
We'd like your support on this project. Before we go "live" and announce this endeavor to the industry at large, we wanted to give a select group of industry leaders an opportunity to review our plan and pledge their support.
The Tolly Group's Switch Metric is a ground-breaking performance measurement that combines the power of raw throughput along with a cost-per-gigabit/throughput metric to give buyers a more complete analysis of the layer 2 and layer 3 switch market, and beyond. The findings of these Switch Metric tests will be presented twice a year to Network World's readership of 164,000 enterprise network executives in feature articles that not only convey the results but, more important, explain their meaning to prospective switch buyers. Moreover, The Tolly Group and Network World are lobbying to create an event at a major trade show to showcase the Switch Metric findings.
The Tolly Group's cost-per-gigabit/throughput switch metric identifies a switch's best throughput result, and divides it by the retail cost of a system to provide a relative cost for each gigabit of throughput, analogous to the cost-per-MIPS metric used in benchmarking computer systems. (For details, see the Test Plan Overview that accompanies this letter.)
Unlike tests conducted elsewhere that have focused on single-topology Ethernet-based switches only, The Tolly Group's Switch Metric will be used to evaluate Fast Ethernet and Gigabit Ethernet switches, as well as ATM, token ring and FDDI switches. And, The Tolly Group will also perform tests on Fast Ethernet-to-Gigabit Ethernet, FDDI-to-Gigabit Ethernet and other cross-topology configurations. While vendors will be offered the opportunity to test a range of switch parameters, all participating products must undergo a core set of performance benchmarks, as described in the accompanying Test Plan Overview.
Tentatively, we have identified publication dates in May and September, which means product testing would have to be completed 45 days prior to publication date. Test results will appear in print in Network World, and all test results also will be published on The Tolly Group's Web site.
Network World and The Tolly Group encourage all switching vendors to participate in this benchmarking endeavor. Costs are nominal; participating organizations simply pay $1,500 per day for testing. Although paid testing is a departure for participation in Network World tests, the nominal testing fee is necessary to defray the cost of what is intended to be an ongoing, industrywide benchmarking effort.
For each day of actual product testing, The Tolly Group will allot a quarter day's time for documentation/analysis/posting. Our definition of testing includes equipment setup, so The Tolly Group and Network World encourage participants to send support staff to ensure products can parachute into the testbed with relative ease.
Results of the switch testing would be communicated to your organization via a brief test findings document that maps out your product's benchmark results under the banner of the Tolly Tested logo. Results will also be posted, as they are approved, on The Tolly Group's Web site.
The aim of the The Tolly Group's Switch Metric is to provide network managers with a substantial base of authoritative, unbiased information about the cost/performance of key switch products, which buyers can use to make purchase decisions. With the guidance of Network World and The Tolly Group, the 1999 Switch Metric will emerge as the voice of performance for the switching industry. Make sure your voice is heard. Contact NW Editorial Operations Manager Cheryl Crivello by January 17th to indicate your company is ready to support this important endeavor. (She may be reached at ccrivell@nww.com or 508 820-7403.)
Sincerely,
John Gallant
Editor-in-Chief
Network World
Kevin Tolly
President/CEO
The Tolly Group
The Tolly Group's 1999 Switch Metric
A Collaboration Between The Tolly Group and Network World, Inc.
Switching Metric Test Plan Overview
The Tolly Group will subject all participating products to a core set of no-loss throughput tests. Switches will undergo tests for minimum and maximum frame sizes, standard topology mixes and a variety of protocol layers.In addition to the standard benchmark tests, vendors may choose to run any combination of supplemental test scenarios. Among the options available:
Protocol Layer: Layer 2 switching, Layer 3 IPX routing, Layer 3 IP routing
Single Topology: Fast Ethernet, Gigabit Ethernet, Token Ring
Mixed Topology: Fast Ethernet to Gigabit Ethernet; and FDDI to Gigabit Ethernet
Number of Ports: Flexible
Using industry standard test gear from Netcom Systems, Inc. and/or Ixia Systems, The Tolly Group will determine the maximum no-loss packet throughput for the configuration under test. Multiple frame sizes will be tested within the allowable range for the topology. (Example: Ethernet-based tests will be run at 64, 256, 512, 1024 and 1518 bytes.)
The throughput numbers resulting from this test will be published and a subset of the data will be used to calculate the cost-per-gigabit/throughput for best case (largest packet size) and worst case (smallest packet size) for the configuration as tested. (These results might be identical for some products.)
Each vendor's Manufacturer's Suggested Retail Price (MSRP) will be used as the basis for such calculations. The Tolly Group will obtain the MSRP either from data on company Web sites, or by working directly with a vendor's project liaison.
RELATED LINKS
