SAN DIEGO - InfoLibria was the winner in the first vendor-neutral caching benchmark test held at the University of California San Diego, although Novell made a strong showing with its beta Internet Caching System software running on a Dell server. Both companies were guaranteed a decent ranking after five participants dropped out.
The test was held in mid-March, and the results were announced last week. It was sponsored by the IRCACHE caching research project, which is run by the National Laboratory for Applied Network Research.
All the major caching vendors - Inktomi, InfoLibria, CacheFlow, Network Appliance, IBM, Cisco, Novell and start-up Entera - said they would participate. But only offerings from InfoLibria, Novell, the University of Wisconsin and the freeware Squid wound up with test scores. CacheFlow, Inktomi and Cisco dropped out before the testing started, and IBM and Network Appliance took a "bail-out" option and declined to have their test scores reported.
Both InfoLibria and Novell/Dell entered two caching products. InfoLibria's numbers were highest: a maximum sustained throughput of 1,680 requests per second, with a mean response time of 1.4 to 4.2 seconds. Novell/Dell was a close second: 1,500 requests per second, with a mean response time of 1.4 to 1.7 seconds. Novell is claiming victory on a price/ performance ratio because its entry costs about $50,000 vs. InfoLibria's $200,000.
Meanwhile, the vendors that dropped out were busy explaining their decisions.
"We didn't want to disclose our detailed performance numbers to our major competitors," says Edward Sharp of Network Appliance.
"We don't think bake-off processes where all the vendors are thrown into a room . . . with a tool that isn't completely verified and matured is an appropriate way to start a benchmarking effort," says Ed Haslam of Inktomi.
"We felt that it was premature for us . . . to participate in that simply because it had not matured enough to represent what we think our customers will need," says R. Vasu Devan of CacheFlow.
