PRINCETON, N.J. - You won't find much good news in the latest research about Internet search engines, which may explain why companies that rely on the 'Net to do business are so anxious to find better alternatives.
The examination of 11 search engines by NEC Research Institute indicates that the expansion of the Web is outdistancing the ability of those search engines to track it. The two researchers, Steve Lawrence and C. Lee Giles, found only one search engine, Northern Light, indexed more than 16% of Web pages. Euroseek brought up the rear of this recent study at 2.2%. Just over a year ago, the best search engine could handle one-third of the Web's then 320 million pages.
The researchers also note the number of Web pages more than doubled between December 1997 and February 1999. They estimate there are now 2.8 million sites and 800 million Web pages. Of those sites, 83% contain commercial content. Incidentally, despite perceptions to the contrary, only 1.5% of the sites feature adult material.
For enterprise network managers, the inadequacies of search engines mean their users are "getting a small subset of what's available to them," Lawrence says. Also, it is usually the more popular sites that get indexed and not necessarily those most relevant or up to date.
This is not exactly shocking to those who rely on the Web to retrieve business data. Search engines often fail to come up with the desired results, and that isn't likely to change until the arrival of a more efficient set of tools, says Timothy Sloane, analyst at the Aberdeen Group, a Boston consultancy. For a company to get some value out of its search engine, there must be some type of indexing of relevant material in a specific category, says Sloane. That can be attained through the use of XML, a tagging language that lets users categorize data and reuse it from application to application without reformatting.
Search engines do seem to work better within internal networks, where everyone can agree on one standard of classification, experts say. One enterprise software company, J.D. Edwards, has found that by coupling a search engine with a trained knowledge management staff, it can save about $10 million a year in research costs. The company has come up with standard ways of classifying internal data by using terms and labels, says manager Wayne Applehans.
Using a single "corporate vocabulary," the 7,000 employees and partners who use the company's Web site are able to access schedules, company policies, technical data and more, he says. Knowledge managers are responsible for pruning and organizing the data in the Web site for easy, up to date access. The system saves employees up to 10 hours a month. The search engine alone "gets you half the way there," Applehans says.
RELATED LINKS
Other recent articles by Songini
Searching for XML
Network World, 03/15/99
