LANs /

Review: What's wrong with my network?

WildPackets' EtherPeek is a low-cost protocol analyzer tool that's easy to use.

By Barry Nance
Network World, 07/09/01

Finding the cause of a network problem seems to invariably take either a few minutes, a few days, a few weeks or forever. The problems that take a few days or weeks to solve are the nasty ones. All too often, we've seen groups of people hold several all-day (or all-night) brainstorming sessions, scratch their heads, drink gallons of coffee and generally thrash about looking for the cause of connectivity problems, application errors or sluggish performance situations. People get frustrated, their self-esteem plummets ("Why can't I fix this?") and nervous managers hover about as they anxiously await a breakthrough. Sometimes the problem-solving atmosphere gets so murky even the managers start making suggestions as they try to be helpful. When the troubleshooters actually listen to the manager's suggestions, you know things are serious.

Problems in the "few minutes" category are intuitively easy to solve (for instance, when you notice a server has rolled over and died), and the ones in the "forever" category can sometimes simply disappear (for instance, because someone makes an unrelated change to the network that inadvertently fixes the problem, or users find a way in their applications to work around the problem). However, any error or warning condition that can't be resolved in a few minutes is a candidate for using the best diagnostic tools you can get your hands on.

Pinpointing a network problem's cause ought to be a simple, quick and sure process. If you know exactly what messages traverse the wire when the problem happens, and if you understand the application well enough to know what its message traffic should look like, wouldn't a quick comparison of the two reveal the nature of most problems? That's the protocol analyzer vendors' key premise. They claim their products dramatically speed up the diagnosis of network problems by showing you the contents of selectable network messages, displaying the overall state of the network in summary form and flagging those messages that the software identifies as problem related.

To find the best software-based tool, we invited protocol analyzer vendors to submit their products for review. The five vendors that responded were Agilent Technologies, which dispatched its Agilent Advisor 11.8 software edition; Sniffer Technologies (a Network Associates business unit), which shipped us Sniffer Portable 4.5; Network Instruments, which sent its Observer 7.1; Shomiti Systems, which sent its Surveyor 3.2.; and WildPackets, which supplied its Network Management Suitcase III. Network Management's Suitcase III is primarily the EtherPeek 4.1 product accompanied by a tool kit of six helper utilities, EtherHelp, iNetTools, NetSense, NetDoppler, ProConvert and WebStats.

Blue Ribbon winner

WildPackets' EtherPeek
While all five tools gave us useful displays of network message contents, WildPackets' EtherPeek wins our Blue Ribbon Award for the breadth of protocols it could decode, its low cost, the ease with which we could navigate its screens to determine the cause of a problem, and the clarity of its reports.

The tutorial material in these products' help screens notwithstanding, we found that you still need considerable expertise in network protocols to make productive use of an analyzer's displays of decoded packets. We'd like to see protocol analyzers incorporate artificially intelligent knowledgebases that subsume the experiences and skills of veteran network troubleshooters.

Legal wiretaps

Observer and EtherPeek claim to be able to decode more than 1,000 protocols, but some of the protocols on the vendors' decode lists aren't really separate protocols but rather just subprotocols. EtherPeek's total is more realistically about 450, while Observer decodes more than 400. Sniffer can do more than 450, and Surveyor handles more than 200. Agilent Advisor can decode more than 400 protocols.

All five protocol analyzers were able to decode the major protocol suites, including Ethernet, AppleTalk, DECnet, NetBEUI, IPX/SPX, NetWare's file-sharing NetWare Core Protocol, System Network Architecture, IBM's and Microsoft's Server Message Block, TCP/IP and TCP/IP utilities such as telnet, HTTP and voice over IP. Similarly, all five refused to drop packets in our packet-flooding tests. All five products came with remote capture utilities for collecting and analyzing packets from nonlocal network segments.

EtherPeek helped quickly pinpoint the causes of the deliberate error conditions we set up (see "How we did it" ). However, EtherPeek lacked a real-time expert mode feature, which meant we had to be judicious in how we selected the packets to monitor. By establishing filters and triggers in EtherPeek and EtherHelp (EtherPeek's remote packet-capturing utility), we easily reduced the number of packets to inspect and focus more narrowly on the problems.

The filtering criteria included network address, protocol, port, specified strings of text inside packets, packet length and error codes within packets. We could also instruct EtherPeek, via what WildPackets called plug-ins, to verify packet checksums, detect duplicate IP address assignments, log FTP file transfer operation file names, monitor network addresses for continuous connectivity (useful for knowing whether a server has died), announce Internet attacks (such as denial-of-service packet floods), track telnet sessions and log Web server and news server accesses. Best of all, a customer can create tailor-made EtherPeek plug-ins with only the most rudimentary of programming skills.

EtherPeek's triggers let us use these criteria to start the capturing of packets. NetSense, a postmortem expert mode analyzer of EtherPeek capture files, was very helpful, but we wished WildPackets had designed NetSense to run alongside EtherPeek in real time.

In addition to the remote packet collector EtherHelp, EtherPeek's other tools included iNetTools, which is a utility for performing pings, traceroutes and DNS lookups; NetDoppler, for determining latency and throughput statistics; ProConvert, a packet capture file format conversion utility; and WebStats, for analyzing Web site traffic.

Agilent's software version of its Advisor lacks the connectivity options of the hardware-based Agilent Advisor, but the two otherwise share the same ability to capture, decode and analyze network traffic. The Advisor's decoding of voice-over-IP packets was the most comprehensive of the five analyzers we reviewed, and it paired Oracle and Sybase requests and responses to show database transactions on an event-by-event basis. We were impressed by the Advisor's decoding of Cisco discovery and routing protocols. It even handled Multicast Open Shortest Path First messages correctly. Like EtherPeek, Agilent Advisor allowed the filtering of packets at time of capture or during later analysis. Like EtherPeek, Advisor was an effective and productive tool as we worked to solve our network's deliberately caused problems.

Network Instruments' Observer's filter options included network address ranges, error conditions, specific protocols and up to 20 concurrent user-definable custom offsets and values. Almost as well as EtherPeek and Advisor, Observer helped us solve the problems we created in the lab. Like Advisor and Sniffer, Observer can use RMON and RMON2 to manage and collect SNMP statistics from SNMP-aware devices. Advisor, Sniffer and Observer can also monitor individual switch ports. Observer's expert mode wasn't as helpful as EtherPeek's, Advisor's or Sniffer's, but the vendor says it'll release a new version of Observer later this year that adds more than 100 new expert mode events.

Network Instruments also claims the new version will decode many more types of network messages and include a Router Observer component that can monitor up to eight separate routers and a Web Observer for keeping a watchful eye on up to eight Web servers.

Surveyor's filters, which allowed us to focus on message traffic by protocol, IP address, and several other criteria, are well crafted. We liked being able to change filter criteria on the fly. Moreover, creating our own user-defined filters based on Shomiti's filter templates was child's play. Surveyor offers a sophisticated scripting environment that doesn't require a great deal of programming skill to use, and we liked its expert mode, which proactively reported problems and even suggested corrective actions. Solving our test problems with Surveyor was on par with Observer and almost as quick and productive as with EtherPeek and Advisor. Surveyor additionally can manage distributed Shomiti devices.

Although Sniffer Portable can decode just about any protocol, we had to spend quite a bit more time examining its displays of decoded packets to solve our test problems. Unfortunately, Sniffer's designers have made locating and understanding culprit packets more difficult than they need to be. On the other hand, Sniffer Portable's rather thorough and accurate decodes of IP Security (IPSec) packets were impressive. Sniffer interprets the attributes and proposals offered in an IPSec handshake, which can be invaluable to anyone who needs to troubleshoot IPSec configurations. Sniffer's running display of summary statistics and diagnostic results based on its expert mode's scrutiny of packets is excellent, during a capture session or a monitoring session.

Rush-hour traffic reports

Agilent Advisor's on-screen displays of statistics and network events may be all you'll need to solve your network's connectivity and performance problems. However, if you need to prepare network status and planning reports for consumption within your company, Agilent's $700 Advisor Reporter option is worthwhile. It generates presentation-quality Microsoft Word documents and Microsoft Excel spreadsheets containing graphical representations of network activity. We easily created useful and effective baseline reports, status reports and capacity planning analyses with Advisor Reporter.

Its built-in AutoReport Profiles are wizard-like in their simplicity. The reports we generated automatically included a cover page, table of contents, inline statistics, glossary definitions and embedded charts. Similarly, the spreadsheets we constructed contained data tables and charts for the protocol analysis measurements we selected.

We used Advisor Reporter's Automated Reporting and Custom Reporting modes. Automated Reporting simply asked us for the type of report to produce and, using the Reporter template, instantly built the document or spreadsheet we wanted. Custom Reporting prompted for the entire table, chart and report options on a step-by-step basis, which gave us complete control over the appearance of our documents and spreadsheets. The Reporter also comes with the AutoReport Editor for assembling custom profiles for use with the Automated Reporting mode. Each profile specified document headers, footers, cover page text, charts to include and formatting to use for numbers and dates in the report.

Even without Advisor Reporter, Agilent Advisor presents valuable displays of nodes, network utilization, error situations and message contents. These displays showed us network bandwidth users, network utilization by protocol, summarizations of connection, protocol and network events, detailed lists of connection events, protocol statistics (including errors and average packet size), connection statistics, lists of discovered nodes, network, transport and data- link layer vital statistics, and physical layer vital statistics.

EtherPeek displays node, protocol, conversation, network, error, size, summary and history information through its Global Statistics and Traffic Statistics windows.

The Global Statistics window shows all activity, while the Traffic Statistics window shows filtered activity. Node statistics, which are useful for tracking bandwidth usage by node, include real-time packet counts and traffic volumes as well as the total number of network nodes. Protocol statistics show network traffic volume, in packets and bytes, by protocol and by subprotocol. This data is useful for determining which protocols or subprotocols are using high amounts of bandwidth. Between pairs of network nodes, conversation statistics show traffic data, in bytes and packets, for each protocol or subprotocol the pair has used. The packet size distribution statistics revealed the number of packets by size that the network has carried.

The summary and history statistics showed network performance over time, graphed according to selectable intervals. For a user-specified interval, EtherPeek's graphing and trending feature can collect, analyze and display (via several different graph options) any of EtherPeek's node, protocol, network or summary statistics. EtherPeek can optionally render the data as Web pages.

Observer's reports showed top talkers, protocol statistics, conversation pair statistics, Internet usage, physical layer errors, transport layer errors, router statistics, switch statistics, network utilization and historical trends. The top-talkers report contains a list of nodes, by bandwidth usage, and it included bandwidth percentages, total packets, broadcasts and multicasts. The protocol statistics report categorized network traffic by protocol, in either tabular of graphical format. The conversation pair statistics report tracked nodes exchanging network messages and graphically illustrated the nodes' conversations by drawing lines between the nodes.

The Internet usage report identified the nodes connected to the Internet, by node, service (HTTP, Network News Transfer Protocol or FTP) and Internet destination. The physical layer report indicated (for Ethernet) the number of wrong-sized packets, cyclic redundancy check (CRC) errors, collisions and alignment errors.

One of Observer's most useful reports, the server analysis report graphically contrasted server response times vs. the number of concurrent requests. The current Router Observer module can monitor a single router device, displaying total packets, total bytes, packets per second, bytes per second and device utilization. A switch monitor can continuously examine the ports on a switch to show utilization and connectivity. The Observer's Web Extension dynamically rendered reports in Web page format.

Observer can also show Web server traffic data, including number of Internet connections and percentage of local network traffic. A vital signs report divulges average and maximum bandwidth utilization, total packets, CRC errors, alignments errors, wrong-sized packets and collisions. An Ethernet collision analysis identifies the top 10 network colliders.

Surveyor's reports include Media Access Control-layer statistics, network utilization, physical layer errors, packet size distribution, protocol distribution, top talkers (MAC layer, network layer and application layer), address mapping, protocol analysis by Open Systems Interconnection layer and a real-time packet decode summary.

Sniffer Portable, like Agilent Advisor, displays useful statistics and the vendor offers an optional, separate reporting tool.

The Sniffer Reporter's tabular and graphical reports are top hosts by traffic; top hosts by protocol; matrix reports identifying top conversation pairs; and protocol distribution reports. A global statistics report disclosed traffic by segment, errors by segment, segment size distribution and segment utilization. An alarm report revealed alarm details the Sniffer captured.

Playing in traffic

The only one of these five products to run on Macintosh computers as well as Windows computers was EtherPeek, which has an intuitive user interface and well-designed display of network activity. Although the default columns available in its packet capture windows were appropriate right out of the box, it was also possible to choose which data elements EtherPeek should display for captured packets.

The selectable columns included source and destination logical addresses, protocol types, packet sizes and time stamps.

Each packet capture window had its own dedicated capture buffer and associated filter criteria. On a computer with 64M bytes of RAM, we could create as many new packet capture windows as we liked. In one test, we had two dozen such windows (and capture buffers) operating concurrently.

EtherPeek's name table holds device and protocol name-address equivalencies for your network. On a network that uses DNS, EtherPeek can automatically discover names for the devices at each IP address, and the vendor says the Macintosh version of EtherPeek similarly resolves AppleTalk names. Editing the name table by hand is a tedious chore.

Agilent Advisor's user interface is also easy to operate. Furthermore, the friendly Windows-based user interface let us run several protocol analysis tests (Agilent calls them measurements) concurrently. Windows 98, our test bed, isn't known for its multitasking capabilities, but Advisor's excellent design showed that good programmers can work around even the worst of operating system limitations.

Advisor's measurements included an expert mode test for troubleshooting general network problems (such as those needing point-to-point filtering of specific traffic); an IP device discovery test; NetWare-specific tests for such problems as a client that's not able to find the nearest server; address resolution tests; MAC-layer node tests; and various tests for gathering and displaying statistics. All of Advisor's measurements were "children" of the "parent" expert mode.

Observer's user interface offered a tree view and toolbar-oriented view of network elements. Selecting the elements to monitor and the statistics to collect was simple. In the expert summary problem analysis, which shows a list of error events, a double click drilled down into the capture-buffer detail for further analysis.

Double clicking on any of the protocol-based or application-based problems shown in the TCP/User Datagram Protocol/Internet Control Messaging Protocol experts window drills down to the conversation level to show which pairs of nodes are involved in the problem.

The window also shows network errors organized by time of day to help you judge whether a problem is intermittent or consistent.Observer also displays a window containing a graphical view of network conversations. Alongside each conversation pair were statistics showing packet-to-packet delay times, retransmissions and lost packets. Clicking on a conversation pair inspects a list of packets exchanged by the nodes, with the contents of each packet displayed in a separate window.

Surveyor's parent window is an overview of the network segments chosen for monitoring. Typically, you'd open a Resource Browser window to list all of the segments you're monitoring - local and remote - and an Alarm Browser to show alarm conditions you've selected. In the Monitor View, you can define the view you want to use for each segment. This view might, for example, show a network-utilization strip chart.

Sniffer's user interface provides a dashboard to show network utilization, packets per second and error counts. One Sniffer window displays a scrolling list of captured packets, while another contains an expandable tree view of protocol event alarms you can set. For a particular packet selected in the scrolling packet list window, another window displays decoded detail.

All five products are easy to install and come with easily understood documentation. The Agilent Advisor documentation is especially comprehensive, and its guided troubleshooting help screens clearly explain the steps to take and the meaning of the results obtained for a range of network diagnostic situations.

Conclusion

Agilent Advisor and Sniffer Portable are excellent but somewhat pricey protocol analyzers.

Large organizations can best take advantage of (and pay for) their many features. Network Instruments' Observer merely needs to decode more packet types to be a world-class protocol analyzer.

Surveyor has an excellent packet-filtering mechanism but also needs to decode more packet types. For day-to-day use by a network administrator who's in charge of from a hundred to a few thousand networked computers, we recommend the capable and relatively inexpensive WildPackets' EtherPeek analyzer.