IPSec's double-edged security

Many network technologies have held the promise of revolutionizing and replacing existing wares. So it was with IP Security (IPSec), a virtual private network (VPN) security technology with integrated support for shared secret key and digital certificate authentication. IPSec also supports encryption with data encryption standard and Triple-DES. IPSec held the promise of replacing less sophisticated security technologies while still guaranteeing a level of interoperability among different vendors of IPSec products.

It's certainly no secret that IPSec - indeed any network encryption technology - is inherently incompatible with the network features and services that require the correct identification of traffic content. For instance, because IPSec hides source and destination IP addresses and port numbers of the real end stations, it is impossible for Layer 4 switches to forward IPSec traffic to appropriate servers or applications.

A similar problem arises in running IPSec connections across the current generation of carrier-class ATM-based VPNs. Unlike IPSec-based VPNs, today's ATM VPNs offer no encryption or authentication between ATM edge devices, but rather rely upon dedicated circuits across the ATM cloud with carrier- controlled access and authentication. However, assigning appropriate circuits to each traffic stream means identifying the traffic content, a task made virtually impossible by the encryption of the data content within IPSec streams.

Many customers could accept IPSec's incompatibility with Layer 4 switches, and even with carrier VPN services. But few were prepared for the incompatibility of IPSec with some of the leading firewall technologies. More specifically, the best firewall securities - those that rely upon application proxies - require that the firewall interact directly with applications passing through it. Unfortunately, the firewall cannot determine the application content of IPSec traffic, let alone attempt to intercept application commands and data because all IPSec content is encrypted.

Allowing IPSec traffic through a firewall would mean punching a gaping hole in the firewall to allow passage of any traffic that matched only rudimentary frame header information that merely suggested that it was legitimate IPSec traffic. This might weaken overall network security rather than strengthen it.

Instead, the strategy many customers have been forced to implement involves dual parallel security. This plan utilizes a firewall and an IPSec gateway in parallel. Incoming IPSec connections target the gateway, whereas non-IPSec traffic targets the firewall.

There is no question that IPSec exceeds the simple authentication and verification of a firewall, providing vendor- independent encryption. The question customers should ask is, "Should we deploy IPSec with its sophisticated authentication and encryption, or rely upon more straightforward security systems such as firewalls and carrier-based circuit VPNs that are more universally available?" The answer, quite simply, is "Yes." Neither is perfect and complete. Neither will replace the other.

Curtis is director of engineering with The Tolly Group, a strategic consulting and independent testing firm in Manasquan, N.J. Curtis is sitting in for regular columnist Kevin Tolly this week. He may be reached at (732) 528-3300 or www.tolly.com.

More Tolly on Technology columns