How ready are the nation's networks?

By Denise Pappalardo and Carolyn Duffy Marsan
Network World, 11/26/01

Disaster preparedness isn't new for carriers and ISPs, which have had to guard against earthquakes, floods and hurricanes since networks began crisscrossing the country.

But no one had planned - at least not fully - for the disaster that hit the U.S. on Sept. 11.

Most national network service providers were affected when terrorists attacked the U.S. Most claim their network readiness prevented a national tragedy from also becoming a technical catastrophe. However, a heightened vigilance has emerged in the two months since.

Experts agree that the Internet performed better than the circuit-switched telephone network during the attacks, but that's because the two networks are designed differently.

MAEs and NAPs (Flash Required)
Disaster recovery: then and now
Security takes center stage
Lehman Brothers' network survives
The terrorist network
Planning for the worst: Bring in the best
Spending shifts

The U.S. military designed the Internet to route around structural problems like those occurring in Verizon's main switching facility in New York.

But when an entire central office switch site that serves thousands of customers is annihilated, it takes time to reroute calls to a different switching facility.

Verizon's West Street central office switch facility, which sits directly across from where World Trade Center Building 7 stood, was severely damaged. This site, which connected 200,000 voice lines and 3 million specialized private data lines, was pretty much out of commission. But with engineers at Ground Zero, Verizon had back-up switches and power supplies on the sidewalks in front of its West Street facility to get customers back online.

An unknown number of users were knocked off the Internet because several ISPs had points of presence in the World Trade Center. However, the Internet's distributed nature let most users maintain access.

Genuity lost a POP in the World Trade Center. While it took time for the ISP to get customers back online, Genuity says it couldn't have done it without the help of Verizon.

ISP dependency on local networks is considered a vulnerability of the Internet, but it's a necessary evil.

Users who accessed Genuity's POP in the World Trade Center were looking to reconnect to the Internet from different sites around New York City and elsewhere. Genuity brought all of its customers back online in a matter of days, says Chris Yetman, vice president of technical support at the ISP.

While all of the service providers interviewed for this story had disaster-recovery and redundancy plans built into their operations before Sept. 11, all agree there is room for improvement.

This event triggered Genuity to examine its data centers and POP locations to ensure these facilities have broadband connectivity that guarantees redundancy.

"In the past we would allow a data center to be homed to the same POP on our backbone if it made the most sense from a network efficiency perspective," Yetman says. "We didn't think it affected redundancy because we weren't thinking about losing a whole data center. We thought about routers going offline or a backhoe chewing up a fiber line."

"It didn't take much to map things differently," he says.

Genuity redirected OC-48 connections from its data centers and POPs to different routers on its network for redundancy's sake. If Genuity lost a facility to a bombing today, traffic would be automatically rerouted.

Genuity has also taken steps to ensure that "certain high-profile customers such as government agencies" are easily and quickly identified and serviced in the case of an emergency. These government customers typically need additional circuits when a disaster strikes, but Genuity also has to ensure circuits aren't taken away from emergency relief efforts at hospitals or Red Cross facilities. Genuity has developed a plan to better locate and contact these customers when disaster strikes, a plan it did not have previously in place.

And while the ISP's network operations center (NOC) - essentially the heart of Genuity's backbone - has a back-up facility that could replicate all monitoring, this site is only 12 miles away. The main NOC is in Woburn, Mass., and the backup is in nearby Burlington, Mass. The ISP also has a back-up facility in Columbia, Md., but this site is not as robust as the one in Burlington.

"We're aware of how close they are, and we're examining ways to lessen that vulnerability," Yetman says.

While redundant ISP networks play a key role in the reliability of the Internet, the fact that this is a network based on hundreds of private and public traffic exchange points is also a key element of its resiliency. The 'Net was designed to withstand outages.

In the late 1990s there were stories written about how vulnerable WorldCom's Metropolitan Area Exchange (MAE) East was because it was housed in the basement of a parking garage. However, even if MAE East were knocked out, the Internet would survive, although users would likely notice a slowdown.

Neither WorldCom nor Sprint, which also operates an Internet network access point in New Jersey, would grant interviews for this story, citing security concerns.

But other providers that operate exchange points on the Internet talked about how operations were affected by the events of Sept. 11.

"We have facilities in New York. The one that was close to Ground Zero suffered some struggles such as power failures as well as carrier failures," says Ali Marashi, vice president of technical services at InterNap, which operates 37 "global service point" exchanges. "But we had another facility, which kept kicking right along.

"The recent tragedy shows that even the best systems can have some weaknesses," he says. "It's a wake-up call to operations centers to better ensure that their back-up plans work as advertised."

Equinix, which operates six exchanges on the Internet, says it hasn't had to make changes to its physical environment or operational procedures since Sept. 11 because security was already high.

"What has changed since Sept. 11 is that customers are focusing on three areas: geographic diversity, network diversity and contractual diversity," says Jay Adel-son, CTO and founder at Equinix. "Our customers are really getting deep into their understanding of how networks are actually built so they can find out if Network A and Network B are actually riding on the same glass."

International service provider Equant understands the need for geographic diversity. Equant, which offers voice, data and Internet services around the world, has five NOCs, but it's in the process of consolidating those locations into three since its merger with Global One. Equant will maintain three NOCs, in Reston, Va., Paris and Singapore. While these NOCs run regional network monitoring, they each have the capability of running full backup for the other.

"Any other NOC can watch the other, which is part of the reason the locations follow the sun," says Jack Norris, director of customer service at Equant.

Equant has learned new lessons about network reliability since Sept. 11. The service provider lost connectivity to Canada after the attacks.

"One of the circuits we purchased to be diverse was routed through Wall Street," Norris says. That was a surprise because the circuit was to connect New York's Long Island to Toronto and Montreal.

"We didn't expect that particular facility would be in a different borough, and we didn't expect it to go through lower Manhattan. If we knew, we would have asked for a change," Norris says.

Equant has reexamined local connectivity for its networks. The service provider is committed to running more frequent audits to be sure local connectivity is in the best location.

Equant has also stepped up physical security at its NOCs and offices.

"There have been more advisories about travel to different countries, information about suspicious packages and building security," Norris says. Instead of leaving the front doors open with a security guard, Equant now has employees "badge in" at security boxes outside all doors.

AT&T also has beefed up security at its NOC in Bedminster, N.J.

"I was able to drive up to the gate and put my ID on a sensor and gain access to the grounds. Now I have to physically be let into the building," says Art Deacon, vice president of network operations.

AT&T has one back-up NOC, but says it has never publicly revealed this location. In addition to a fully redundant NOC, each network has its own operations center, Deacon says. Frame relay, ATM, private line, IP and voice each has its own mini-NOC.

Deacon says that on Sept. 11, AT&T's voice network performed exactly as it should by handling 431 million calls, considerably more than the typical volume of 330 million calls.

But because AT&T shared space at Verizon's West Street facility in lower Manhattan, it needed to deploy its disaster-recovery team to the site.

AT&T had its switching node, which is essentially a mobile central office voice switch deployed on two tractor trailers, in New York within 48 hours.

AT&T's typical disaster-recovery plan calls for engineers to fly into the disaster site from parts of the country that were not affected. That was more challenging in this case because flying was not an option. Deacon says staffers drove to New York from as far away as Jacksonville, Fla. Engineers manned the mobile facility, working 12-hour shifts, for six weeks.

"We spent $300 million to have the ability to recover a switching facility," he says. "We have tractor trailers in four different geographic areas in the country that can be accessed from any point within the contingent U.S. within 24 hours," he says.

Verizon, which suffered the bigger blow after the attacks, is making progress. The core of Verizon's network in downtown New York will end up in better shape than before the attack because many components are being upgraded. Asynchronous technology is being replaced by SONET. And many buildings, which formerly had copper connections, will now have fiber.

There also won't be as much reliance on the company's damaged West Street central office.

"We've had to increase the infrastructure in the other central offices with West Street down, and they will continue to take a heavier load in the future," says John Bell, senior vice president of network operations at Verizon.

Details of preparing for another catastrophe will differ from one carrier to the next, but the foundation of all plans appears to be common, experts say.

"One assumption that has absolutely changed," Equinix's Adelson says, "is the notion that you can build a data center so well that it can withstand anything."

Senior Writer Michael Martin contributed to this story.