Content delivery networks carry the heavy load

News sites credit Akamai, others with keeping sites available after Sept. 11 tragedy.

Steve White, CTO at MSNBC.com, had been leading his team through stress tests, beating up on his Web site as it prepared to handle millions of viewers during the upcoming Winter Olympics. He was planning to add servers early next year and had already cranked up to two gigabits of bandwidth to handle huge loads.

But on Sept. 11, shortly after two hijacked planes crashed into the World Trade Center in New York, White faced stress on his site that was no test. He turned to content delivery network (CDN) provider Akamai to help handle the sudden, unprecedented load.

Akamai "was absolutely critical" to handling traffic that reached 12.7 million page views by the end of the day, more than four times the number of views on MSNBC.com on a typical day, White says. Its previous record audience was the day after last year's presidential election, when MSNBC.com recorded 6.5 million page views.

It's a scenario that played out across the Internet as news sites were inundated with viewers. Most sites, such as MSNBC.com, typically use CDNs to serve streaming video and other images, but on Sept. 11, MSNBC.com wasn't alone in finding CDNs could help make all sorts of Web site content, including basic text, more accessible to visitors.

CDN numbers speak volumes

CDN providers such as Akamai, Speedera and Digital Island had been the darlings of the Internet boom, pushing content to their own networks of geographically dispersed caching servers on the edge of the Internet, bringing content closer to users and speeding content delivery. Akamai, for example, has nearly 12,000 servers in 62 countries.

While the fortunes of CDN providers have fallen with many of the dot-coms, it's not as if the providers haven't been signing up new customers. In fact, the number of companies using CDNs has doubled to 2,500 in the past year, says Peter Firstbrook, a senior research analyst at Meta Group.

The performance of CDNs on Sept. 11 may help that number grow. Akamai handled three times its normal load that day, as many of its news site customers increased their use of CDN services by pushing out more content, such as HTML pages, and new customers called to quickly add the service.

"There were so many corporations that were touched," says Paul Sagan, president of Akamai. "Unlike the news sites, they usually don't build their capacity for big spikes. It's a much more predictable kind of flow. They suddenly had great traffic, and we were able to help many of them very quickly."

Sagan wouldn't identify which corporations turned to Akamai, saying only that most were in the airline and rental car industry. Akamai's network proved its mettle that day, he says, the same day that company co-founder Danny Lewin died on American Airlines Flight 11.

To handle the increased demand, Akamai engineers distributed the authorized code needed to use the Akamai network to customers and balanced servers to ensure that there was enough capacity to handle content where demand was the greatest.

"That's what Danny [Lewin] was all about: Building the world's largest, most scalable network, and there's horrible irony in that fact that [Akamai] faced its biggest challenge and came through it in the same event that killed him," Sagan says.

Tricky transition

But news sites found that turning more traffic over to a CDN wasn't quite as easy as flipping a switch.

White and his engineers, for example, had to code HTML pages to be "Akamaized." In fact, MSNBC.com's team initially stripped down the site to highlight nothing but the top four or five news stories, but before long, White knew he had to turn to Akamai to handle more than the streaming video it usually serves for MSNBC.com.

"We had a rough patch early on in terms of the transition. When you have effectively five or six big, huge breaking stories coming every 15 minutes . . . I hesitate to say we should plan for that. But you know what? We should," he says. "We thought we would have some time to go through the process in a more paced manner."

White says the task will be to figure out how to transition things to the CDN more quickly. "How do we modify workflow and production process to ensure there is no downtime?" he says.

By midday, White says, Akamai was serving about 80% of MSNBC.com's traffic, which included about 12.5 million streams of video.

At CNN.com, Kenneth Craig, vice president of enterprise systems for CNN Internet Technologies, says he used Akamai to off-load images and other bandwidth-intensive content. All HTML was served from CNN servers in Atlanta, although CNN.com had to take servers from CNNfn.com and CNNsi.com to handle the load.

At Washingtonpost.com, there was a similar scenario. During an average day, the site runs about 20% to 30% of its content, primarily images and video, from Akamai servers. On Sept. 11, about half of the site's content had been pushed out, including text, which had not been pushed out to edge servers in the past. The site had intermittent outages for the first hour as it was slimmed and content was sent out to the CDN, says Eric Schvimmer, vice president of technology.

The same thing was true for ABCnews.com, which used Akamai, Digital Island and other CDNs to serve video and images but handled text at its data center in Seattle.

Doug Parrish, senior vice president and CTO for Walt Disney Internet Group, which owns ABCnews.com, says he tripled the number of Web servers to about 40, using systems from sites such as ESPN.com and ABC.com.

That, however, is where Akamai truly came in handy for sites that don't have such deep resources. "We would have had to have 10 times the capacity sitting idle for 364 days of the year. It's ridiculous," says MSNBC.com's White. "Akamai is a very elegant solution for not throwing money at hardware that you'd have to leave 90% of turned off most of the time."

Content delivery networks carry the heavy load

CDN numbers speak volumes

Tricky transition

Related Links