Out-of-this-world content delivery network

Tasked with figuring out how to deliver 40 hours of training annually to each of its more than 180,000 employees around the country, without spending a fortune, the U.S. Department of Veterans Affairs looked to the heavens - literally.

The VA found the answer to its training dilemma in the satellite network it used to provide unidirectional broadcasts to its major medical centers across the country, including Puerto Rico. The VA had served up four channels of live and prerecorded content, mostly training and VA news - essentially a private cable TV service - out of an uplink center in St. Louis. It used dedicated bandwidth on PanAmSat's Galaxy 10 satellite, explains Craig Davis, a computer specialist for the VA in Austin, Texas.

Advertisement:

Davis and a team of four others, including consultants from Cisco, IT Broadcasting and Northrop Grumman, figured the satellite offered them the best means of beaming bandwidth-intensive live and on-demand video content to individual desktops in the hospitals without disrupting the terrestrial WAN. Unidirectional broadcasting, in which each user gets a dedicated copy of the video programming, wasn't going to work. Heavily populated network segments would have clogged quickly, he says.

Multicasting, which is inherent in a satellite network, made the perfect solution. With IP multicast, the VA Employee Education Service (EES) easily could blast out a single version of a stream or file to each employee needing the training. With IP multicast, a single video stream gets sent to multiple LAN users.

Network World honors the VA as a User Excellence Award runner-up for its innovative approach in building a massive content delivery network (CDN) for e-learning programs. The $4.5 million project lets the VA meet federal and departmental training mandates while creating a better-educated workforce and reaping huge cost-savings over alternative methods.

Getting over the LAN hump
While IP multicasting made sense, Davis and his team had one big technical hurdle: figuring out how to get the multicast streams from the satellite receiver into the local network at each site and beyond. Enabling multicast in the LANs would be tricky because of the need to support the Internet Group Management Protocol (IGMP), an Internet Engineering Task Force standard for managing communication between desktops and the local multicast device. "Each medical center could have 1,000 users with 100 intermediate LAN switches. For multicast to work, each switch has to support [IGMP]," Davis says.

Compounding the problem, each of the 210 major medical centers and offices that the satellite network serves has an autonomous LAN design, making it hard to use a one-size-fits-all approach.

The team broke the problem down into more manageable pieces, which helped with the initial installations and future troubleshooting.

First, it considered the two types of content that needed to be delivered to the desktop. The four existing unidirectional channels (plus one used for special events) would be encoded at the St. Louis broadcast facility into MPEG 1 streams and multicast at a data rate of 1M bit/sec, each over Galaxy 10. Meantime, the on-demand video content would need to be cached locally, closer to the users, to make for a better viewing experience and reduce the amount of traffic traversing the WAN.

To manage the two content types, the team created what it calls a "multicast demilitarized zone" (MDMZ), where the satellite WAN meets the individual LANs. By placing the satellite receiver/decoders, routers and LAN switches in an MDMZ at each site, the VA gets around the IGMP problem. While live multicast channels are directed into the LAN, the on-demand content is routed to a Cisco Content Engine (CE507) and cached for later viewing.

The cached content - a video demonstrating blood-handling techniques, for example - is delivered upon request using unicast. Each content engine can store about 30G/bytes or approximately 60 hours of material at a time, with content rotated in and out by the EES. Users access the available content via a Web interface that IT Broadcast designed and view the streams using Cisco's IPTV player client.

In addition, the VA is putting content engines in the community-based outpatient centers that are linked over low-bandwidth connections - down to 56K bit/sec in some instances - to the main medical centers in each region. The VA can push big video files over the network during off-peak hours, and users can retrieve them locally on-demand, Davis says.

Currently, 95 medical centers are on the IP multicast CDN, called the VA Knowledge Network (VAKN). The remaining 115 will come online by year-end, Davis says.

Some benefits of being able to view training materials at the desktop are obvious: Employees don't have to crowd into conference rooms and can watch on their own time. Unlike with the original VAKN TV channels, the EES can track live and on-demand streams to see who watched, when and for how long.

"We have a back-end SQL database that tracks usage that we call our 'Nielsen ratings,' " Davis says. "We can drill down by region and medical center."

Beyond pie in the sky
Eight months into this massive CDN deployment project, and only about halfway through, the VA is seeing dividends. One instance alone saved the agency $800,000, Davis says.

Rather than sending 800 people to Chicago for a 16-hour Occupational Safety & Health Administration training class, the VA broadcast the session to multiple facilities. During the four-day period, the VA estimates it saved $1,000 per person, or a total of $800,000, he says.

Additionally, the VA can educate its employees on a broader basis. For example, after VA specialists attended the annual American Telemedicine Association conference, they held an eight-hour seminar in Long Beach, Calif., to share what they learned and related it the VA's needs. To make the event available to everyone in the VA health system, the team added the St. Louis broadcast facility as an endpoint in a standard ISDN videoconference. It took the feed, encoded it and pushed it live over the open channel on the multicast network to user desktops. "A single H.320 [ISDN] video call at 384K bit/sec is about $50 an hour, plus some money for the bridge that connects participants," Davis says. "Now, there's no additional cost for all the streaming users."

Overall, the EES expects to get a $25 million return on its CDN investment over the next three years by delivering content directly to users at their desktops, Davis says.

And Davis sees uses for the CDN beyond streaming video. Software distribution is an example. "Back when the Code Red virus hit, the VA reacted by shutting down its electronic border. Most LAN managers started looking for fix files on their own," he says. "In one fell swoop, we could have pushed the virus definitions to every site in the country in 5 minutes using the CDN."

The team also is looking at the newer MPEG-4 video standard for serving live content to the outer reaches of the WAN that have limited connectivity. MPEG-4 is better suited for low-bandwidth situations such as dial-up and even handheld and cellular devices.

Multicast's requirement that each router and switch in a network must support the proper protocols makes it difficult to extend beyond a LAN. Most ISPs and backbones do not support it natively. With a nationwide reach and some 200,000 potential users, the VA has created a scalable and efficient delivery network to meet its training and education needs.