Testing your mettle: Zinc whiskers in the data center

It's been more than a year since Wayne Harris and his IT comrades at a Canadian healthcare organization exorcised the little demons, but the memories still haunt them.

"We spent an average of 40 hours of overtime a week banging our heads against walls trying to figure out what the heck was going wrong with our servers," says the manager of technical services for Baycrest Centre for Geriatric Care. "We wondered if we were being sabotaged."

In a sense they were, but by the most unlikely of suspects: microscopic metal strands called zinc whiskers that were growing on the bottom of the data center's raised-floor tiles.

It all started in 2002, shortly after the Toronto company had an outfit in to clean up its data center.

"A couple of weeks later our servers started failing. Motherboards, hard drives, you name it," Harris says. "We put in new boxes and sure enough, they failed too."

Harris says the IT team, which at the time was overseeing a collection of about 50 servers, exhausted all avenues trying to solve the mystery. Finally, the mention at a conference of a similar problem led Baycrest to find that when the cleaning crew raised the data center floor tiles, the conductive zinc filaments - just a few millimeters long and a few microns in diameter - went airborne, short-circuiting the servers.

"We try to spread the word now," says Harris, who estimates Baycrest spent at least $100,000 replacing floor and ceiling tiles and giving the data center a deep cleansing. "We don't want others to go through what we did."

While metal whiskers were new to Baycrest, they've actually been known since the 1940s when Bell Labs discovered them in telecom environments. Zinc whiskers are thought to "grow" as a result of molecular stress, whereby the zinc used to keep steel on the bottom of the tile from rusting tries to separate itself from the steel. Whiskers have been found to form in a vacuum, but heat, humidity and other environmental factors also have been suggested as triggers. The metal filaments have been discovered growing in cabinets and other data center spaces.

Like many IT problems, zinc whiskers aren't something that companies victimized by them often discuss openly, perhaps for fear of making the IT infrastructure appear vulnerable or the IT management team seem negligent. As a result, many IT shops don't even think to look for whiskers when data center equipment goes on the blink. In fact, looking for them is pretty tough in the first place because they are barely distinguishable with the naked eye from dust. However, by shining a light parallel to the bottom surface of a zinc whisker-covered floor tile will let the viewer see the whiskers, or more precisely, reflections of them.

David Loman, a power and environmental specialist for HP, speculates that if he asked 10 IT managers about zinc whiskers only two would know what they were. "When I tell people they've got zinc whiskers they look at me like I've grown antennas out of my head," he says.

Loman says one way that zinc whiskers are identified is through a distinctive popping sound that power supplies emit as they are snuffed out by the whiskers. He recalls one customer whose data center lost dozens of power supplies after an old upflow air conditioning system and a new downflow one were turned on at the same time, scattering zinc whiskers everywhere. "It sounded like popcorn," he says.

Those familiar with zinc whiskers say it would behoove IT shops to study up on the contaminant. While the sort of electroplated wood-core floor tiles thought to have spawned most zinc whiskers are for the most part no longer being made or installed, plenty of older tiles remain in data centers. What's more, new compact data center gear, such as blade servers that squeeze components into smaller spaces, are thought to be more susceptible to whiskers.

"I thought the problem would have peaked once manufacturers ran out of the old tiles, but over the last couple of years I haven't seen the problem abate. I think it's grown," says Rich Hill, who heads up a data center cleaning company called Data Clean that comes across a zinc whisker problem about every two weeks.

"People had mainframes for years without any problems from whiskers," Hill says. "Invariably, the newer equipment is what has the problems."