At the end of October New York, one of the world’s most important data center markets, experienced a test of a lifetime. Many data center professionals compare Sandy’s impact on the industry to that of 9/11. It was a disaster of epic proportions, and not everything went according to plan. Many data centers failed – regardless of elaborate backup power systems and continuity processes – and some stayed down for several days.

Datagram takes a hit

One of the most widely publicized outages was the protracted downtime of a data center owned and operated by Datagram at 33 Whitehall St. in Lower Manhattan. According to anecdotal accounts, there were a great many data center outages; however, very few data center operators actually spoke publicly about the incidents. One of the few operators that did was Datagram. That, in combination with the fact that it hosts a number high-profile customers, such as Gawker Media and AOL’s Huffington Post, made Datagram’s outage one of the biggest Sandy-related data center stories.

Con Edison (ConEd), the utility company that serves New York City, shut power off in Lower Manhattan pre-emptively around 7pm local time on 29 October. The company said it did so to protect its own and customers’ equipment from potential damage by Sandy, which was preparing to make landfall on the US Northeast that evening.

Once utility power was shut off, Datagram’s data center was running on generator power. Several hours later, however, it went completely dark, Alex Reppen, the company’s CEO, recalls. He and three other staff members were on site at the time. The data center lost power because fuel storage and pumping equipment in the building’s basement had been damaged by a massive flood Sandy had caused. The Datagram team, however, had no idea there had been a flood downstairs.

The thing was, they were on the 25th floor of the building, which is where their data center is located, and the area of Manhattan surrounding them was completely dark because of the power outage. They simply could not see that the streets had been flooded. “We didn’t see the water,” Reppen says. “We were up high and couldn’t see anything.” Once they checked the generator on the 31^st floor, they realized there was no fuel pressure, at which point they went downstairs and saw that the building’s lobby had been flooded with about five feet of water.

The outage started on the evening of 29 October and the data center was powered back up by a generator four days later, on 2 November, says Mohammad Soliman, Datagram COO. Because the neighborhood had been flooded with salt water, not only did all the water have to be pumped out before any electrical equipment could be plugged in, all electrical infrastructure had to be checked for damage from salt water. “They weren’t going to let a generator be parked anywhere on the street while the city and ConEd were still pumping,” Reppen explains.

The building on Whitehall was one of Manhattan’s several major data center and connectivity hubs that lost utility power. Other buildings include 111 8th Ave. (owned by Google) and 60 Hudson St. Both buildings serve as major network interconnection points and house lots of data centers. Some data center operators have data centers in both of these carrier hotels. One of these operators is zColo, a subsidiary of Zayo Group.

zColo’s data centers in Manhattan rode through the outage without downtime, but that is not to say there were no issues. “We definitely had some adventures,” says Nat Tafuri, VP and head of engineering and operations at zColo.

The company leases multiple spaces at 111 8th Ave., including one it shares with Equinix on the seventh floor. Utility power was switched off and the data center was running on generator power. The generator that backs up both Equinix and zColo data centers is controlled by Equinix.

Equinix lightens the load

At one point, the generator started having “hiccups” as air was getting into the fuel pumping system. To prevent the machine from shutting off completely, Equinix engineers lightened the load on it, which resulted in zColo losing some of its cooling capacity and temperature on its data center floor going up. This happened three days after ConEd had switched off power in Manhattan. At one point that day, temperature inside zColo’s data center reached above 100F. zColo staff had to scramble – putting fans on the data center floor, opening windows, shutting down some equipment – to keep the temperature from rising. They ordered a mobile 2MW generator from Pennsylvania. Equinix eventually took its generator offline and made repairs. Meanwhile, zColo’s data center was transferred onto the company’s battery bank.

Tafuri supports Equinix’s decision to bring down the load once there was an issue with the generator. “It was probably the right move to shed some load,” he says. That was a simple choice between losing power to some equipment and losing power completely. Most of the equipment in this particular zColo space is network transport equipment, which has a fairly high tolerance for high temperatures.

The generator they had ordered arrived the same day and was plugged in as backup for the Equinix generator. On Friday, the building was put back on utility power. A major challenge for all facilities running on generators through the week (and some much longer) was getting enough fuel for those generators.

“Fuel was crazy,” Tafuri says. While 111 8^th Ave had enough fuel onsite, other buildings, such as 60 Hudson St. and 165 Halsey St. in Newark, New Jersey, did not. zColo staff had to stand outside of 60 Hudson and beg fuel truck drivers who were making deliveries to other buildings to deliver some for them. “Trucks would pull up with big tankers and try to dump all the fuel they could at one location,” Tafuri says.

The fuel challenge

Another issue was the cost of fuel. Running a data center on generators for multiple days around the clock costs a pretty penny. Datagram was spending about US$10,000 per day on generator fuel. During the last week of November, it was still running on generator power. While the company will not be getting a power bill from ConEd for the month, it is little consolation, as its total monthly utility bill is usually between US$75,000 and US$100,000.

While Sandy is going to cause people to rethink how data center space is designed and operated in Manhattan, it is not likely to make it a less desirable data center location, Reppen says. Manhattan is one of the main interconnection points between the US and the rest of the world.

Hunter Newby, a long-time network connectivity entrepreneur who lives in New York, says many core network interconnections exist in Manhattan and will continue to exist there. It is such an intricate system of interdependencies, with so many players involved, that it is nearly impossible to move it elsewhere.

“It’s hard to replicate it elsewhere, when you’re dealing with so many independent entities that would all have to get together and move at the same time,” he says.

Many rules about operating a data center in Manhattan will have to be reviewed, among which is the city’s restrictions on how high fuel can be stored in a multi-story building. This is a rule that was enacted after the terrorist attack on 11 September 2001. The rule, Reppen says, is unfortunate, as there were many data centers in Manhattan that did not go down, precisely because they had fuel on rooftops in tanks that were installed prior to 9/11.

Datagram won’t wait until the city reviews its rules. The company has made plans to install an extra set of fuel pumping and storage equipment on its mezzanine level, which is 65ft above street level – they will be redundant systems for the ones in the basement.

Another big issue is generator fuel, says Newby. While generators and failover systems are built to withstand utility power outages, what if the power outage lasts longer than a week and there is a fuel shortage? “How long can you run on a generator?” he says.

The city has also learned something about water; salt water destroys electrical equipment a lot faster than fresh water, as residents of Long Beach, New York, discovered after the salt water damaged the electrical systems in their cars. Data center operators and building managers at 33 Whitehall and 75 Broad St. are dealing with salt water corrosion too.

Data center provider Peer 1 has surely learned a lesson or two at 75 Broad St. As was widely reported, the firm’s staff, together with other companies’ staff, had formed “fuel brigades” that carried buckets of fuel to a generator located on the 17th floor of the flooded Manhattan carrier hotel.

Learning from others

Newby finds himself in a unique position to take advantage of these rewritten rules. He and two partners recently bought a building at 325 Hudson Street in Manhattan, which they are building out into a network interconnection facility. “We watched other people’s stuff get submerged and fail. Those [incidents] make you rethink where you’re going to put your generators, your fuel pumps, etc.,” he says.

New York has always been an interconnection hub, Newby says. Cities were built along waterways to enable commerce, which eventually led to submarine intercontinental cables to land in the major coastal cities. Complex webs of deals between network infrastructure owners, operators and customers have been woven in these hubs and it will take more than a hurricane for these ecosystems to be undone and rebuilt in other places. But the rules by which these ecosystems operate will change. They have to.

Learn more about Sandy’s wrath on the New York and New Jersey data center industry at DatacenterDynamics Converged New York on 12 March.

This article first appeared in the 27^th issue of DatacenterDynamics FOCUS magazine. Get a free subscription on our website.