Chris Crosby of Compass Data Centers blogged about the many companies that issued press releases saying their data center ran without a hitch during Hurricane Sandy. He likened it to advertising that an aircraft had landed safely and said the real shame was that few were vocal about problems experienced throughout the event.

Equinix, however, used the media to its benefit during Sandy. Even newsdesks received regular updates – mainly highlighting outages – which became even more detailed as the event ran its course.

While it may sound strange that a company that prides itself on its resiliency would promote an outage, the approach ended up benefiting Equinix, according to regional operating chief of the Americas Raouf Abdel.

“All customers were in the midst of their own disaster recovery (DR) plans, and having accurate, timely information was instrumental in their ability to plan and react with their own DR measures,” Abdel says.

“Without information they were handicapped in their ability to carry out these. Customers wanted to know if we were still running on generator power, how much fuel was on site, how many deliveries did we have coming in and when – for us the issue with Sandy was really about maintaining power.”

Lights out
While Equinix suffered a minor roof issue in its New Jersey data center, it was its NY9 data center at 111 8th Avenue in New York that suffered most. Equinix did not run the building – it is a carrier hotel. Google purchased the 270,000 sq m, 15-story building in 2010. The problem at NY9 lay with the building infrastructure itself, and in most cases the situation was out of Equinix’s hands.

“When power went out on the evening of 29 October, Manhattan lost power for the most part until 2 November. We had several outages at one part of the building NY9 is in, and this was largely to do with building infrastructure we do not own or control,” Abdel says.

“We had fuel supplies, but there were equipment failures in the building that occurred several times throughout the week and a major fuse that blew, which had a severe impact on us,” he says. “Manhattan was devastated for a week, and lots of employees suffered personal damage, so it was a difficult situation. For some the recovery was still going on weeks later.”

Equinix had five sites in the New York and New Jersey area on generator power following for a number of days. A failed generator also affected NY8, but the generator was brought back up the same day. It could have been a lot worse. The storm hit landfall on the coast of New Jersey around Atlantic City – due east of Philadelphia and 50 to 60 miles north of Washington DC. Equinix has a number of data centers in the DC area, two in Philadelphia and eight in New York and New Jersey.

“We got heavy rain in DC, and in Philadelphia, the impact was not too severe but the storm did head to North Jersey and Manhattan, and that is where the impact really occurred. We had prepared for this a number of days in advance, fuelling all our generator tanks. We kept people onsite because we expected limited access to facilities; we prepared basic supplies for them and organized spare equipment in case of a failure as it is difficult to get vendors and suppliers to ship or deliver spares in these events,” Abdel says. “You never know what to expect but, frankly, we had not anticipated a storm of this magnitude.”

Since the event, Equinix has taken some of the lessons learned, which Abdel says reinforces current procedures, from having drawings of the buildings on board to setting up lines of communications with end users and suppliers. Equinix has had talks with its landlord at NY9. “We realized we needed to have a tighter relationship, one we could call on during emergencies, with more appropriate responses during these events.” –