Fuel-delivery system for a group of back-up generators at a major Manhattan connectivity hub has been fixed and power was restored to a section of the building after three outages for one data center and a dangerous rise in temperature at another.
An Equinix data center at Google-owned 111 8th Ave. has gone down three times since utility power at the building was shut off during Hurricane Sandy’s landfall in the US northeast Monday night, Raouf Abdel, Equinix regional operating chief of the Americas, said in an emailed statement.
This happened because of a fault in a fuel-pumping system that feeds fuel to a subset of generators at the building.
The same generator subset was backing up the cooling system in one area of the building. Low flow of fuel to the generator and the subsequent drop in the load it was generating brought down this cooling system, causing temperature in one of the suites in the building leased by the network and colocation provider Zayo Group to rise beyond 100F at one point on Thursday.
Rising temperature in Zayo’s Suite 734 on the building’s seventh floor was putting at risk the provider’s networking equipment, as well as customers’ equipment in the Zayo-operated colocation space.
“The root cause of the problem with the generator was a faulty isolation valve, allowing air into the fuel supplied to the generator, resulting in sub-optimal operation,” Zayo representatives wrote in a status update posted online. “The generator was no longer able to support the load of both the networking equipment and the site air conditioning systems.”
By placing fans on the data center floor, opening windows and shutting down network equipment not serving customer traffic, Zayo was able to bring the temperature down to 95-100F and keep it there.
Around 2pm EDT, the generator was taken offline for repairs. To ride through this window, Zayo’s data center was powered by the company’s own battery plant, not shared with Equinix.
A 2MW mobile generator was ordered, but the delivery from New Jersey took a long time because of gridlock on the region’s roads.
Meanwhile, Equinix started planning to make permanent repairs to the pumping system. In an update posted around 7pm, Zayo said those repairs had been made and facilities managers had started to bring the cooling system back online in phases.
With only the first cooling unit back up, temperature in the data center had dropped by eight degrees.
Equinix also had trouble with its data center at another major New York City connectivity hub at 60 Hudson. The data center went down Thursday morning as a result of generator failure, impacting a number of customers, Abdel said.
The generator has been restarted and the data center has been brought back online.