‘Bucket brigade' keeps Peer 1’s Manhattan data center up’

Staff run buckets of fuel by stairs to 17th floor generator

31 October 2012 by Yevgeniy Sverdlik - DatacenterDynamics

‘Bucket brigade' keeps Peer 1’s Manhattan data center up’
Generator on the 17th floor of 75 Broad St. in Manhattan. Source: Squarespace

Three companies’ staff were able to catch a break around noon EDT on Wednesday from carrying buckets with diesel fuel to the 17th floor of the Manhattan building that houses a Peer 1 data center.

The Lower Manhattan neighborhood of New York City has been without power since Monday evening, when Con Edison (the local utility) shut off electrical services to protect its infrastructure and customers’ equipment from damage by Hurricane Sandy.

ConEd said today it expected to power Manhattan back up within three days. Peer 1’s data center at 75 Broad St., however, may not have that much time.

“As we’ve said before, this situation is untenable,” a status update posted around noon by Squarespace, one of Peer 1’s tenants involved in fuel delivery to the backup generator, read. “We can’t keep manpower going 24/7 for days.”

The “fuel brigades”, as the Squarespace blogger called them, were running fuel upstairs manually because flooding had submerged reserve fuel tanks in the basement and damaged pumps that would normally deliver fuel to the backup generator.

The brigades consisted of Peer 1 and Squarespace employees, as well as the software company Fog Creek, also a tenant at the facility. Squarespace’s core product is a web publishing platform.

Around 5pm, Peer 1 was hard at work installing a pump system to the 17th floor, according to a status update from Fog Creek. By this time, several hundred gallons of spare fuel had been delivered to the 17th floor in addition to a full tank.

Squarespace said they were “cautiously optimistic” that the pumping system being installed would start pumping fuel upstairs by the end of Thursday or Friday morning.

“Water is being successfully pumped from the basement at a rate of one foot per hour,” read a 5pm update from Squarespace. “Fifteen feet of water remain.

“We have increasing confidence that we may avoid downtime.”

Enough people have signed up to form a big bucket brigade expected to start running up and down the stairs at midnight.

Another data center services provider at this location is Internap, which has been working together with Peer 1 to ensure trucks deliver enough fuel. Unlike Peer 1, Internap data center in the building did not manage to ride through the Monday-evening flood.

“As a result of the flooding, both our redundant fuel pumps and our generator fuel tank were compromised and shut down,” senior VP of development and operations at Internap, wrote in a blog post. “The system continued to run until all fuel within the secondary feeder tanks [was] exhausted, and our facility lost power.”

In an update posted Wednesday, Orchard said the data center was back up but did not mention how long of a downtime window it had gone through.

Internap’s other New York City data center, located in a massive Google-owned building that houses data centers, network infrastructure and office space at 111 8th Ave. went down Tuesday night. Orchard said this outage was also caused by a malfunctioning fuel system.

“When the issue occurred, the fuel pumps could not provide diesel to the rooftop generators, causing them to stop supplying power to our UPS system,” he wrote. This data center was also back online on Wednesday.


Sign in

Forgotten Password?

Create MyDCD account


region LATAM y España North America Europe Em Português Middle East Africa Asia Pacific

Whitepapers View All