A fire at a Cyxtera data center in Boston that took down Oracle NetSuite services across the US was the result of an electrical arc flash that blew the doors off a battery cabinet, DCD can reveal.
On 14 February, Waltham Fire Department was called to an electrical equipment fire at a data center run by Cyxtera. The building was undamaged, but inside an explosion had destroyed a battery cabinet.
The fire services evacuated the building, shut down power, and ventilated the premises to remove poisonous gases, before checking the damage and eventually allowing operations to resume. One person was taken to hospital with possible smoke inhalation.
Cyxtera's BOS1 campus offers 16MW of capacity across 40,500 sq ft of raised floor space across three buildings at two nearby sites in Waltham, Massachusetts. The company has declined to comment beyond its initial statement, reported on 16 February, and the Fire Department says the cause of the fire is "undetermined."
Deputy Richard Grant led the response to the data center at 1 First Avenue in Waltham: Upon arrival, fire crews found smoke coming from a door where an electrical supply was situated. "The building was evacuated," says the fire report.
At this stage, Cyxtera staff told the fire crew that power and backup were still on, and were instructed to switch everything off before anything else could be done. The building has three electrical services (A, B, and C).
When fire crews entered, they found that breakers had tripped on the B service, and an explosion and arc flash had caused smoke and fire damage.
"There was damage to other cabinets adjacent to the origin cabinet caused by panels blowing off at the time of the explosion," says the report.
Smoke detectors and a fire alarm system were set off, and the Fire Department arrived at 11:34am local time, six minutes after the alarm was raised. The incident was cleared up by around 14:30.
The report notes that work was being carried out on the A system, in the same electrical control room, but this "appears to be unrelated to this incident."
Luckily these workers were on an early lunch break at the time of the explosion, so the only person in the room at the time of the explosion was a 28-year-old security guard on a routine patrol. He remained fully conscious but was taken to hospital to check for possible smoke inhalation.
Meanwhile, Cyxtera manager David Armstrong reported seeing sparks from the B service following the explosion.
"It is the collective opinion of the investigative team that the cause of the fire will be undetermined," says the report. " It was most likely caused by an unspecified electrical event in the B service causing the breaker to trip under load. After this event occurred, the generator power was automatically initiated and back fed the main breaker causing it to arc and burn inside the cabinet."
Despite the burning breaker, the report says "there is no evidence of an incendiary event."
Staff were not allowed back in until the building had been fully ventilated, as hazardous levels of carbon monoxide and hydrogen cyanide were found inside.
"The investigation is considered closed until such time as further evidence may warrant its reopening," concludes the report.
Last year, three people were left in critical condition after a substation arc flash electrical explosion next to Google's Iowa data center.