As IT hardware and software continues to evolve at a mind-boggling pace, innovation in areas of electrical systems powering that evolution is a much scarcer phenomenon and downtime as a result of a power failure rightfully remains one of the industry’s top bugbears. And there have been plenty of reminders of this in the past few months.
From a fire in the switch-gear room of a large Seattle-based plaza to a less-than-smooth transition to generator power at an Equinix facility in Sydney, Australia, during a blackout that affected most of the nation, causes vary but painful effects remain the same: each instance of data center downtime can lead to interruptions for thousands of customers and huge financial losses.
“We treat them [blackouts] all the same,” said Klee Kleber, vice president of marketing and product development, at Rackspace. “All hands on deck. Fix it as soon as possible.”
The most severely felt (and publicised) data center outage was caused by a fire in a switch-gear room at a large data center, office and retail plaza in Seattle. According to a summary of the incident put together by Adhost – whose data center was shut down as a result – the fire burned out two bus ducts of 5,000 amps each. The heat activated the water sprinklers, shutting down switch gear that provided power to the building’s three central electrical risers and was responsible for automatically switching between street and generator power.
The outage occurred shortly after 11pm on 2 July and by about 3.30am on 3 July it became apparent that the engineers would not be able to use on-site generators to restore power to the building. By about 1.45am on 4 July all three risers were energised by the temporary generators that had been rush-ordered.
CyberSource Corporation’s Authorize.net data center, located at said plaza, experienced the shut down of its online payment service for more than 11 hours, CyberSource spokesman Bruce Frymire said.
“If there is anything lucky about this – and there isn’t much – the timing of the outage happened when most traffic was at a relatively low ebb,” Frymire said. The outage happened during a holiday weekend.
CyberSource engineers undertook a failover from Authorize.net data center in Seattle to the company’s new space in San Jose, California, but the process took longer than expected because “a number of unanticipated errors”, associated with final testing and configuration processes that were ongoing at the back-up site. Those errors, in combination with delays in access for the company’s engineers to the Seattle site, resulted in a longer-than-expected downtime.
Another data center whose owners and engineers slept with one eye open was a Rackspace facility in Dallas, Texas. It experienced two interruptions within a space of nine days.
First, on 29 June, the facility lost its utility power feed. According to an incident report, the loss occurred when the breaker on the primary utility feeder tripped.
Once the data center was switched to generator power, four of its eight generators failed to synchronise – causing an “excitation failure” – and had to be shut down, said Kleber. Downtime for customers in the section of the data center powered by the disagreeing generators lasted between the point they were shut down and the point engineers were able to switch back to utility power. Initial attempts to switch to the facility’s secondary utility feeder failed because of an issue with a padmounted switch.
Initial power loss occurred at about 2pm and the final module was brought back online by 9pm that evening, according to the incident report. The following day Rackspace filed a document with the US Securities and Exchange Commission, notifying commissioners that it intended to issue $2.5-3.5m-worth of service credits to many of its customers who were impacted by the downtime.
More downtime in the same data center came the following week when a bus duct (the same part that caused the Seattle mayhem) stopped conducting electricity. Luckily, most of the earlier generator issues had been resolved and Rackspace engineers switched over to generator power until on-site consultants put together a temporary bus duct, said Kleber. The incident led to less than one hour of downtime for customers in the affected portion of the facility.
SYDNEY AND PARISEquinix data centers in Sydney and Paris also experienced outages in July.
The Sydney incident on 2 July was caused by a combination of a power outage across a large geographical area in Australia and a delay in switching a portion of the Equinix SY1 facility to generator power. One of the affected colocation customers was a large Australian VoIP provider, MyNetFone, which experienced downtime of 1.3 hours, according to the company’s web site.
The Paris incident, according to news reports, was caused by an error during maintenance of a UPS system by a vendor.
Equinix issued the following statement: “On Thursday 2 July 2009, Equinix experienced two unrelated incidents at its Paris 2 (PA2) and Sydney 1 (SY1) IBX centers.”
“The PA2 incident occurred at 11.33am and resulted in the loss of power to customers in one section of the center for approximately one minute. The outage was caused by the human error of a vendor that was conducting routine maintenance on a UPS system. The error caused one UPS unit to go offline, impacting the delivery of power to approximately one-third of the PA2 center. Within one minute of the initial loss of power, all systems were restored.”
“The SY1 incident occurred at 10.56am and resulted in the loss of power to customers in one section of the center for approximately 12 minutes. The outage was caused by the failure of a UPS system to deliver power after the utility providing power was interrupted in a statewide power outage with Energy Australia. This UPS unit serviced approximately one-third of the center.”
Related News:
Rackspace suffers Dallas outageRelated News:
Tenants in Seattle's Fisher Plaza powered by generators after weekend outageRelated Event:
DatacenterDynamics Paris, Command and Control - December 10th 09Related Event:
DatacenterDynamics London, November 10/11th 09Related Event:
Datacenter Dynamics Dallas, December 10th 09
