A poorly executed maintenance operation on a data center’s UPS system caused an outage in one of Vocus Communications’ Sydney facilities in the early hours of the morning, according to sources quoted by iTnews.
The power at the data center in Alexandria, New South Wales, went down around 7am AEST; Jaden Roberts, a network engineer at Servers Australia - a company that leases space there - posted on the company’s status page, saying that the company was working with Vocus to identify the root cause of the problem, and notifying customers “wishing to move from this facility immediately” that it had “made emergency space available” elsewhere.
Alexandria is one of the company’s smallest sites in terms of server count.
Whoops
At 10:15am, Servers Australia’s managing director, Jared Hirst, posted on industry forum Whirlpool, stating that there had been “a complete loss of power at the Vocus data center whilst they had scheduled maintenance,” and that “all A and B power feeds” were lost “for a brief one minute.”
He added that he was “extremely disappointed” by the event, and that all of the site’s servers and switches “have A and B power to avoid outages, especially during maintenance.”
Hirst continued: “I will not tolerate a loss of both A and B power under any circumstances, as this means that there is no redundancy. We have enacted an emergency data center move for any customers in this site and will be vacating this site within the next few weeks.”
“All services are coming back online now, but due to the inrush of power, there has been a huge amount of damage to PSUs. Engineers have been on-site since this morning replacing servers. There is a total of around 500 servers at this site, and we are around 60% complete in checking them all.”
In his final post on the company’s status page at 2:20pm AEST, Roberts stated that engineers were “still working on the remaining list of services” that were down.
Meanwhile, Vocus reported that not all equipment had been recovered, with an ongoing router failure on its Ipera network.
From bad to worse
The incident comes at a time of upheaval for Vocus, which recently announced its intention to sell 20 Australian data centers and the entirety of its operations in New Zealand.
The sale is hoped to make up for losses of around AU$100m (US$78.7m) made last year, mostly attributed to the failed integration of recently acquired companies - Nextgen Networks, Amcom and M2 - as well as a decline in the resale of copper-based services to the country’s national broadband network.
DCD has asked Vocus for more information about the incident.