OVHcloud has announced plans to begin reopening data centers on its Strasbourg site next week, after yesterday's disastrous fire which has caused more damage than at first thought.
As well as destroying the whole of the SBG2 data center, the fire destroyed four rooms out of 12 in the adjacent SBG1 data center. Power to all four data centers on the site is off, and the UPS at SBG3 has been shut down according to OVH's status page for the incident. OVHcloud hopes to bring power back to SBG1 and SBG4 on Monday March 15, and it will take until Friday March 19 to restore the UPS at SBG3.
No cause has yet been announced for the fire, which destroyed the SBG2 building in six hours on Wednesday morning before it was extinguished by a team of more than 100 international firefighters working with a pump boat on the Rhine.
Updates:
OVH fire: SBG3 servers come to life as data center cleanup continues
OVH fire: OVHcloud abandons efforts to restart SBG1 in Strasbourg
OVH fire: while police investigate the cause, OVHcloud promises free backups in future
OVH plans recovery after disaster
"Many thanks for all the empathy messages you sent us today!" tweeted OVHcloud founder Octave Klaba. "I want to thank the teams who have been working all night/day. It's been the worst day for the last 22y and there is no word strong enough to say how sorry I feel today. We keep working hard to restart SBG1 / 3/4 asap!"
OVHcloud customers with servers in the Strasbourg facility will be offered replacement servers elsewhere in France: "We have a stock of new servers at the Roubaix and Gravelines sites, ready to be delivered to the majority of affected customers," says the most recent OVHcloud statement. "We will further enhance availability in these datacentres, with the production of nearly 10,000 new servers in the coming weeks. Affected customers will be notified about this process as soon as possible."
The site's fibers have been checked and were not affected, so it will be possible to get the facilities back online once the network equipment is patched and any repairs made.
Overnight, new network equipment was sent to Strasbourg to set up a temporary network room. The worklist includes rebuilding a network room referred to as "network room B (SBG5).".
To restart the Strasbourg site, OVHcloud's plans for the next two weeks include rebuilding the 240V circuits for SBG1/SBG4, a process expected to be done by Monday March 15, and rebuilding the 20kV power supply at SBG3 which should be done by March 19.
Although SBG1 lost four rooms, the network room survived, which means it can come back online with SBG4.
3.6 million sites were offline
During the initial disturbance 3.6 million websites, on 464,000 distinct domains were taken offline by the fire, according to network monitor site Netcraft.
Websites that went offline included government websites such as the Polish Financial Ombudsman; the Ivorian DGE; the French government procurement site Plateforme des achats de l’Etat; the Welsh Government’s Export Hub; and the UK Government’s Vehicle Certification Agency website. The latter site moved swiftly, getting a new SSL certificate by 10am and shifting to a UK hosting company.
The causes are still unknown, but The Register points out that SBG1, SBG2, and SBG4 had a three-and-a-half hour power outage in 2017, prompting a major upgrade to the power system on the site, and quotes a Gartner analyst speculating that an UPS failure may be the root cause.
Other Internet sites have scoured the OVHcloud update pages and noticed that OVHcloud has been replacing power supply cables at other data centers. "We are going to intervene in the bay in order to replace a set of power cable that may have an insulation fault," says an update for the Beauharmois data center (BHS-1). A similar set of cables were replaced at Limburg (LIM-1) yesterday, and similar updates have been spotted at other data centers back to December.
With an incident like this, an official investigation has to take place, and any connection with these maintenance issues can't be assumed. If nothing else, one might expect OVHcloud to prioritize any and all maintenance work with any connection to fire prevention before the full cause is known.
OVHcloud is planning to float on the stock market with an IPO; no change has been announced to this, given that it won't be taking place until the end of this year at the soonest.
For more breaking data center news, features, and opinions, be sure to subscribe to DCD's newsletter
