OVHcloud has given the police crucial electrical equipment from its SBG2 data center which burnt down last week. As shocked customers react to the loss of their data, OVHcloud founder Octave Klaba has promised better policies on backups in future.

The UPS systems from the SBG2 data center in Strasbourg have been given to the police, along with batteries and fuses, for investigation by OVHcloud's insurers, said Klaba, in a ten minute video posted on Twitter yesterday. The video detailed efforts to restart the three surviving data centers on the site, and addressed customers' surprise that data had been lost, promising that free backups would be included in all OVHcloud services in future.

Klaba promised a further announcement on Friday which will give more information on the root cause of the incident.

Updates:

OVH fire: SBG3 servers come to life as data center cleanup continues

OVH fire: OVHcloud founder promises data center fire lab, gives server restart update

OVH fire: OVHcloud abandons efforts to restart SBG1 in Strasbourg

UPS still the likely cause

"On Friday I will post another video going deeper into the root cause," said Klaba, but added no new information: "Right now, the police, the insurance and the experts are checking the facts."

The day after the fire, Klaba had suggested the cause of the fire could be in the data center's uninterruptible power supplies (UPSs), when he revealed two things: UPS7, which served SBG2 received maintenance on Tuesday, and both UPS7 and UPS8 were ablaze a few hours later, when the firefighters arrived.

In yesterday's message, the OVHcloud founder said the UPSs from SBG2, along with their batteries and fuses, were being handed over for examination, along with all OVHcloud's video relating to the fire for a definitive answer: "We will have the conclusion of this investigation hopefully in the next weeks."

Following the discussion, many customers have been surprised that disaster plans are their responsibility and that their data has been lost: "Some customers don’t understand exactly what they bought," said Klaba, announcing that all customers will have free backup included in their service going forward, and suggesting that this issue will have to addressed more widely by other providers.

"We will start on two projects," he said. "The first is to improve the security of our existing data centers. The second point is about the backups." In future, OVHcloud will deliver "higher security backups for all customers in the different data centers," offering backup by default, without payment.

"This incident will change our way of delivering the services, but I believe it will also change the standards of the industry and the market. And it will increase the security of the backup by default without payment," he said

Rebuilding with the insurers

"We are working with the insurance to be allowed to restart," said Klaba, saying the three surviving data centers, SBG1, SBG3, and SBG4, will be restarted "very quickly", and detailing the process by which high voltage power and network services are being restored to each building, and servers restarted rack by rack. After the provision of basic services, "we are working with experts from our insurance to restart different rooms," said Klaba.

"The high voltage is now rebuilt and verified," he said, confirming that the three data centers now have 20kV feeds.

"Now we are finishing checking the UPSs in the different data centers," he said, promising that today (Wednesday) all the UPSs would be checked and validated for use.

The site now has network links with Paris and Frankfurt and the LANs within the buildings are being rebuilt. Water cooling for the servers is checked and working, as well: "There’s no issue on that side," according to Klaba. The next step is restarting one test rack, before expanding the effort over the coming weeks.

Some servers are having to be cleaned up because of smoke damage (especially in rooms 61E and 62E) but, offering users a crumb of hope, Klaba said: "Data seems to be still on the hardware. It will take a few days to reveal all the servers and replace all the mother cards, or find a way to clean up the mother cards... Each server has to be checked to see if there was an issue on it, even if it seems to be clean."

While SBG1 and SBG4 are relatively simple, SBG3 had smoke on the fifth floor, which needs to be cleaned up.

To set displaced customers up in its other French data centers, OVHcloud is running extras shifts at its Roubaix factory and, by the end of this week, will have delivered 6500 new servers. It has sufficient parts on the way: "We have secured the supply chain... to deliver the 15000 servers the customers need."