OVHcloud has apologized to customers for the disastrous fire which destroyed one of its data centers in March - but says it can't reveal what caused the fire until 2022.

Michel Paulin, CEO of OVHcloud, apologized to customers in a video released yesterday, 50 days after the incident. He also praised the efforts of OVHcloud's recovery teams. In the same video, OVHcloud chairman and founder Octave Klaba promised to implement a resilient data center infrastructure, describing it in terms similar to the availability zones already in use at the major cloud services Amazon, Azure, and Google.

However, details of the resilient infrastructure were held off for a press release due next week, and no mention was made of the cause of the fire. The company clarified that this will not be revealed until 2022.

Under a cloud

"With regards to the underlying cause of the fire I anticipate the official information won’t be disclosed until next year," said an OVHcloud spokesman in response to DCD's request for clarification after the OVHcloud video released yesterday, explaining that this was because of "the involvement of the French authorities, insurance companies, etc."

Outage reports can take time, but 10 months is a long while. The Singapore government was involved in inquiries when a November 2017 failure caused a three-hour outage at the Singapore Stock Exchange (SGX) - but the report was given to SGX in four months (the following March), and published in June.

The delay till next year may have a serious impact on OVHcloud's plans to have an IPO later this year.

"We are very sorry for what happened in Strasbourg," said Paulin in the video, before praising his colleagues: "I would also like to thank the teams. They have worked, night and days be able to restore as quickly as possible, all our services."

OVHcloud has restored 118,000 of the 120,000 customer services affected by the fire, said Paulin, giving details of the restoration effort: "We have been able to produce up to 16,000 servers to be able to restore the services, and we had up to 200 people on the site. Now we still have 100 people there."

Paulin said OVHcloud wants "to answer to each customer one by one," and has reinforced its support teams to deal with commercial, technical, and billing issues; as well as setting up a dedicated team to field social media communications.

Availability zones?

In his part of the short video, OVHcloud founder Octave Klaba, whose family still owns 80 percent of the business, promised the company has a "strategic plan for the next five years" which would give customers free backups and resilient infrastructure. OVHcloud has promised to publish more details on the infrastructure next week.

"Of course, OVH has internal backups for all services, for our internal needs," said Klaba. "And we decided to upgrade the way that we do the internal backups."

Some backups were in the same data center, while others were in remote data centers. In future, he said "we will create a region with four data centers outside of all our data centers. And this is where we will post all the internal backups. So it will be completely totally outside of all our data centers in another area."

Customers will be able to restore their servers for free, from this backup, he promised: "We want to be seen as the expert of the backup."

In future, to provide a "hyper resilient" service, Klaba said OVHcloud would reinforce the data center, and build its new data centers according to new "regulations."

He also promised to use software to build "regions" with three data centers, "working together." Fuller details of OVHcloud's regions are due in a release next week, but the approach sounds very similar to established practices at AWS, Azure, and Google, which have had regions and availability zones for some time.

"The resilience will be based on the software," said Klaba. "And this first region that we offer to our customer will be in Paris." Further regions would follow in other European countries, as well as in the US and Asia.

Criticism builds

As OVHcloud continues to work to restore services criticism is building in the French technical community. An article in VO News by Clever Technologies accused OVHcloud of making a series of "gross errors," including :

• SBG4 was not independent, drawing power from the same circuit as SBG2.
• SBG2 did not have its own network room
• Backups were made in the same data center
• the floors of the five-story SBG2 data center were made of wood.
• it has been reported in various places that OVHcloud did not have an automatic fire extinguishing system

The presence of wood and plastics, and the lack of a fully automatic fire extinguishing system were also mentioned in a ZDNet.fr article in April.

"Those in charge will have to draw the consequences of so many accumulated errors," said the VO News article. "Wanting to sell at the lowest price is a fact, not taking responsible measures to ensure a minimum of security is a strategic error that is totally attributable to them, and some customers who have lost everything will make them pay dearly."