It was not a good day at OVH yesterday. Two separate issues at two of the hosting company’s French locations cut off services for all of their customers: an electrical problem at the company’s Strasbourg site left three data centers (SBG1, SBG2 and SBG4) without power, and a software bug brought down the company’s fiber optic network at its facility in Roubaix.

The company’s founder, owner, CEO and chairman, Octave Klaba, kept customers updated on Twitter throughout.

Blame it on your rough beginnings

“We are sincerely sorry. We have just experienced two simultaneous and independent events that impacted all RBX customers between 8:15am and 10:37am and all SBG customers between 7:15am and 11:15am.”

In a detailed post explaining the root cause of the electrical failure in Strasbourg, Klaba stated: “Why this failure? Why didn’t SBG withstand a simple power failure? Why couldn’t all the intelligence that we developed at OVG, prevent this catastrophe?

“The quick answer: SBG’s power grid inherited all the design flaws that were the result of the small ambitions initially expected for the location.”

In fact, the post continued, two of the three facilities (SBG1 and SBG4) having been built in haste (2 months for SBG1) out of shipping containers, did not allow for the integration of standard-level redundancy.

What’s more, the facilities were not equipped with dual electrical feeds, nor were SBG1 and 2 on separate power grids. This, according to OVH, is exceptional to the Strasbourg location.

As a result of the outage, the company plans to spend between €4m- €5m ($4.66m-$5.83m) on an alternate, 20MVA electrical feed, separating the facilities onto distinct power grids, and migrating customers whose servers are located in the shipping container structures, SBG1 and 4, into SBG2 and 3, before completely uninstalling the former.

Good things come in threes, not twos

At the same time, the company’s fiber optic network was cut off in its Roubaix facility - which has interconnection points in Paris, Frankfurt, Amsterdam, London and Brussels - due to a software bug in the optical equipment, which “caused the configuration to be lost and the connection to be cut.”

The problem was fixed by loading the software configuration backup as soon as the issue was identified as such.

Klaba assured customers that the company would be in touch with manufacturers to identify the cause of the bug. 

“Bugs can happen, but incidents that affect our clients can’t.”

“The ‘Paranoid mode’ in our staff needs to be pushed further in all of our designs.”

Later, he told customers to expect emails regarding their respective service level agreements in coming days.