Swiss technology firm Proton has suffered an outage after a cooling failure at a data center in Frankfurt, Germany.

“We have identified a critical cooling failure in one of our data centers. We're in the process of shifting traffic to our backup sites,” the company posted online after midnight Central Time today (August 27).

Proton services, which include the privacy-focused ProtonMail, were brought back online around 2am CEST.

It is unclear which facility Proton services are hosted in.

Proton founder and CEO Andy Yen apologized for the incident online, calling it an “unprecedented total cooling failure” that required a more time-consuming manual recovery.

Noting the response was slow and outside the company’s SLA, Yen admitted: “We fucked up, and will do better.”

He later posted a graph of the temperature spike its environment saw after the cooling failure.

“It's truly frightening how fast temperatures can rise in a high power data center in the event of a total cooling failure,” he said. “The baseline is 30°C (86°F), and the peak is 60°C (140°F). It's a 30°C climb in about 20 minutes.”

On why failover took so long, Yen said that it “took longer than usual” as the data center was “only partially dead,” meaning automatic failover didn’t kick in, and staff were “busy initially trying to get the cooling back before our servers are permanently damaged.”

He added: “This particular failure mode was actually known to us, but mitigations were not prioritized because the scenario was judged to be extremely unlikely. We are now working on implementing changes to ensure this type of incident cannot recur.”

When asked by a customer in the replies why the Swiss company was using a data center in Germany, Proton said: “We have our initial data centers in Geneva and Zurich which [are] still running. Frankfurt is an addition, not a replacement. It's there for bandwidth because Frankfurt has Europe's largest Internet exchange point.”

The company previously held a poll of its Reddit community to decide on a new data center location, with its users selecting Germany. The company launched there around 2021 in a facility “next to” the DE-CIX Internet exchange.

Another poll taken three years ago saw the community select Norway for another new location, which is now live.

Proton noted this week, however, that: “As a Swiss company, Switzerland is still our legal jurisdiction, regardless of our data center location. While zero-access encryption means we can't read your data wherever the servers sit.”

Proton previously hosted its infrastructure in a former K7 military bunker, located in Switzerland in Attinghausen under 1,000 metres (3,300 ft) of granite. The company was also previously hosted at a site in Lausanne.