Hurricane Harvey, the first major hurricane to hit the US in over ten years, has gained in momentum and is tearing through south-eastern Texas, causing catastrophic flooding and displacing tens of thousands of people from their homes in Houston, which has been the worst hit in the incident. Collectively the US Coast Guard and the Houston police rescued over 4,000 people yesterday alone, and at least nine people have died as a result of the storm.
Yesterday, it was estimated that over 280,000 Texan customers were dealing with ongoing power outages as a result of flooding of electric infrastructure. Meanwhile, data center operators’ contingencies have been put to the test - and according to representatives, have held up (so far).
Head above the water
None of the obvious players seem to have experienced any major disruptions: IBM’s Bluemix cloud facility was reportedly fully operational and serving clients without disruption yesterday. Telco and ISP Level 3 Communications stated it had experienced several isolated incidents over the weekend but that it had not experienced any major disruption to its service.
Equinix has a facility in Houston also, HO1 IBX, which it gained as part of its acquisition of Verizon’s data centers earlier this year, and this has been reported to be online but inaccessible to customers. The company also confirmed that all of its Houston employees are accounted for and safe.
Data Foundry’s two data centers have remained online throughout the storm. The company’s CTO, Edward Henigin, stated: “both our Houston data centers continue to be staffed 24×7.”
“We have not experienced any flooding concerns at either site, and both sites have had 100% uptime. Our facilities are stocked with food, water and cots.”
“Houston 2 is a purpose-built facility designed to withstand category 5 hurricane wind speeds. This site has not lost utility power, and we have not had to transition to our back-up generators. Houston 2 has showers and washing machines for staff and customers staying on site for extended periods.”
”Houston 1 is in West Houston. It is located inside the Marathon Oil Tower. We have not lost power to our systems, and we have not had to transition to our back-up generator.”
Dallas’ CyrusOne has engaged its ”emergency management and response plan,” and is reportedly still relying solely on utility power.
Although it is in Houston, Digital Realty’s site is located outside of the floodplain, and thus is unlikely to have been affected by the storm, but is unable to comment pending its merger with DFT.
Update: As per our comments’ section Fibertown director of operations Jacob Bertling said: “DRT may not be able to speak to their resiliency during the storm, but I certainly can. We operate a Suite on this property and we didn’t have as much as a flicker.
“As waters rose we managed to stay above it by the grace of God. Fibertown was up through the course of this event as were our many customers and we can credit DRT with a large portion of that for the data center they built for us.”
Precedents
Flooding is a common cause of electrical outages, and data center operators’ ability to endure the forces of nature have been put to the test time and time again.
Torrential rain and flooding in Seattle brought down T-Mobile’s network for half a day in 2007, and similarly an electrical fire caused by floods in London in 2010 caused interruptions in Internet services for tens of thousands of BT customers.
Another relevant example is Hurricane Sandy (aka Superstorm Sandy) of 2012, which was said to be the cause of the most flood related outages in New York City’s history (though as often is the case with failures, few companies were inclined to speak publicly about the incidents).
We do know that both InterNAP and ISP Datagram had their diesel generators submerged with water. Datagram’s servers hosted Buzzfeed, Huffington Post, and Gawker. IP backbone operator Init7 failed to switch over to its backup generator properly, losing connectivity to Miami and Los Angeles.
Similarly (and rather unluckily), Australia’s ICT infrastructure was hit consecutively in 2010 and then again in 2011, first in Melbourne when a panel on the roof of a Datacom owned facility was broken by heavy rain, causing extensive damage to clients’ equipment inside the facility; and the following year when major floods in Brisbane caused optical fiber ruptures on Optus’ network and caused damage to Telstra’s data centers.
In 2013, flash flooding in Toronto drowned a transformer station under 20 feet of water, and with it brought down one of the country’s biggest carrier hotels’ cooling systems, but its operators managed to keep their facilities online by relying on backup generators.
Finally, British mobile network operator Vodafone’s customers were unable to access services during the festive period in 2016 due to flooding in the company’s data center, which was located in central Leeds, a city which had been severely affected by storms throughout the month of December.
The future is wet
However unpleasant, such occurrences are likely to multiply, according to the UK’s climate change report published last year. The paper laid out the potential risks of an increase in global temperature on British infrastructure, noting that flooding as a result of changes in weather patterns and increased rainfall poses the greatest risk of all, and this even despite redundancy measures taken by the ICT industry.
Meanwhile, flooding in Houston is not expected to subside, as rain continues to pour down. Between Saturday and Monday, more than half of the area’s average annual rainfall fell. The National Hurricane Center has predicted that rainfall accumulations would rise by another 20 inches – and as much as 50 inches in isolated areas - before the storm is over. And even when it finally stops raining, it is expected that the flooding will take a long time to recede.
US President Trump is due to visit Texas today, having yesterday pledged to “support the effort” of recovery after the storm, which promises to be costly.
An earlier version of this article incorrectly stated that Data Foundry had needed to toggle between mains and backup power. The facility has remained online throughout. We regret the error.