Glasgow City Council is still struggling to restore IT services after a catastrophic fault on Tuesday, in which IT equipment was damaged by a fire suppression system, apparently after the failure of an air-conditioning unit.
The council denies reports that its business has “ground to a halt”, and is reported to be considering legal action against its IT partners, after a “powerful blast” of gas hit IT equipment including storage systems. Officials have said that staff will still be paid and services will run, although the council confirmed to DatacenterDynamics, that its email system is down.
Fire suppression triggered
“The catastrophic failure of an air conditioning unit has damaged some of our IT equipment,” said a council spokesperson quoted in the Glasgow Evening Times on Wednesday, “Our email system is down but vital services, particularly those for vulnerable people, are being delivered manually.”
The failure happened at the council’s data center in a ground floor unit of the CityPark business center near the main A8 road. The data center is run by Access LLP, an “arms-length” joint venture set up by the Council and services company Serco. Phone calls to Access about the problem are being routed straight to the council’s switchboard.
Full details have not been released, but a Glasgow City Council spokesman told DatacenterDynamics that an air conditioning unit failed, releasing “some kind of gas”. This gas appears to have set off a fire suppression unit, perhaps by giving the false appearance of smoke in the facility, said the Council source.
Some media reported that IT systems were ”sprayed by fire extinguishers”, but fire suppression units are typically designed to flood data halls with an inert gas to extinguish flames. Reports also mention a powerful blast of gas which ”shook the building”.
The Register reports that the problem caused a fault with an IBM storage array, which apparently holds the council’s tax and benefit systems, as well as its email service, based on Microsoft Outlook.
The issue also apparently caused problems on the Council’s switchboard, which relies on a Cisco network, and gave schools trouble in logging into their IT systems which are also hosted on servers at CityPark.
The Council spokesman told us that a business continuity plan was in place, but not one which involved a hot standby system. All payments due to go out will be made, and staff are using manual systems where necessary.
Shockwave?
The exact cause won’t be known till a full report is made, but this sounds very much like a fault scenario reported in our artcile “Causes of Failure”, where the burst of fire suppression gas creates a shockwave whose vibration is extreme enough to kill hard drives in storage units.
Failures like this can be avoided by design, but problems are often kept secret because the industry doesn’t share information about failures: “Reliability is much worse than it need be, because we don’t share,” data center reliability expert Ed Ansett of i3 Consulting told DatacenterDynamics earlier this year.
Long term, the council’s servers should be back up in time: “All our data was backed up and the business continuity plans in place meant those services were manually delivered,” the council told DatacenterDynamics. ” said. ”Our business continuity plans have proven to be effective.”