A faulty air conditioning unit took down the Tab betting system in Australia, on the day of a major race last year, according to an accident report.
Tabcorp's services failed, on the day of the Melbourne Cup in November 2020, because an air conditioning fault in a Global Switch data center triggered a fire prevention system - which in turn created a shockwave that damaged Tabcorp's IT equipment. The incident cost Tabcorp AU$100 million (US$73m) in turnover, and an estimated AU$2 million (US$1.5m) in profit.
No fire, just smoke
Early reports said there was a fire in the Global Switch data center in Ultimo, Sydney, but a detailed post-mortem confirmed that although there was no fire, a faulty bearing in an air conditioning unit created friction, heat, and smoke, which triggered the fire suppression unit.
This in turn released a fire retardant gas, producing a shockwave strong enough to damage what the report describes as "highly sensitive IT equipment" in racks at the data center.
The equipment failure took down the Tab betting service for more than 27 hours over the weekend - as well as destroying data on internal systems. The outage also disadvantaged staff: Tabcorp was in the midst of a corporate restructuring with senior managers and IT heads required to reapply for their jobs in the week before the race meeting. Because data was lost, those staff had to reapply for their jobs all over again on the following Monday morning, according to the Sydney Morning Herald.
The outage took place on "Stakes Day" when high gambling was expected. Race organizer Racing Victoria reported in November that "turnover dropped from $104 million in 2019 to $80 million on the Stakes Day meeting – a decline of 23 percent year-on-year". However, this was somewhat offset by domestic wagering over the four days, which grew by 6 percent to $667.3 million. Racing Victoria has said it is reserving the right to seek compensation.
Tabcorp initially blamed a "smoke and likely fire incident” at the Global Switch-owned data center as the cause of the outage, which meant they were unable to take bets for more than 27 hours over the weekend, costing more than AU$100 million in turnover.
“It was very unfortunate the outage occurred and especially on such a big day for our customers, for TAB and the racing industry,” said Tabcorp’s managing director of wagering and media Adam Rytenskild. “We have responded and are taking steps to minimize the chance of anything like that happening again.”
The report - which ITnews reports was produced by Deloitte - says the incident was entirely unforeseeable, as the air conditioning unit gave no indication a mechanical failure was imminent.
The air conditioning unit which failed was in Global Switch's area of the colocation data center while the IT systems were in Tabcorp's space.
Fire suppression units have caused IT failures in the past; For instance, In 2016, ING bank in Turkey destroyed data on its hard drives, when a fire suppression system test caused a shockwave in its data center. In 2015, a fire suppression system went off in Glasgow Council's data center, causing vibrations that corrupted disk drives and disabling the Council's IT for some time.
Responding to the criticism of fire suppression systems, in 2016, Tyco launched "acoustic" nozzles which would no longer damage IT equipment, reducing the force of emission in line with increasingly sensitive hard drives.
The Global Switch datasheet for Sydney West says the facility uses an "Inergen" gas fire suppression system. Inergen is a trademark of Tyco, which merged with Johnson Controls in 2016. DCD has contacted Johnson Controls to ask whether a Tyco system was involved in this incident.
