An oil leak inside one of the computer room air conditioning units (CRACs) took a Canadian university’s entire data center offline for nearly 20 hours. According to the Kingston Whig-Standard, Queen’s University in Kingston, Ontario suffered an outage of its largest data center after the leak activated the fire suppression system.

Grant Hall at Queen's University
Grant Hall at Queen’s University – Wikimedia Commons

A series of unfortunate events

Bo Wandschneider, head of information technology services at Queen’s, told The Whig that a faulty air conditioner was spraying a mist of mineral oil onto IT equipment.

Fine particles of oil were eventually interpreted as smoke, and caused the activation of the inert gas fire suppression system, which took the servers offline and began to reduce concentration of oxygen across the data center.

As a result, all of the university’s information systems were inaccessible between about 11:30 am Sunday and 7:00 am Monday. These included the learning management system and applications necessary for course registration, which was due to kick off on Sunday.

The learning management system files were also corrupted, and needed to be restored from backup - something that took additional time.

“We delayed registration slightly, but got it running by midnight,” said Wandschneider. He added that in the course of the response all of the sensors in the data centre were damaged and will need replacement.

This episode is the latest to illustrate just how unpredictable data center outages can be. Last month, a chemical leak at one of Apple’s data centers in Maiden, North Carolina, sent five people to seek treatment at a hospital. Apple said the incident was probably caused by mishandling of chlorine used to clean the cooling systems.