A number of UK television channels went off the air over the weekend after a fire suppression system was triggered at a broadcast facility in London.
The evening of Saturday 25th saw broadcast services company Red Bee Media post on twitter that the fire suppression system had been triggered at its London broadcast center and staff had been evacuated. It said “several services originating from the Broadcast Centre have been disrupted” as a result. The London Fire Brigade attended the scene.
Red Bee Media, formerly Ericsson Broadcast and Media Services (EBMS), provides a number of services including channel playout; essentially the technology that push out programming to viewers.
After staff were allowed back into the center, the company said it would make no further statements to “avoid speculation” until it had fully restored services and concluded a full investigation.
30 minute break
Channel 4 and Channel 5 were both temporarily taken off the air for around 30 minutes, while Welsh broadcaster S4C said its programming on Freeview was affected.
Channel 4 also confirmed to the BBC that two of its channels, Channel 4 and More 4, were temporarily off air on Saturday while its online All 4 service was also affected.
While on-demand was restored, the company’s live online streaming services were still suffering some disruptions as of last night (September 27) and resorted to telling users to 'watch television' instead, but online services seem to now be fully restored.
“Due to a recent incident in one of our broadcast centers, we are continuing to experience disruption to some of our services.,” Channel 4 said in an email to customers. “As a result, you may experience some interruption to our Live TV service on All 4. We apologize for this disruption to your All 4 service. Our teams are working hard to restore it as soon as possible.”
The Times said that smoke was detected at the Red Bee facility in west London, activating a fire suppression system that sucked all the oxygen out of the room. However, the safety measure also reportedly triggered a “sonic wave” that shut down broadcast servers, according to those familiar with the matter.
Channel 4 reportedly had to broadcast via an emergency recovery system as a result. The BBC avoided the worst of the disruption as it was able to switch its operation to Salford where its services are duplicated.
Fire suppression systems have caused outages in the past, notably at AWS in Frankfurt in June 2021, which closed for a time when staff had to evacuate because a fire suppression system made the atmosphere unbreathable. Back in 2017, Microsoft lost a European Azure data center for seven hours, because a fire suppression system caused air handler units to shut down.
When fire suppression systems are set off, they can cause a shock wave which can damage sensitive hard drives. This happened this year (July 2021) at betting site Tabcorp in Australia, and previously at ING bank in 2016, and at Glasgow City Council in 2015. However, this problem is well understood and should be receding by now, as suppression system makers now baffle their nozzles to prevent the shock wave .
More in Outages
Conference Session Fireside chat: Could Generative AI hack a data center?