In April 2023, Google services went offline across Europe as the company’s europe-west9-a cloud zone was powered down. Customers, including mobile network Orange and video game developer Ubisoft, were reportedly impacted by the 24-hour outage.
The root of the problem was a fire caused by a cooling system water pipe leak at a data center in Paris, used by the company and operated by Global Switch. According to Google’s incident report, the leak “originated in a non-Google portion of the facility, entered an associated uninterruptible power supply room, and led to a fire."
According to the report, “the fire required evacuation of the facility, engagement from the local fire department, and a power shutdown of the entire data center building for several hours.”
Global Switch’s other customers also lost access to their servers as a result of the blaze, and though it was swiftly brought under control, the incident highlights the damage that even a small water leak can do if not detected swiftly.
Luckily, help is at hand in the form of water leak detection systems. Though these systems have often been an afterthought in the data center construction process, the increasing amounts of liquid flowing through data halls in the AI era mean they are more important than ever before.
Taking a leak
Water leaks can emanate from a variety of sources. Computer room air conditioning units, of the type found in most older data centers, have liquid flowing through, so any damage to these systems can lead to flooding.
For smaller data centers, particularly those located in larger, general-use buildings such as office blocks, corroded pipework elsewhere can be a major issue, says Henry Ettinger, European service manager at Infiniti, a vendor that provides a range of data center solutions including leak detection.
“If you’re an SME company you might be in a high-rise building with a kitchen or a toilet directly above your data center,” Ettinger says. “There is not much in terms of protection between these facility rooms and data centers that facilitate critical hardware and supporting systems.”
As well as pipes, water can enter through leaky roofs, or due to human error, be that from accidental spills or improper maintenance of equipment, Ettinger adds.
For larger data centers, the challenges around water leak detection are somewhat different. “With bespoke buildings, a lot of the problems are engineered out pretty well,” says Iain Ames, director at Diamond Controls, a company that specializes in leak detection for data centers and other industrial and commercial settings. “Though you can get condensation in the cooling corridors in the summer.”
External factors, such as extreme weather, can also be a source of leaks, and as well as the threat such incidents pose to IT equipment, data center operators have to beware of the impact that leaks can have on the world around them. In November, a data center at an industrial park in Offenbach, Germany, suffered a leak that saw cooling water seep into the soil. The leak originated in the pipe system on the roof, and entered the ground beneath the building via a rainwater infiltration system.
The cooling water reportedly contained a "low concentration of additives for corrosion protection and preservation" and two of the substances in it were considered hazardous. Fortunately for the unnamed developer of the data center, the nearest drinking water wells were located 1.5km away, so the chances of contamination were low, but since the leak occurred, environmental health authorities in the region have been continuously monitoring the groundwater for signs of contamination.
Not only are the environmental stakes getting higher, but the amount of water in data centers is also on the rise. While cooling fluid is present in traditional air-cooled systems, many developers, particularly those operating high-density AI-focused environments, are plumping for liquid cooling solutions. These can involve either direct-to-chip cooling, where cooling fluid is pumped directly onto cooling plates which keep components running at a steady temperature, or immersion cooling, where servers are dunked into vats of fluid.
Because of this, and the increasingly large investments being made in data centers, operators are ratcheting up their spend on leak detection technology. DCD spoke to several vendors that have seen the size of the orders received for systems mushroom over the last 18 months.
Splashing out
Diamond Controls is one company which has found its services in high demand. It works with a range of data center companies including major colo providers.
Ames says the need to mitigate the risk of water damage has never been greater. “There’s more money in data centers than ever before with all the spending on AI,” he says. “Because of this, people want a foolproof investment and don’t want to take risks - they need to be able to certify and prove to their clients they’ve got safeguards in place.”
This hasn’t always been the case, Ames says. He founded his business in 2004 as an electrical contractor, carrying out a variety of jobs for clients including the installation of building management systems (BMS) at data centers and other industrial buildings. The BMS provides a central point of control for all electrical systems in a property, and through this work, Ames and his team became acquainted with water leak detection.
Spotting an opportunity, in 2008 Ames refocused his company to concentrate primarily on leak detection. “We’d previously just been doing installations,” he recalls. “But we started offering services across the whole life cycle of a system, including design, maintenance, and training. That has allowed us to become a specialist business and do it really well.”
Specialists were needed at the time, according to Ames, because water leak detection had been considered an afterthought when installing a data center BMS. “Specifications for these systems are often not project specific, consultants just copy and paste them in.
“We can bring an on-site perspective to the design process, and we’re starting to see customers demanding higher levels of coverage, showing more respect to leak detection systems, and having more engagement at the design stage. This is how you get a better solution for the end user, and it’s gone from an area where people would look to save money to something that’s an important part of the design and build process.”
Tale of the tape
While many parts of the data center have undergone rapid technological change in recent years, water leak detection technology has remained relatively unchanged.
Detection systems can broadly be split into two types: those that use water sensing cables, known as “leak detection tape,” and spot sensors.
Infinity’s Ettinger says his company recommends tape-based set-ups for most data centers. “These are physical sensors that provide a secure perimeter around the racks, potential water leak sources such as pipework, or even the perimeter of the data center,” he says. Detection tape involves installing meters of cable that snake around a data hall’s servers, either running along the top of racks with a drip tray underneath to catch any water that falls from above, or installed below the data center’s raised floor.
“Water leak detection cables consist of multiple conductive wires encased in a protective, flexible material,” Ettinger says. “These wires are usually spaced apart by non-conductive materials to prevent contact under normal conditions. When they come into contact with water, an alarm will be triggered.”
Indeed, the detection tape is based on the principle of a simple electrical circuit. The tape will usually contain two stainless steel wires with an insulating material between them. When the wires come into contact with water or any other conductive liquid, it reduces resistance, completing the circuit and triggering the alarm or alert on the facility’s BMS.
Lengths of tape can be up to 50 meters, and a data center will typically be split into different zones, meaning a leak can be pinpointed to a particular area of the server room. Diamond Controls’ Ames says this can be as small as a single square meter, making it easy to find and address a leak. “The important thing is what the BMS does with the information it receives from the leak detection system,” he says. “Many come with an integrated digital leak map.”
Ames says tape is “very sensitive,” meaning it can pick up signs of a leak very early and potentially spare a data center from major damage. But there are downsides to this, too, he explains.
“A common problem with tapes is that if they’re in areas of high-volume foot traffic, or places where plant equipment is being moved around, they can easily get contaminated,” Ames says. “This means they give alarms frequently and start to be ignored.”
For such areas, spot detectors are available which can be placed around the room, or under specific pieces of equipment. These sound the alarm when they come directly into contact with water, but don’t offer the same range of coverage as tape.
Liquid cooling has added an additional dimension to the leak detection conundrum, but Ames says developers are taking “more precautions” when it comes to moving coolant around new data centers. “They’re being a lot more careful about how they do it,” he says. However, liquid can pose problems in the design phase, he adds. “I’ve seen contractors occasionally try to run pipework through electrical switch rooms with a drip tray underneath,” Ames says. “In that situation, I would always try to get in early and ask them to reroute those pipes.”
Keeping it open
When it comes to running water leak detection systems, Ettinger says data center operators must stay on top of maintenance to ensure their equipment is protected.
“The general rule of thumb with detection tape is that it needs to be replaced every ten years,” he says. “Our advice is to carry out annual maintenance checks, and test each individual zone and the emergency backup batteries.”
Ames says that when installing new systems, opting for an open protocol can make life easier. Vendors such as nVent build their technology using open standards, meaning it is easier to connect and maintain.
“With an open protocol, multiple installation companies can work on a system,” Ames says. “That way, the end user gets a system that’s manageable and offers better long-term value. It also makes it easier for a company to train its own engineers so that they can keep it maintained throughout the year.”
When it comes to training, Ames says a lot of data center companies are still neglecting leak detection systems. “Very few end users get us in to train their staff,” he says. “We often do some after installation, but then people move on and accurate records aren’t kept, so no one knows what has been done and by whom.”
He adds: “My overall message when it comes to leak detection is to ‘keep it simple.’ These systems are there to detect water, so you don’t need to overengineer them.
“But once you’ve got coverage, it is important to consider how you will react when you get an alert. How do you respond? Which valves get shut down? Users often buy a leak detection system, arrange coverage, but don’t start looking up the chain. It’s important to take a holistic view.”