Water is a core part of many data center cooling systems. But as densities - and therefore temperatures - increase, questions need to be asked about the right temperatures of the water cooling these systems.
As the chips running servers become denser and more powerful, operators are faced with questions around whether to lower the temperature of the water going to these chips, to the point we will have to start focusing more on cooling the water systems.
The move to liquid
Historically, data centers have been kept at around 20°C to 22°C, but groups such as the American Society of Heating, Refrigeration and Air-Conditioning Engineers (ASHRAE) have been advising organizations to set thermostats higher for years. As a result, data center temperatures have been creeping up: Facebook parent company Meta raised its temperatures to 29.4°C, Google went up to 26.6°C, and Microsoft has published guidelines suggesting temperatures could go up to 27°C.
Typical legacy data centers have chilled water set points between 42-45°F (6-7°C). Facilities that have gone through optimization of their cooling systems have successfully raised their chilled water temperatures to 50°F (10°C) or higher. According to Johnson Controls, it is estimated that for every 1°C (1.8°F) increase in the temperature of chilled water, there is approximately 2-3 percent savings in power consumption for a typical chiller.
A recent DCD Broadcast analyzed a case study about how a service provider in the UK achieved a £1.5m ($1.9m) annual saving by increasing the temperature of the data hall, which only translated into a 0.3 percent increase in hardware failure risk.
“Cooling has always been the second-largest consumer of energy in the data center after the IT load, and this is mostly energy used to cool whatever the heat transfer medium is - be it air or liquid. So the less energy is spent there, the better the overall efficiency of the facility,” says Vlad-Gabriel Anghel, director of solutions engineering at DCD’s training unit DCD>Academy.
The picture is changing as the industry moves towards predominantly liquid-cooled data centers, where a liquid such as water circulates directly over the heat-producing components and removes heat. Water has a much higher thermal capacity than air, meaning data centers can support higher-density chips and use less energy to cool them.
Though air-based cooling options exist for racks drawing more than 20kW, the drawbacks start to outweigh the benefits, leading operators to switch to liquid systems. For years, 30kW was seen as the top-end of high-density deployments, and air was good enough. With the advent of generative AI and what classes as ‘high-density’ now potentially reaching more than 100kW, sticking with air cooling alone is no longer an option.
The fluid running over liquid systems has a much higher temperature than that found in chilled water systems, but the industry is yet to standardize on the best approach. At the same time, chips are becoming increasingly dense, and the temperatures of the water being supplied to these systems is coming down.
Data center operators have long been accused of being over-cautious by overcooling their air-cooled data centers to protect the IT hardware and avoid even the merest risk of overheating the data halls. Showing too much trepidation on liquid cooling risks the same issue.
Higher water temperatures mean less energy used for cooling – great for PUE – but risks running chips closer to their thermal limit. So, how hot is too hot?
What is the right temperature for water?
ASHRAE introduced a paper on liquid cooling back in 2011. The paper set out broad classes - W1, W2, W3, W4, and W5, based on the cooling temperature. Originally those classes were 17°C, 27°C, 32°C, 45°C and Over 45°C, respectively. When the work was updated in 2022, new temperature refinements were required, including a temperature of 40°C, and ASHRAE moved to new class definitions: W17, W27, W32, W40, W45, and W+.
DCD>Academy’s Anghel says there is no optimal temperature for water in liquid-cooled systems, because the best temperature will vary depending on the set-up of the facility.
“This will depend entirely on the type of liquid cooling used as well as the environment the liquid cooling system is in, the type of chip and its TDP as well as the utilization of the chip,” he says. “A rear-door air-assisted liquid cooling solution will have different temperatures to a closed loop direct to chip cooling system.”
According to Uptime, water temperatures in liquid-cooled systems today seem to be converging around 32°C (89.6°F) for facility water – what is described as a “good balance” between facility efficiency, cooling capacity, and support for a wide range of DLC systems. The company notes, however, that this often requires additional heat rejection infrastructure either in the form of water evaporation or mechanical cooling for higher-density chips.
“Many operators have already opted for conservative water temperatures as they upgrade their facilities to incorporate a blend of air and liquid cooled IT. Others will install DLC systems that are not connected to a water supply but are air-cooled using fans and large radiators,” the company said in a recent report.
The analyst firm notes current high-end processors (up to 350W thermal design power) and accelerators (up to 700W on some GPUs) can be “effectively” cooled even at high liquid coolant temperatures, allowing the facility water supply for the Direct Liquid Cooling system to be running as high as 104°F (40°C), and even up to 113°F (45°C).
Andrew Bradner, general manager for Schneider Electric’s cooling business, tells DCD, however, that after chips reach 500W, supply water temperatures have to come down to 85°F (30°C). And for 700W, the temperature may have to come down to as low as 80°F (27°C).
“This idea that you're going to run liquid cooling at 122-140°F (50-60°C) water is probably going to be highly unlikely, especially in the training loads,” Bradner says.
And in the same way air-cooled data centers have generally been run colder out of caution, customers using liquid cooling deployments are being equally prudent.
As part of its AI-focused redesign, Meta has settled on 85°F (30°C) for the water it supplies to the hardware, and hopes to get the temperature more widely adopted through the Open Compute Project.
Anecdotally, however, DCD has heard operators who expected customers to go with the ASHRAE definition of W27 (27°C/80°F output water) and are instead opting for the W17 (17°C/62°F) option.
“There's a lot of discussion at times that the water temperatures are going to go to 104-122°F (40-50°C),” says Bradner. “But as the power densities of the GPUs start to get over 500W to 700W each, the case temperatures that they're starting to see are requiring that that water comes down lower.”
“Once you've hit 700W, the water temperature has to come down to about 80°F (27°C). And we have customers that are asking for between 68-75°F (20- 24°C) supply water.”
Free cooling vs assisted cooling
When you need 68-75°F (20- 24°C) temperature water, some assisted cooling technologies are required in many cases – especially in hotter climates.
As an example, Bradner said Schneider recently performed an assessment with a partner around free-cooling – which relies on pulling in naturally cool air or water instead of mechanical refrigeration – at higher densities.
As long as the chip densities were at 300W, 95 percent of their sites could get away with not having any type of mechanical assist to provide the water temperatures they needed to run a liquid system.
However, once chips went over 500W, only five percent of their sites could support free-cooling, and 95 percent of their sites needed some sort of compressor mechanical-assisted solution.
“So I think that's the challenge,” says Bradner; “As the chips get more powerful and more power hungry, the internal dissipation that needs to happen to the chip case housing requires colder water to be able to still support reliable cooling of those chips.”
But he notes that for water temperatures around 80-86°F (27-30°C), there are still large parts of the year that you're going to get free cooling, and operators may only need assisted cooling for the hottest summer months.
“Right now, you can run the 300W-400W chips that are available with far higher water temperatures," Bradner says. “But that's going to change dramatically once these more powerful GPUs become readily available and deployed at scale. We're seeing many of our largest customers are talking about water temperatures that are more in the 80-86°F (27-30°C) range, not 104-122°F (40-50°C).”
DCD>Academy’s Anghel warns that if water temperatures are set too low, operators risk overcooling chips and ultimately wasting energy and repeating mistakes long made with air cooling.
“Any watt spent cooling water is another watt removed from the IT load,” he says. “The same efficiency mistakes are being made regardless of the cooling medium.”
This feature first appeared in the DCD Cooling Supplement. Register here to read the supplement free of charge.