“Liquid cooling at the processor level is the wave of the future, and it will transform the data center industry as we know it within the next five years.” That statement, and variants of it, have been made for at least ten years!
This technology has been on the verge of disrupting the market for a long time, yet apparently only modest progress has been made. Industry experts once said that the manufacturers were hesitant to invest huge amounts of capital into changing the form factor of their IT equipment to accommodate or optimize the liquid cooling technologies, at least not until they were confident that there was a market for it. Over the years, developments in technology have trickled into the marketplace and demonstration projects have been undertaken to test the technology’s viability. Confidence and acceptance of the liquid-cooled systems are on the rise, and we can finally say “liquid-cooling at the processor level is the wave of the future, and it will transform the data center industry in the next 5 years!” Perhaps that is an overstatement but history does tend to repeat itself.
Here comes the flood!
There are multiple liquid cooling technologies available today. One example of such liquid cooling is immersion cooling, which requires the IT equipment to be fully immersed in a liquid. That liquid, for obvious reasons, cannot be water since water conducts electiricity. The selected liquid must be non-conductive (or dielectric). There are two types of dielectric liquids; some are appropriate for single phase immersion (i.e. temperature increase to the liquid), others use phase change (i.e. evaporative cooling).
Another type of liquid cooling technology uses water, but the water never touches the electrical components. Rather, a heat exchanger (or cold plate) creates the barrier between the IT equipment component and the water, while cooled water circulates across the cold plate. The cold plates usually sit on the microprocessors, and sometimes also on the memory devices. The water keeps those components cool, while the remainder of the server, such as power supply, disk drives, etc. remain air-cooled. These types of liquid-cooled devices are frequently referred to as “hybrid” systems because of the combination of devices to deliver both air and water cooling.
So, with the availability of these liquid-cooling technologies, and with numerous successful demonstration projects, why are end users not rushing to buy these devices today? The reason may actually have more to do with the economics of the industry than with the viability of the technology. The “economics of the industry” in this context does not refer to the cost of the liquid-cooling technology, though there may be a first cost concern. The bigger issue relates to the investment the industry has made to date in the air-cooled IT equipment and its associated cooling infrastructure. The industry can’t afford to abandon this investment unless the air-cooled technologies have reached their maximum heat removing potential. Until the industry gets there, there would be a tremendous loss of value in not utilizing the installed capacity to its maximum potential.
How far can air-cooled technology go?
Not too long ago, owners and designers had the sense that cabinets approaching 20 to 30 kW load were approaching the natural limits of air-cooled devices. On the other hand, demonstration projects have shown cabinets upwards of 40 to 50 kW.
The manufacturers of the processors and internal components have, in essence, extended the life of the air-cooled IT devices by designing for warmer temperatures and higher air flow delta T across the devices. Whereas a typical delta T across servers was 20 to 30 ºF some years ago, the IT equipment manufacturers are now saying 40 to 45 ºF, or more, is achievable. Coupled with best practices in the data center, such as the use of containment, more cooling can be delivered with the same air flow. These incremental advances have pumped new life into technology that had previously been thought to be approaching a dead end.
Perhaps that natural end of the line will be reached in the near future, but staying on track with the current technologies, while making incremental advancements in air cooling, is the most cost effective use of the current investment in the industry.
At some point, the air-cooled devices will reach the limits on their potential, and the device manufacturers will need to adjust their designs to accommodate liquid cooling. These modifications will require the elimination of fins over the main processors, elimination of the server fans, and the elimination of all moving parts. They may also require a form factor change, perhaps changing some configurations to allow boards to be mounted vertically, instead of horizontally (though blade servers are already mounted that way). Other modifications that cannot be foreseen at this time may also be required, but there is little doubt that the change in designs will be costly to the industry because it will require major modifications to existing product lines.
How will the transition happen?
Perhaps there is a transitional technology that will ease the industry into this change that is destined to come; if there is, that transitional technology may be the hybrid coolers noted earlier. Hybrid coolers can be more efficient than today’s air-cooled devices, but they have the disadvantage that the data center supporting hybrid systems require two cooling plants, or at least two cooling loops: one to cool the water that runs through the cold plates on the IT devices, and an independent one that cools all the other air-cooled loads in the building.
Providing for two plants (or loops) does get costly and may have a poor return on investment. However, the best advances that can be made in the industry will occur when existing air-cooled data centers have reached their physical limit for cooling, and the only option for adding more capacity comes from refreshing the air-cooled IT in the data center to water-cooled IT equipment. Even if this change comes with the addition of plants to provide the cooling water to the new water-cooled equipment, the addition of this infrastructure is still more favorable than building new data centers from the ground up.
As those modifications become commonplace, the industry will see a clearer path to the purer, more leading edge design of immersion cooling. That change will come in the next five years, it will dramatically change the industry, and we’ll check back again in ten years.
Vali Sorell is president at Sorell Engineering, and Marc Hourican is a partner at Syska Hennessy Group