As part of a demonstration sponsored by the California Energy Commission in support of the Silicon Valley Leadership Group’s data center summit, Lawrence Berkeley National Laboratory (LBNL) collaborated with APC by Schneider Electric to demonstrate a novel prototype data center cooling device. The device was installed at an LBNL data center in Berkeley, California.

It included two air-to-water heat exchangers -- unlike common single-heat-exchanger configurations -- one of them supplied with cooling-tower water. The other heat exchanger was connected to the building’s chilled-water supply, like it would be in a typical single-heat-exchanger system. It was discovered that using cooling water supplied by the cooling tower to the maximum extent possible before using the chilled-water supply can in some situations cut the amount of energy used by cooling infrastructure by about 50%.

Tower water use maximized

This system is different from conventional closed water-cooled devices (cooling one or two racks as an enclosed unit) or open (placed between a hot and cold aisle) and deployed in a row of IT equipment racks. Conventional data center cooling devices, supplied by companies like Rittal, HP and Emerson, transfer heat from the hot air originating from the IT equipment to water supplied by the building’s chilled water using one heat exchanger.

Chilled-water supply temperatures are usually near 45F because, in many cases, needs other than data center cooling must be accommodated. In many situations, data center cooling requirements can be met using much warmer water (for example 66F), which is especially true if the ASHRAE recommended IT equipment air inlet temperature of 80.6F is used.

The required cooling capacity using warmer water can be produced much more economically if supplied directly from the cooling-tower/treated loop, compared to the same cooling capacity provided by the building’s chilled-water system. Fortunately, cooling water from a cooling tower is often available nearby, requiring only a simple plumbing addition to route it inside the data center. Depending on the data center’s design, the cost of this plumbing modification may be a key consideration.

The net result of using this technology is to transfer the heat from your IT equipment to the outside environment using tower water to the maximum extent possible. This technology minimizes the use of compressor-based cooling that may or may not be needed in a few areas inside a given data center. This can be thought of as a spot water-side economizer that uses tower/treated water most (if not all) of the time, and chilled water only when needed.

The design

The target application for this APC InRow prototype design is deployment on the data center floor, interspersed between racks of IT equipment. The number of cooling units needed depends on heat density of nearby racks. The cooler prototype design is constructed with a set of fans that pulls air through two air-to-water heat exchangers in a series configuration.

The heat exchangers are configured such that air from the hot aisle enters the first heat exchanger, exits and enters the second heat exchanger. All the air flows through both heat exchangers as indicated in Figure 1.

Figure 1: InRow cooler prototype dual heat exchanger design schematic

The first heat exchanger is supplied by water from a cooling tower, provided in a separate treated loop after going through a plate-frame heat exchanger. In this prototype, water flow rate through the first (tower water) heat exchanger was not controlled and allowed to flow at approximately 25 gallons per minute for most of the tests.

The second heat exchanger had water supplied by a compressor-based chilled water process that is part of the larger building heating, ventilation and air conditioning system. The cooling water flow rate through the second heat exchanger was controlled by a control system supplied as part of the prototype device to obtain desired server air inlet temperatures.

The InRow cooler control system monitors environmental conditions of the InRow cooler entering and leaving air along with the temperature of the air entering IT equipment nearby to provide the required temperature and airflow rate going to the cold aisle. The embedded control system accomplishes this by adjusting fan speed and modulating valve that sets the chilled water flow rate through the second heat exchanger.

Approach to analysis

The prototype unit was installed in a row of IT equipment racks in a data center on the LBNL campus. The initial hot aisle/cold aisle airflow containment was improved to raise the temperature of the air supplied to the prototype cooling device by adding some plastic curtains on both ends of the hot aisle.

A Btu meter was installed on each heat exchanger and connected to the local Modbus for data recording. Each Btu meter provided the water flow rate along with entering and exiting water temperatures. This provided a heat transfer rate for each heat exchanger as a function of time.

An electrical power meter was installed and also connected to the Modbus network to provide data on the device’s energy use. The air temperatures entering and exiting the device were recorded using a SynapSense wireless network and recorded on the existing LBNL SynapSense system database.

A series of experiments were completed while varying water flow rates on each heat exchanger and fan speed selection. Nearby IT equipment power varied over time as part of a normal course of events, which provided some variability of heat load presented to the cooling device.

In order to assess performance for the other conditions, an equation for cross-flow heat exchangers with one fluid mixed (air) and one fluid unmixed (water) was fitted to the recorded data. The resulting equation provided a closed-form solution for obtaining an estimate of heat transfer rate as a function of fluid flow rates and fluid inlet temperature combinations that were not recorded during the demonstration, allowing analysis for a wide range of conditions.

The volumetric air flow rate for each of the five fan speed settings was estimated using a heat balance method that solved for flow rate as a function of entering and exiting air temperatures and heat transfer rate.

For the analysis, server thermal performance was fixed at 100 cfm per kW of IT equipment power. This constrains server airflow rate as a function of different IT equipment power levels. A tower water temperature of 68F and chilled water temperature of 45F was used.

Using a range of IT equipment power, various cooling water flow rates and air flow rates. performance for a number of typical data center configurations was studied.

The metric used for comparison was Partial Power Usage Effectiveness (pPUE) as defined by The Green Grid. It was used for comparison of IT equipment power and cooling effectiveness.

The results

Energy efficiency of the intended configuration for this InRow cooler prototype was investigated using the partial pPUE and then compared to conventional configurations.

Energy use components included in the pPUE analysis included the power consumed by a chilled water plant equipped with water-side economizing and the power consumed by the prototype InRow cooling device. The energy consumed by the chilled water plant was obtained using a model provided by Taylor Engineering that used San Jose, California, climate information.

Results for other installations may vary from those presented here, depending in part on climate information used. For example, climates with high wet bulb temperatures will have fewer hours when the tower water can be supplied at 68F.

In Figure 2, the results for a conventional single heat exchanger design using tower water is represented by the line indicated as Case 2. Case 2 indicates that a single heat exchanger design has better efficiency at lower cooling capacities.

Currently, the most common configuration uses building-supplied cooling water from a chilled water plant supplying a cooling device design that has a single heat exchanger. This configuration is indicated by the line labeled as Case 3 in Figure 2.

The temperature of the building supplied chilled water used for the analysis was 45F. If Case 3 (single heat exchanger - chilled water supplied) is compared to Case 2 (single heat exchanger - tower water supplied) a large efficiency difference occurs, due to the fact that a larger amount of energy is needed to produce the same amount of cooling for Case 3, because temperature of the cooling water provided is colder than required.

The line indicated as Case 1 in Figure 2 represents the configuration targeted by the dual heat exchanger InRow prototype cooler design. Case 1 has considerably more cooling capacity compared to Case 2.

Increased cooling capacity is obtained by adding an increasing amount of chilled water starting at approximately 36 kW of IT power cooled.

Case 1 has a small (approximately 1-2%) efficiency penalty compared to Case 2, because a larger amount of fan power must be allocated due to the increased airflow resistance through two heat exchangers compared to one. This penalty is highlighted by the offset seen between Case 1 and 2 between 30 kW and 36 kW of IT power cooled in Figure 2.

Figure 2: This demonstration (Case 1) energy use efficiency results compared to single heat exchanger configurations (Case 2, 3)

Others thinking outside the box

There are a number of other promising IT equipment cooling technologies that show potential for energy efficiency improvement, including direct liquid and immersion cooling. Direct cooling is typically constructed by attaching a heat-transfer device or material directly to or very near a surface of high heat generating components. This approach provides a more direct heat transfer path to the building cooling infrastructure, thus avoiding or reducing the need for room-wide air-cooling.

An immersion cooling design is typified by immersing the entire IT equipment electronic circuit board in a dielectric fluid. Cooling systems using mineral oil or a Halon-alternative fluoroketone are available or being actively developed. These approaches promise to provide IT equipment cooling without the need for cooling towers, cooling tower water use or chiller systems, even in extreme climates.

Efficiency in economization

Using warmer water supplied from a cooling tower for computer room cooling solutions can provide energy efficiency improvements on the order of 30% to 50%, compared to water supplied from compressor-based cooling.

The APC prototype InRow device demonstrated that it can act as a localized water-side economizer cooling solution. An additional benefit is provided by this design, colder compressor-based cooling is only needed in the hot spot locations on the data center floor.

Depending on data center design and IT equipment power density distribution, much of the cooling could be provided using only warmer water provided by a cooling tower. Significant energy efficiency improvements should be possible if InRow coolers using the design concept discussed here are put in place.

About the author: Henry Coles is a program manager at Lawrence Berkeley National Laboratory

This article originally appeared in the 28^th issue of the DatacenterDynamics FOCUS magazine. See the digital edition here.