The IBM Design Center Data Center in Poughkeepsie, New York, was running out of cooling capacity and needed to address continuing IT growth requirements.
The center was originally built in 1999 to host a client center whose mission was to enable client IT infrastructures with server and storage technologies. The facility was built in a former working lab within one of the campus buildings. It consisted of several client meeting rooms and data center space, which was divided between a showcase area and a production IT data center. The showcase area displayed the leading-edge server technologies in a glass enclosure, and the working space contained a production IT area of servers, storage, and network gear.
Cooling of the area to be transformed was provided by a single Liebert air-conditioning unit. A range of power configurations (standard 110V/15A to 3 Phase 220V/60A circuits) provided power to the IT equipment. The power and cooling was deemed sufficient to serve the IT requirements for the center’s mission for years to come. The center was built on a raised floor structure with perforated tiles to assist in air circulation.
Over the years the server inventory was continually updated to the latest technologies and additional capacity was added, putting strain on the infrastructure. The IT equipment – including BladeCenter, System p, System i, System x, System z, and TotalStorage – were regularly added to the Design Center. Each of these servers was a foundation for new technologies such as multi-core processors, extremely dense server enclosures, and virtualisation features.
The expansion was routinely handled by introducing new hardware. If needed, new power circuits were added. This trend continued for several years until it was noticed that the ambient temperature of the area was increasing. Some tuning of the cooling infrastructure was performed but this only provided a short-term fix.
By mid-2008, it was clear there was a problem with the server infrastructure in the Design Center; the power and cooling capacities of the facility had been increased to maximum levels and there was no further opportunity to add power or cooling without significant costs. Triggered alarms from the cooling unit became a common occurrence as temperatures rose. Something had to be done to fix the problem.
MEASUREMENT AND MANAGEMENT TECHNOLOGIES (MMT)
The Survey and Analysis module of the MMT service offering is a methodology for assessing the efficiency of the cooling within the entire data center. It is based on detailed temperature, humidity and airflow measurements taken using IBM-developed equipment and software.
MMT captures data using a unique sensor array, scanning all relevant sections of the data center from floor to just above the tops of the racks. A cart, shown in Figure 1, with thermal sensors mounted in a defined 3D pattern, is rolled through the data center while data is logged and mapped onto a co-ordinate grid. Measurements are also made on the air handling equipment, perforated tiles, and IT power levels within the facility.
The MMT cart and associated analysis software have been used in data centers throughout the world to identify inefficiencies, improve equipment cooling performance and save energy.
Along with the temperature and humidity data, the airflow and power information that is collected is then analysed using IBM software. A full 3D map is produced detailing temperature, humidity and airflow, as well as evaluation of the cooling system performance and energy consumption metrics.
Some of the key measurements surveyed in the Poughkeepsie data center include:
- Approximately 9,100 temperature measurements
- Measured airflow from all perforated tiles
- Return and discharge temperatures and airflow on the Air Conditioning Unit (ACU)
- All relevant data center power levels
Figure 2a and 2b show MMT scans at Z=0.5ft and 7.5ft, respectively, and the associated American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) recommended inlet air temperature guideline of 64.4 to 80.6 oF. Z denotes the vertical height above the raised floor. Note that at a height of just 0.5ft there are already very high inlet temperatures on the storage unit directly in front of the ACU. The inlet side of the rack is indicated with a blue line.
Figure 2a: MMT Scan at Z=0.5 ft
Figure 2b: MMT Scan at Z=7.5 ft
The MMT cart collects temperature data in intervals of 1ft, starting at 0.5ft off the floor. Note there is a significant temperature increase at the inlets at the top of the racks as seen in Figure 2b. In addition, there is significant recirculation on the servers in the top-left corner. The inlet air temperatures in some areas exceed the ASHRAE recommended rack inlet temperatures.
From the MMT-surveyed data various analytics, such as ACU utilization, airflow effectiveness, hot-spot areas, cold-spot areas, recirculation areas, rack inlet out of spec conditions, and other metrics were generated. These metrics were then used to improve on the cooling effectiveness of the data center and mitigate wasteful conditions.
Figure 3a: New data center layout
Figure 3b: CFD analysis of new data center layout
Figure 4: MMT measurements of new data center at 5.5ft above raised floor after renovations
REAR-DOOR HEAT EXCHANGER
A goal for the upgraded data center was to increase its IT capacity and to allow for future IT growth. An inventory of the facility’s IT equipment identified servers that could be consolidated into higher density racks. The target inventory included a 75 per cent increase in IT load.
A number of factors contributed to the design decisions for the data center. The ACU was nearing maximum utilization and the layout of the data center was not conducive to adding an additional ACU to the room. Also the business needs of the production data center did not allow for downtime. The decision was made to use use rear door heat exchangers (RDHx) to remove a significant portion of the heat from the distributed high-density racks. The implementation of the RDHx is much more energy efficient than an equivalent heat removal design using an air-handling unit. The RDHx was awarded the most efficient rack cooling solution at the 2008 Chill Off sponsored by the Silicon Valley Leadership Group and LBNL.
To determine the best possible layout of the equipment, the data center was modelled using computational fluid dynamics (CFD). More than 40 simulations were run to optimise the floor layout for the IT equipment. Figures 3a and 3b, respectively, show the recommended layout and corresponding CFD run.
The CFD model depicted in Figure 3b is the recommended solution for the new data center. It uses the RDHx liquid cooling solution to remove most of the heat gained by the air exiting through the backs of racks 1, and 3 to 8(see Figure 3a). Each RDHx can remove all the heat load of these racks which were less than 15 kW/rack thereby reducing any further load on the ACU. The ACU is the primary cooling element for the mainframes and storage racks.
IMPLEMENTED DATA CENTER FACILITIES ENHANCEMENTS
After the Data Center was renovated to support the new IT equipment, a final MMT Survey and Analysis was performed to verify how the changes made to the data center improved the thermal performance of the space. Major renovations included realigning all racks into hot/cold aisle arrangement, a patented air baffle placed between the zSeries and DS8000 storage racks to prevent air recirculation, LED lighting, uncluttered raised floor, and liquid cooling via 7 RDHx units on racks 1, and 3 through 8. The table below summarizes the key energy measurements before and after the renovation.
The Power Usage Effectiveness (PUE) was improved slightly while doubling the IT load in the same space footprint. This was achieved through use of the MMT tool and the seven RDHx’s. FreeCool in the table refers to the use of Hudson River water in a water-to-water exchanger to cool the chilled water loop on the campus. This allows data center managers to turn off the chillers and only power the pumps (reduces energy required by facilities) to provide cooling water to the site from the river.
By comparing the MMT data in Figure 4 to the CFD modelling results of Figure 3b and adjusting for color scale differences, it can be seen that the temperature distributions match well. The temperature profile is also significantly better than before (see Figure 2). There is no recirculation in Figure 4 and the temperatures are much lower in the cold aisles. The results show that even with a significant increase in the total IT heat load in the room, temperatures are maintained within the ASHRAE recommended rack inlet air temperature limits. This can be attributed to the careful study of the original layout of the room, MMT studies, the CFD modelling of the possible layout changes, and implementation of the recommended layout following data center power and cooling best practices.
SUCCESS FACTORS
Significant improvements were made to this data center. These were achieved through the use of the MMT Survey and Analysis study ( IBM service offering), the Rear Door Heat Exchanger, CFD modeling of possible lay-outs and thorough implementation of data center power and cooling best practices. These are all proven technologies or techniques to consider when planning or designing a data center facility. As an example of the effectiveness of one of these, the MMT service offering has been performed on numerous customer data centers and has consistently shown energy savings of 10 to 15% of IT power.
The key success factors of the Poughkeepsie data center are:
- Increase in IT Power by 53 kW ( 100% increase)
- Increase in IT Power Density by 51 W/sq ft
- Reduced average Rack Inlet Temperature by 6.6 F
- Increase in ACU Air Heat Load by 14.6 kW ( 25% increase)
- RDHx’s removed all of the heat load from racks 1, and 3 through 8
- PUE improved to as low as 1.12.
Related News: IBM takes on computational fluid dynamics with EMEA roll out of mobile measurement
