In today's information-driven environment, the ability to quickly and efficiently process the maximum amount of data is a necessity for today's organisations. In order to accommodate this sea of crucial data, organisations are racing to purchase more powerful computer servers. While this new generation of slimmer, faster servers is more than capable of processing the data, there may be challenges in implementing such technology in facilities that are not properly equipped to handle this new higher density equipment.

The increased power densities of these new servers result in increased air conditioning requirements. When installed into a data centre that is not outfitted with the proper cooling equipment - and often more importantly configuration - overheating may cause server thermal overloads.

When the University of Southern California's (USC) Center for High Performance Computing and Communications (HPCC) sought to upgrade its existing data center, the goal was to create an environment that could accommodate a higherranked, high-density supercomputing facility.

 The HPCC brings together a breadth of computing and data resources through a Linux cluster supercomputer and a Sun Fire 15K shared memory system. HPCC also has a condor cluster that uses spare cycles on Unix workstations in USC's public user rooms, as well as a large facility that provides more than 30 terabytes of combined disk storage and potential access to more than one petabyte of tape storage.

This high-density centre is a primary engine for USC's broad research capabilities, providing interdisciplinary support for numerous schools and departments across the university.

Unfortunately, the main data centre that houses the supercomputer had a number of inherent limitations. The following case study will document the collaborative efforts of the Information Services Department (ISD) staff, Facilities Management Services (FMS) staff, and Syska Hennessy Group to evaluate and overcome these limitations and to deliver one of the fastest supercomputing facilities in the world.

The Need
While the academic environment is not regularly associated with "high dollar value,' the grants and funding associated with an institution's research capabilities are highly dependent on its ability to meet performance benchmarks. As a result, USC set out to increase its computing capacity and the supporting infrastructure.

The two USC departments involved in the project were the Information Services Department (ISD) and the Facilities Management Services (FMS).

Unfortunately, as USC's IT department began partial installation of the equipment, they quickly discovered that the server output reached excessively high temperatures. Aggravating this condition was poor air circulation, causing recirculation of the high temperature discharge. Rack inlet conditions in some areas were found in excess of 95 degrees F.

In order to circumvent these dangerously high temperatures, FMS deployed a series of short-term solutions, which included relocating some of the floor tiles within the data centre to help improve air distribution but the problems continued.

When it was evident that these solutions would not be sufficient for long-term operations, FMS selected Syska Hennessy group to study and measure the facility in a live environment to validate their assumptions and deliver a solution that would efficiently cool the data centre. Despite the broad scope of this endeavor, Syska was given only two months to design and implement the solutions needed before benchmark testing of the computer equipment had to be performed. This required Syska to work closely with both the FMS and ISD departments, with all parties leveraging several key partnerships with contractors, vendors and other consultants.

Initial Findings
Prior to setting out a strategy, Syska underwent an initial evaluation of the facility. While USC had recently performed an electrical infrastructure upgrade, Syska suspected that other infrastructure systems, particularly the air conditioning systems, were not sized to match the electrical upgrade. Surprisingly, Syska found that the facility did indeed have enough HVAC capacity to support the electrical load in the space. However, most of the load was concentrated within a smaller portion of the data centre, causing a distribution problem. Simply put, the air could not go where it was most needed.

Several factors contributed to the distribution problem, beginning with the building. Originally built in the 1930s, the building was originally a convenience store, before being converted to an academic building by USC and eventually, a data centre. As a result, Syska would have to work with a facility that had a number of built-in limitations.

One such limitation existed in the raised floor. The raised floor was only 12' high. This severely limited the amount of air that could be delivered through the underfloor environment. In addition, the underfloor was utilised for power and communication cabling distribution. In the densely populated areas of the data centre, there were significant cable obstructions to the airflow. In addition to the raised floor height, there was an inherent limitation in the type of floor tile being used for distribution. The raised floor air distribution system utilised traditional 25 per cent free area floor tiles, which were unable to provide the needed airflow to cool high-density racks.

In addition to the free area problem, these tiles had volume dampers which had over time, closed and was severely limiting the discharge airflow. Measurements of the airflow prior to performing any modifications to the data centre showed that these tiles were providing an average airflow of 80 CFM per tile. Many tiles had so little flow, that our flow measuring device did not register any airflow.

Additionally, the data centre also had varying rack configurations, which also contributed to the poor air distribution. Racks were not initially configured in a hot/cold aisle arrangement. Some racks were passively cooled (no cabinet fans), some had top mounted fans, some had rear mounted fans, and others had some combination of both. The different airflow patterns often conflicted with each other.

CFD Modeling
Syska's extensive use and knowledge of "Computational Fluid Dynamics' (CFD) modeling allowed us to quickly identify many potential problems during our observation phase. Following these initial observations, Syska used CFD Modeling to help validate the root causes of the distribution problems. This method employs a computerised model of the data centre as a tool to predict in-cabinet temperatures to determine if server equipment is adequately cooled. The final result is a "CFD Model', capable of producing elaborate animations using 3D motion graphics and color plots that can show temperature and pressure variations in the room.

CFD modeling requires careful consideration of input conditions. The CFD software is a number crunching device. With any such device, if you put garbage in you get garbage out. Care must be taken, that the model accurately depicts what the conditions will be in real life. In addition, a certain degree of foresight with regard to technology must be considered. Since it is counterproductive to develop a model that only optimises equipment that is in the data centre today, the model must be developed with the consideration of the equipment of the future. The primary goal is to design a facility with an infrastructure that is flexible enough to support both current and future technologies.

In order to assist Syska in providing the utmost flexibility in design, ISD was able to provide an extensive equipment roll-out schedule that allowed the proper capacity planning to be put forth during the design phase. This allowed Syska to enter this data into the CFD model and also enabled them to better prepare the facility for the equipment before it arrived on site.

The CFD Model illuminated four areas that needed to be addressed:

1. Server inlet conditions were not being satisfied with some inlet conditions as high as 95 degrees F. Much of which was caused by recirculation.
2. The standard floor tiles in the data centre did not provide the adequate quantity of air needed to cool the high-density servers and cabinets.
3. Re-circulation and air distribution problems existed at the upper end of the cabinets due to the low ceiling heights of 8'-6'.
4. The facility's 12-inch, raised floors were not conducive to providing necessary airflow to the equipment, aggravated by the high degree of blockage underneath the floor.

The Solution
Once the CFD model helped determine the primary causes of the distribution problem, Syska was able to quickly and efficiently provide a number of solutions to address the problem.

Syska began by addressing the floor tiles, using non-traditional floor tile, one with higher free area. The result was a dramatic increase in airflow by 300-400 percent in most areas and remarkably higher in certain areas. While this dramatically increased the cooling airflow to the cabinets, it did not fully address the air distribution problem. With several of the cabinets in the facility with airflow requirements measured between 1,200 and 2,000 CFM, even a 1,000 percent increase in airflow the adequate cooling airflow to satisfy the load in the HPCC cabinets.

As a result, Syska needed to evaluate additional ways to boost the facility's cooling capacity in the high density area. The raised floor alone simply was not capable of providing the required airflow in this situation. Had the raised floor been higher, the raised floor would most likely been able to support the cooling demand.

The building was built as a convenience store and when it was converted to a datacenter, the high ceiling was not utilised and a lower drop ceiling at 8'-6' was utilised. The low ceiling allowed the high temperature discharge air to stratify below the ceiling level and promoted the high temperature air to re-circulate into the rack inlets. Opening up the ceiling plenum to allow the high temperature air to stratify higher and utilise the large ceiling cavity was an attractive idea. However, the configuration of the fire system would not allow for the use of valuable space. As a result, Syska was required to evaluate alternative options to relieve the high temperature condition.

The solution was to implement a unique supplemental cooling system. Syska optimised the floor tile layout and employed an XD system as a supplemental air solution. Working closely with Liebert Corporation, a provider of cooling solutions for critical systems, Syska employed a combination of Liebert's Extreme Density Chiller (XDC) and Cabinet Mounted Supplemental Cooling Unit (XDV).

It was during this stage of the project where relationships played a vital role. While many cooling systems require anywhere between four and 12 weeks for delivery, relationships with Liebert by all three parties, ISD, FMS, and Syska were key in expediting equipment delivery. Furthermore, since FMS had strong contractor relationships, they were able to bring Graycon Mechanical on board during the design process to help facilitate a faster construction completion schedule. Since the XD system had the longest lead times, the facility was comprehensively piped and prepared for the equipment even prior to its arrival.

Syska was able to leverage relationships with floor tile vendors to allow for immediate delivery of the 56 per cent free area tiles.

The most remarkable fact of the quick delivery time was that the installation of the XDC was a first field application (FFA). Liebert's development staff was proactively involved in the installation process, by coming on site to support the start-up of the equipment, the first installed of its kind in the country. This helped to insure that the chillers functioned properly from the outset, which was paramount to keeping the project on schedule.

One of the major challenges to this aspect of the project was the need to keep the data centre running throughout the construction process. Being a new technology, there were some challenges with the installation of the XDV units. The rack mounted units were designed to bolt on to the top of the cabinet. However, the racks in this case had running computer equipment that could not be taken down during the construction process.

Fortunately, Graycon assisted in the identification of a unique mounting method - a Unistrut system was installed along the top racks, with each unit mounted to the Unistrut. This enabled smooth installation without disturbing the servers.

Another key consideration was cost. Ideally, two XDC units would have been installed to provide redundant capacity. In this case, only one chiller was employed mated with 20 XDV units. The design allowed for an additional further XDC to be installed in the mechanical refrigeration room. In addition, the Unistrut was installed to allow for future XDV installations. While the optimum model called for one XDV on every rack, the compromise was to use approximately one XDV for every other rack. The XDV piping was installed to allow for a redundant set of pipes to be installed in the future on the same pipe hanging system.

Code compliance issues also represented a substantial challenge. Since the high-density environment blends the IT technology with the facility's HVAC infrastructure, a great deal of coordination was needed between the engineers, consultants, facility staff and IT staff to work within the parameters of the Los Angeles city codes.

Adhering to deadlines in these types of projects is of the utmost importance. If the project was not capable of meeting the benchmarking deadlines, USC would have been denied crucial funding. As a result, these multiple challenges needed to be addressed and quickly and efficiently through a multitiered effort, which involved multiple parties at USC, Syska, Liebert, and the contractor. This approach enabled the project to be completed within the specified timeframe.

Flexibility for Future Growth
When designing a facility that is heavily dependent on technological advancement, flexibility is integral to long-term operations. Syska needed to deliver solutions that were capable of supporting future growth and would seamlessly integrate with next-generation technologies.

In order to ensure this flexibility, Syska allocated the space and piping throughout the facility, enabling for USC to add to and expand their systems to accommodate new technologies.

The Outcome
Syska and USC's ability to deliver this groundbreaking project within a short timeframe yielded a multiplicity of favorable outcomes.

The most notable accomplishment was Syska's ability to dramatically increase the capacity of the data centre. The upgrade increased the system's total number of compute nodes from 989 to 1,364.

As a result, according to Top 500 Supercomputer Sites June 2005 rankings, USC's HPCC is currently the fourth fastest super computing facility in a US academic setting, housing the 37th most powerful computer system in the world. This distinction was achieved as a result of the cluster's capability to make a 5.5 trillion calculations per second.

Also of note was USC's ability to achieve this lauded distinction through only modest local investments. Conversely, many of the other ranked systems were the result of major projects supported by national funding sources.

By combining its visionary design engineering capabilities with USC's strong vendor relationships, Syska Hennessy Group successfully delivered one of the world's most powerful computing facilities in only two months from the beginning of design to the end of construction. Furthermore, the facility's flexible design will enable both technological and functional upgrades, positioning the data centre as a leading-edge facility for years to come.