The data centre industry is still relatively young and is constantly growing and changing. In recent memory we have gone through a significant number of changes within the industry, some small steps in approach and others more fundamental paradigm shifts. 

Unfortunately, the unpredictability of the industry has meant that there are many organizations that have good IT facilities, but are now looking at implementing higher densities than they were originally designed for. 

Build new, or retrofit?

In this situation there exists two options.

Firstly, if there is budget available or a significant need to improve efficiencies, reliability and resiliency, then making the decision to construct a brand new facility can be the most appealing. There are several advantages with this – the foremost being a blank canvas to start with – but there is one obstacle which can be prohibitive and that is cost. In many cases the costs for this can be justified and a return on the investment (ROI) realised, but it can be a challenge to convince others of this. 

The second option is to upgrade, retrofit or otherwise modify an existing facility to align more with the new IT strategy. In this way, the returns can be less, but so too is the investment required. 

Ultimately, the best way forward for any organization needs to be decided based on the specifics of that business; its drivers, its aspirations, and its limitations. 

‘High density’ means IT systems which have been engineered to take up a smaller physical footprint than more traditional systems and which, in turn, allows them to be grouped closer together.  Using these types of approach, a typical server rack, which may have averaged 4-5kW in the past, is often now designed to accommodate 20-30kW, with higher densities possible. 

There are several driving factors to this, which mean that many organizations are starting to explore the high density options which meet their specific application dependent requirements. 

Some users are also looking to adopt virtualization systems, such that at partial overall load, they can have a small number of systems running at peak power rather than many systems running at partial loads. Other users are implementing very high performance computing, where latency between systems is critical and close physical spacing is required for performance. 

In addition to these performance related drivers, some implementers are using higher density computing strategies to allow for more bespoke cooling solutions, with potential for more free cooling or even opening the possibilities to heat reclamation. 

Finally, space is one driver which is almost universal. High density solutions, ultimately, allow operators to get more compute power into the same physical space.

New technologies, new challenges

When looking to implement a high density deployment, there are several considerations to be aware of, some which can make a project a success or failure from the outset. 

High air-flow and heat generation are some of the more tangible characteristics of high density loads, and so it is no surprise that one of the first things people look to consider when deploying these types of systems is the ability of the cooling systems to cope. 

Most cooling systems are sized to deliver a certain amount of cooling, at specific conditions, and moving outside the limits of the design will impact the cooling capacity available. One of the first things to check before deploying a high density load, in a traditionally low density environment, is the original design conditions. 

Depending on the cooling system deployed, it may be that the systems are able to operate with the proposed load types, provided that they do not exceed the original total capacity of the system.  With other systems we may need to reassess the system in line with the new application. 

One consideration is that modern high density systems generally operate with a higher increase in temperature, which may be an issue for traditional refrigeration based systems, especially if deployed with strict air containment.  

Cooling options

If the existing cooling system is not able to support the load, with the new design temperatures and air flow rates, then new alternative systems may need to be explored. Fortunately, there are a wide variety of options here such as rear door cooling and in-row cooling systems. 

Importantly, it’s not just the ability of the cooling system to be able to work with the air- flow rates and temperatures, but also it’s the ability of the space. A traditionally installed un-contained hot and cold aisle environment will likely run into problems with higher density loads, not due to the inability of the cooling systems to manage the load, but because of the large amount of air which might bypass the servers, and the amount of recirculation which would occur. Both of these factors serve to reduce the effectiveness of the cooling systems, increasing the temperatures experienced in the IT devices themselves. Contained hot and cold aisles can support high density loads but only provided sufficient air is supplied to them and sufficient air containment is installed. Wide aisles can deliver more air, but this then comes at the cost of useful whitespace. 

Alternatively, the use of chimney cabinets is an approach which is able to support high density loads, with a minimal impact on the general room layout. One of the main deciding factors here is if a suspended ceiling is available or possible to be installed. 

Simulation’s what you need

It is difficult sometimes to assess if a space is able to perform, as intended, without risking the performance of the systems already installed. One option is to use computational fluid dynamics (CFD) tools to simulate how a high density load would impact the space, and also to review if the proposed mitigations, such as aisle containment or additional air tiles, are going to be sufficient. 

Once we are confident that the cooling systems can support the load, and the room and cabinet configuration can accommodate the air flow, then we have addressed the main issues with high density loads, from a cooling perspective, under normal conditions. 

Outside of these ‘normal conditions’ it has been generally accepted in low density environments for the cooling systems to be interrupted, albeit briefly, between power failure and generator load acceptance. High density loads, however, can quickly become out of condition without active cooling systems. Because of this, in many high density environments, consideration needs to be given as to how to provide continuous cooling through any power failure, possibly with the use of a secondary UPS system.  

Other than the potential need for a mechanical UPS, the main electrical distribution is, fortunately, not quite as sensitive to density. Provided we are to remain within the current power capacity of the facility, the main challenges here are in routing the power to the point of use. 

Upgrading power

Traditional low density environments generally use a 2N final distribution with A and B outlets at each cabinet footprint. Single Phase 32A is common, with some very low density environments provided with only 16A outlets.  This distribution is limited to around 6 - 7 kW in the UK for 32A, with some variance due to voltage of the supply and the power factor of the load. Achieving high density with cabinet loads up to 30kW, we need to either replace these supplies, including outlets, cables and breakers, with newer, higher power feeds or alternatively consider installing multiple outlets per cabinet. 

Replacing feeds with 63A single phase outlets is an option for loads up to around 13kW, but generally three phase power distribution is something which starts to make more sense. Three phase supplies can support up to around 20kW for 32A, with 63A three phase supplies being an option for loads up to 40 kW. Alternatively, there is the option to install additional outlets, so that cabinets are provided with more than one ‘A’ supply and more than one ‘B’ supply, and yet another alternative is moving from 2N to N+1 by introducing a ‘C’ supply.

All that considered, the most practical way of provisioning the power required will greatly depend on how the power systems are designed upstream. A limited number of breakers or a lack of capacity on room PDUs or UPS output panels can seriously limit the practical options available. 

Regardless of the way in which the cabinet level power is provided, there is a very real need to review and understand the level of redundancy and topology of the power supplies to the high density equipment. Lower density servers may have fairly simple, A and B power supply units, which match nicely with A and B power distribution, but, newer high density systems are often supplied with multiple power supplies, potentially with an odd number, and with various redundancy options. Not fully understanding this can compromise the redundancy of the overall system and ultimately cause unplanned downtime. For example, within a cabinet which is provided with A and B supplies, a high density system with 6 PSUs, at N+1 redundancy will not be being powered resiliently without additional source transfer switches.

Finally, once the cooling and power systems are considered and able to support the loads proposed, there are some further, more practical considerations, which are commonly overlooked. A key example of this is the power and data distribution within the cabinet. Untidy and congested power and data cabling within the rear of the cabinet, exacerbated by the more condensed nature of High Density systems, can cause both a practical challenge with accessibility and future maintainability, but, also  may present a serious issue with regards to air flow. 

If considered early in the procurement of the systems, provision can be made by selecting more suitable cabinet sizes and configurations, or by specifying hardware with direct feeds, rather than multiple C19s, or with high density data connectivity, such as MPO / MTP connectors.  Decisions made during the procurement of the hardware will significantly affect the requirements on the facility design. 

No one size fits all

In data centres, there really is no true ‘one size fits all’. What works for one user may not work for another. There are some systems which are more flexible, but, providing flexibility will generally increase capital costs and ultimately is likely not to be future proof. That said, there is certainly a balance to be achieved here, and consideration of current trends and future upgrade routes is definitely advisable. 

With any high density deployment, even within a brand new data centre, there is a need to consider carefully all the limitations of the facility against the demands of the systems being installed.  Trying to do this in an existing low density environment is not easy, but, it’s also not a total non-starter. Planned properly, a high density system can be installed in environments originally designed to support cabinets with one tenth of the new design density. 

Some of the best advice here is to plan the deployment fully, before committing to buy any specific hardware.  Hardware vendors provide several varieties of each type of device for a reason. There is a general consensus around best practices but if trying to work within the constraints of an existing environment, what is ‘best’ is going to be very specific to your exact problem.  

Zac Potts is critical infrastructure director at Sudlows